o ht&@s~dZddlZddlZddlZddlZddlmZddlTddlm Z m Z ej j ej j gZddZdZd Zd Zd Zd d dddZddZejddddZejddddZgdgdgZgdgZgdgZddZgdgdgZejgd ejd!ejgd"ejd!gZ gd#gZ!ejgd$ejd!gZ"gd%gZ#ejgd&ejd!gZ$d'd(Z%d)d*Z&d+d,Z'd-d.Z(d/d0Z)dS)1z Very simple test of the mwt counting functionality in tokenization/data.py TODO: could add a bunch more simple tests, including tests of reading the data from a temp file, for example N)Pipeline)*) DataLoader NUMERIC_REcCstjdd|dd}|j}||Wdn1swYtjdd|dd}|j}||Wd||fS1s@wY||fS)z Writes raw_text and labels to randomly named files in test_dir Note that the tempfiles are not set to automatically clean up. This will not be a problem if you put them in a tempdir. wzutf-8F)modeencodingdirdeleteN)tempfileNamedTemporaryFilenamewrite)test_dirraw_textlabelsfouttxt_file label_filerg/var/www/html/env_mimamsha/lib/python3.10/site-packages/stanza/tests/tokenization/test_tokenize_data.pywrite_tokenizer_inputs   rz2Sehr gute Beratung, schnelle Behebung der Probleme200010000100000000110000000010000000010001000000002z* Die Kosten sind definitiv auch im Rahmen.*000100000010000100000000010000100300000012de) space_before capitalizedi,F)lang feat_funcs max_seqlenuse_dictionarycCstjtd7}t|tt\}}tt||dd}|rJt|t t \}}tt||dd}|s5JWddS1s@wYdS)z0 One dataset has no mwt, the other does r txtlabel)args input_filesN) r TemporaryDirectoryTEST_WORKING_DIRr NO_MWT_TEXT NO_MWT_LABELSrFAKE_PROPERTIEShas_mwtMWT_TEXT MWT_LABELS)rrrdatarrr test_has_mwt4s "r0module)scopecCtdtddd}|jd}|S)Nentokenizer download_method processorsrTEST_MODELS_DIRr8pipeline tokenizerrrrr=A r=cCr3)Nzzh-hansr5r6r9r;rrrzhtokGr>r?)Trhrirsr rrDrFrHarrHtrerrFrL)frorrR)r@rBrDrFrHrDrFrHrJrHrLrNrFrLrHrPrRrR)r@rBrDrFrHrDrFrHrJrHrLrNrFrLrPrRrRcCsd}t|j||jd|jjd}|jtksJd}t|j||jd|jjd}|jtks,Jt|j}d|d<t|||jd|jjd}|jt ksHJdS)z; Tests converting a couple small segments to units zThis is a test fooT input_textvocab evaluation dictionaryzThis is a test foo skip_newlineN) rconfigrVtrainerrXr/EXPECTED_TWO_NL_RAWEXPECTED_ONE_NL_RAWdictEXPECTED_SKIP_NL_RAW)r=rbatchesskip_newline_configrrrtest_convert_units_raw_textRs rb)r@rBrDrFrHrDrFrHrJrHrLrNrFrL.)rrrrrrrrrrrrrrre)dtype)rrr)r@rBrDrFrHrDrFrHrJrHrLrNrFrLrcrHrPrRrR)rrrrrrrrrrrrrrrerrrr)r@rBrDrFrHrDrFrHrJrHrLrNrFrLrcrPrRrR)rrrrrrrrrrrrrrrerrrcCs<t|t|ks Jt||D] \}}t||sJqdS)N)lenzipnp array_equiv)rexpected_labelsr$expectedrrr check_labelsosrmcCs>tjtd}d}d}t|||\}}t|j||d|jd|jjd}|j t ks*Jt | t d}d}t|||\}}t|j||d|jd|jjd}|j tksTJt | tt|j}d|d <d}d}t|||\}}t|||d|jd|jjd}|j tksJt | tWd d S1swYd S) zJ Tests reading some text from a file and converting that to units r!z00000000000000000001 000 zThis is a test. foo r"T)r&rVrWrXz000000000000000000010000 zThis is a test. foo rYN)r r'r(rrrZrVr[rXr/EXPECTED_TWO_NL_FILErmrEXPECTED_TWO_NL_FILE_LABELSEXPECTED_ONE_NL_FILEEXPECTED_ONE_NL_FILE_LABELSr^EXPECTED_SKIP_NL_FILEEXPECTED_SKIP_NL_FILE_LABELS)r=rrrrrr`rarrrtest_convert_units_filets,   "rtcCs|jjdusJ|jjdusJd|jjvsJd|jjvs Jd|jjdvs*Jd|jjdvs4Jd|jjdvs>Jd |jjdvsHJdS) z Tests some features of the zh tokenizer dictionary The expectation is that the Chinese tokenizer will be serialized with a dictionary (if it ever gets serialized without, this test will warn us!) Nu老师u 蛋白质u蛋白prefixesu蛋u白质suffixesu质)r[lexiconrX)r?rrrtest_dictionarysrxcCsd}t|j||jd|jjd}|j}t|dksJt|ddks$Jgdgdgdgdgd gd g}t|D]\}}||d|}||ksNJqsR        "