o h+@sdZddlZddlZddlmZddlTddlmZddlmZddlm Z ej j ej j gZ dd Zd d Zd d ZddZddZddZddZdZddZdS)z~ Very simple test of the sentence slicing by tags TODO: could add a bunch more simple tests for the tokenization utils N)Pipeline)*)doc)data)utilscCsgd}t|dgksJgd}t|dgksJgd}t|dgks*Jgd}t|dgks8Jgd}t|dd gksGJd S) z0 Test various raw -> span manipulations )unbar moxr r pr l)r)rrr r rr r r rr r rr r)rrrr r rr r r rr r rr rr))rrrr r rr r r rr r rr r)rrrr r rrr r rr r rr r)r)rN)r find_spans)rawrh/var/www/html/env_mimamsha/lib/python3.10/site-packages/stanza/tests/tokenization/test_tokenize_utils.pytest_find_spanssrcCst|jt|ks Jt|j|D],\}}t|jt|ks Jt|j|D]\}}|j|dks3J|j|dks Test the conversion of pretokenized text to Document )Thisisr test Thisisatest))r)r,r)rr)r )unbanmoxopal!zThisisatest unban mox opal!)) )))r7ThisisatesttttN Thisisates)r(izr r*)rmatch_tokens_with_textr'pytestraises ValueError)rr"rrrtest_match_tokens_with_text0s"      "r@c Cs tdtdd}|jd}dd}tj|j||jd|jjd}d|_ t t  t jd|j||jdd ||jd d d \}}}}Wdn1sIwYtj|j||jd|jjd}t jd|j||jdd ||jd d d \}}}}t||}t|jdksJdS) zL Test the tokenizer's capacity to break text up into smaller chunks entokenize)dir processorszATIL not to ask a date to dress up as Smurfette on a first date. dT) input_textvocab evaluation dictionaryNi no_ssplitF) orig_textrJ)rTEST_MODELS_DIRrDr DataLoaderconfigrGtrainerrIadvance_old_batchr=r> TypeErrorroutput_predictionsgetrDocumentrr)pipeline tokenizerraw_textbatches_documentrrrtest_long_paragraphFs$     r[c sgdgdgd}ddddddd dd dd d d ddd ddddd ddddddddddgddddddddddd d d!d"ddd dd"d#ddgg}fd$d%}t|||}||ks`Jd&S)'z Check that the postprocessor behaves correctly by applying the identity postprocessor and hoping that it does indeed return correctly. IamJoe. ⭆⊱⇞Hi.I'mr chickenrb&I am Joe. ⭆⊱⇞ Hi. I'm a chicken.rr]ridtextr r!r^r,r_ r` r2rar SpaceAfter=Norhrir r!miscrrbrdr3r r5rer6 csSNr)rYgood_tokenizationrr postprocesorksz4test_postprocessor_application..postprocesorN)rpostprocess_doc)ri target_docr{resrryrtest_postprocessor_applicationas  rc Csgdgdg}dd|D}dd|D}d}ddd dd d d d d d ddddd d dddd ddddddddddd gddddd d d d!d"d dd#d$d%ddd dd%d&ddgg}t||||}||ksiJd'S)(z` Check that the reassembly code counts the indicies correctly, and including OOV chars. r\rccS"g|] }ddtt|DqS)cSg|]}dqSFr.0rYrrr x7test_reassembly_indexing...rangerrirrrrx"z,test_reassembly_indexing..cSr)cSrrxrrrrrryrrrrrrrryrrfrr]rrgrjr^r,rkr_rlrmr`rnr2rarrorprqrrbrsrdr3rtr r5rurer6rvrwN)rreassemble_doc_from_tokens)rz good_mwtsgood_expansionsrir}r~rrrtest_reassembly_indexingrsrc CsRgdg}ddtt|dDg}ddtt|dDg}gdg}ddtt|dDg}ddtt|dDg}gd g}d dtt|dDg}d dtt|dDg}d } ttt|||| Wd n1s}wYttt|||| Wd n1swYt|||| d S)zi Check that the reassembly code complains correctly when the user adds tokens that doesn't exist )JoeSmithlivesinSouthern CaliforniarbcSrrrrrrrrrz6test_reassembly_reference_failures..rcSrrxrrrrrrr)rrrr CaliforniaarbcSrrrrrrrrrcSrrxrrrrrrr)rrrrrrbcSrrrrrrrrrcSrrxrrrrrrrzJoe Smith lives in California.N)rrr=r>r?rr) bad_addition_tokenizationbad_addition_mwtsbad_addition_expansionsbad_inline_tokenizationbad_inline_mwtsbad_inline_expansionsrzrrrirrr"test_reassembly_reference_failuress"     ra # sent_id = weblog-juancole.com_juancole_20051126063000_ENG_20051126_063000-0003 # text = DPA: Iraqi authorities announced that they'd busted up three terrorist cells operating in Baghdad. 1 DPA DPA PROPN NNP Number=Sing 0 root 0:root SpaceAfter=No 2 : : PUNCT : _ 1 punct 1:punct _ 3 Iraqi Iraqi ADJ JJ Degree=Pos 4 amod 4:amod _ 4 authorities authority NOUN NNS Number=Plur 5 nsubj 5:nsubj _ 5 announced announce VERB VBD Mood=Ind|Number=Plur|Person=3|Tense=Past|VerbForm=Fin 1 parataxis 1:parataxis _ 6 that that SCONJ IN _ 9 mark 9:mark _ 7-8 they'd _ _ _ _ _ _ _ _ 7 they they PRON PRP Case=Nom|Number=Plur|Person=3|PronType=Prs 9 nsubj 9:nsubj _ 8 'd have AUX VBD Mood=Ind|Number=Plur|Person=3|Tense=Past|VerbForm=Fin 9 aux 9:aux _ 9 busted bust VERB VBN Tense=Past|VerbForm=Part 5 ccomp 5:ccomp _ 10 up up ADP RP _ 9 compound:prt 9:compound:prt _ 11 three three NUM CD NumForm=Digit|NumType=Card 13 nummod 13:nummod _ 12 terrorist terrorist ADJ JJ Degree=Pos 13 amod 13:amod _ 13 cells cell NOUN NNS Number=Plur 9 obj 9:obj _ 14 operating operate VERB VBG VerbForm=Ger 13 acl 13:acl _ 15 in in ADP IN _ 16 case 16:case _ 16 Baghdad Baghdad PROPN NNP Number=Sing 14 obl 14:obl:in SpaceAfter=No 17 . . PUNCT . _ 1 punct 1:punct _ # sent_id = weblog-juancole.com_juancole_20051126063000_ENG_20051126_063000-0004 # text = Two of them were being run by 2 officials of the Ministry of the Interior! 1 Two two NUM CD NumForm=Word|NumType=Card 6 nsubj:pass 6:nsubj:pass _ 2 of of ADP IN _ 3 case 3:case _ 3 them they PRON PRP Case=Acc|Number=Plur|Person=3|PronType=Prs 1 nmod 1:nmod:of _ 4 were be AUX VBD Mood=Ind|Number=Plur|Person=3|Tense=Past|VerbForm=Fin 6 aux 6:aux _ 5 being be AUX VBG VerbForm=Ger 6 aux:pass 6:aux:pass _ 6 run run VERB VBN Tense=Past|VerbForm=Part|Voice=Pass 0 root 0:root _ 7 by by ADP IN _ 9 case 9:case _ 8 2 2 NUM CD NumForm=Digit|NumType=Card 9 nummod 9:nummod _ 9 officials official NOUN NNS Number=Plur 6 obl 6:obl:by _ 10 of of ADP IN _ 12 case 12:case _ 11 the the DET DT Definite=Def|PronType=Art 12 det 12:det _ 12 Ministry Ministry PROPN NNP Number=Sing 9 nmod 9:nmod:of _ 13 of of ADP IN _ 15 case 15:case _ 14 the the DET DT Definite=Def|PronType=Art 15 det 15:det _ 15 Interior Interior PROPN NNP Number=Sing 12 nmod 12:nmod:of SpaceAfter=No 16 ! ! PUNCT . _ 6 punct 6:punct _ cCst|d}t|ddd }|tWdn1swYtd|\}}t|}gd}||ks8J|tdd |DksEJdS) z Test a couple aspects of building a lexicon from training data expected number of words eliminated for being too long duplicate words counted once numbers eliminated z train.conlluwzutf-8)encodingNen_test)z'd announcedbaghdadbeingbustedbycellsdparinterioriraqiministryof officials operatingrun terroristthatthethemtheyzthey'dthreetwoupwerecss|]}t|VqdSrx)r)rrrrr sz2test_lexicon_from_training_data..)stropenwrite TRAIN_DATArcreate_lexiconsortedmax)tmp_path conllu_filefoutlexicon num_dict_featexpected_lexiconrrrtest_lexicon_from_training_datas   r)__doc__r=stanzar stanza.testsstanza.models.commonrstanza.models.tokenizationrrmarktravisrU pytestmarkrr'r@r[rrrlstriprrrrrrs(     ) +