o –hÇâã@sndZddlZddlZddlZddlZddlZddlZddlZddlZddl m Z ddl m Z ddl mmmZddl mZmZmZmZmZmZddlmmmmZddlmmmmZddlmmmmZddlmmmmZddl mmmm!Z!ddl"mmmm#Z#dd„Z$dzd d „Z%d d „Z&d d„Z'd{dd„Z(d{dd„Z)d|dd„Z*dd„Z+e ,d¡Z-d}dd„Z.d~dd„Z/dd „Z0d!d"„Z1gd#¢Z2e ,d$d% 3e2¡d&d% 3e2¡d'¡Z4gd(¢Z5gd)¢Z6dd+d,„Z7d-d.„Z8e ,d/¡Z9d0d1„Z:d€d3d4„Z;dd6d7„Zdd?„Z@d@dA„ZAdBdC„ZBdDdE„ZCdFdG„ZDdHdI„ZEdJdK„ZFdLdM„ZGdNdO„ZHdPdQ„ZIdRdS„ZJdTdU„ZKdVdW„ZLdXdY„ZMdZd[„ZNd\d]„ZOd^d_„ZPd`da„ZQeEeOePeQeCeNdbœZReHeGeIdcœZSddde„ZTdfZUdgdh„ZVdidj„ZWdkdl„ZXd‚dmdn„ZYd{dodp„ZZd2Z[dqdr„Z\dsdt„Z]dudv„Z^dwdx„Z_e`dykrµe_ƒdSdS)ƒaƒ Prepares train, dev, test for a treebank For example, do python -m stanza.utils.datasets.prepare_tokenizer_treebank TREEBANK such as python -m stanza.utils.datasets.prepare_tokenizer_treebank UD_English-EWT and it will prepare each of train, dev, test There are macros for preparing all of the UD treebanks at once: python -m stanza.utils.datasets.prepare_tokenizer_treebank ud_all python -m stanza.utils.datasets.prepare_tokenizer_treebank all_ud Both are present because I kept forgetting which was the correct one There are a few special case handlings of treebanks in this file: - all Vietnamese treebanks have special post-processing to handle some of the difficult spacing issues in Vietnamese text - treebanks with train and test but no dev split have the train data randomly split into two pieces - however, instead of splitting very tiny treebanks, we skip those éN)ÚCounter)Útreebank_to_short_name)Úread_sentences_from_conlluÚwrite_sentences_to_conlluÚwrite_sentences_to_fileÚINT_REÚMWT_REÚMWT_OR_COPY_REcCsR|›d|›d|›d}|›d|›d|›d}td||fƒt|ƒ}t||ƒdS)Nú/Ú.ú.conlluzCopying from %s to %s)Úprintrr)Ú tokenizer_dirÚtokenizer_fileÚdest_dirÚ dest_fileÚ short_nameÚoriginalÚcopiedÚsents©rúk/var/www/html/env_mimamsha/lib/python3.10/site-packages/stanza/utils/datasets/prepare_tokenizer_treebank.pyÚcopy_conllu_file-s rTc Cs&tj|ddt|ƒ}| d¡d}t ¡s}t|ƒ}||d<t ¡} || _ d| _ t |||| ƒtj|dd|dur=t }||d|d |ƒ||d |d |ƒ||d |d |ƒ|t jjury|t jjurt |d |d |ƒt |d |d |ƒWdƒdSWdƒdSWdƒdS1sŒwYdS)z— This utility method copies only the conllu files to the given destination directory. Both POS, lemma, and depparse annotators need this. T©Úexist_okÚ_rÚTOKENIZE_DATA_DIRFNz train.goldztrain.inzdev.goldzdev.inz test.goldztest.in)ÚosÚmakedirsrÚsplitÚtempfileÚTemporaryDirectoryÚdictÚargparseÚ NamespaceÚaugmentÚprepare_labelsÚprocess_treebankrÚcommonÚ ModelTypeÚPOSÚDEPPARSE) ÚtreebankÚ model_typeÚpathsrÚ postprocessr%rÚshort_languagerÚargsrrrÚcopy_conllu_treebank7s0 ëí"ír2c Cs¨t d¡t|ƒ}t |¡tt|ƒtƒ}|dksJdƒ‚t|ƒ|}|d|…}||d…}td|||fƒtt|ƒ›d|›d|›dƒt||ƒt||ƒdS) NéÒéz"Dev sentence number less than one.zPTrain/dev split not present. Randomly splitting train file from %s to %s and %sz total sentences found: z in train, z in devT) ÚrandomÚseedrÚshuffleÚintÚlenÚXV_RATIOr r) r,Útrain_input_conlluÚtrain_output_conlluÚdev_output_conllurÚn_devÚn_trainÚ dev_sentsÚ train_sentsrrrÚsplit_train_fileYs       rBcCs8|r|dkrdS|dkrdS| d¡}tdd„|DƒƒS)NrFú SpaceAfter=NoTú|css|]}|dkVqdS)rCNr)Ú.0ÚtrrrÚ xs€z%has_space_after_no..)rÚany)ÚpieceÚtagsrrrÚhas_space_after_nors  rKcCs`|dks|dkr d}|S| d¡r| dd¡}|S| d¡dkr(| dd¡}|S|r.tdƒ‚|S)zº Removes a SpaceAfter=No annotation from a single piece of a single word. In other words, given a list of conll lines, first call split(" "), then call this on the -1 column rCú|SpaceAfter=NorzSpaceAfter=No|Úrz5Could not find SpaceAfter=No in the given notes field)Ú startswithÚreplaceÚfindÚ ValueError)rIÚfail_if_missingrrrÚremove_space_after_no{s ú ü þrScCs(|dkrdS|rt|ƒrtdƒ‚|dS)NrrCz1Given notes field already contained SpaceAfter=NorL)rKrQ)rIÚ fail_if_foundrrrÚadd_space_after_no‹s rUçš™™™™™©?cCstg}|D]±}t|ƒdkrtdƒ‚d}|d d¡r#|d d¡r#d}n.|d d¡r4|d  d¡r4d }n|d d ¡rE|d  d¡rEd }n td |d ¡d ƒ‚t ¡|krXq||d dvrµ||ddvrµt|d ¡d ƒrµt|d  ¡d ƒd krµt|ƒ}||dd …d||d ||<|d d¡}t|d ƒ|d <d |¡|d<||ks°J‚|  |¡q||S)a§ Basic Arabic tokenizer gets the trailing punctuation wrong if there is a blank space. Reason seems to be that there are almost no examples of "text ." in the dataset. This function augments the Arabic-PADT dataset with a few such examples. TODO: it may very well be that a lot of tokeners have this problem. Also, there are a few examples in UD2.7 which are apparently headlines where there is a ' . ' in the middle of the text. According to an Arabic speaking labmate, the sentences are headlines which could be reasonably split into two items. Having them as one item is quite confusing and possibly incorrect, but such is life. éz"Read a surprisingly short sentenceNrz# newdocéú# textz# newparéz # sent_idr4zCould not find text line in %séÿÿÿÿ)r õØŸú?ú!éþÿÿÿ)r r\r]r^ú r`ú ) r9rQrNrr5rKÚlistrSÚjoinÚappend)rÚratioÚ new_sentsÚsentenceÚ text_lineÚnew_sentÚpiecesrrrÚaugment_arabic_padt•s8  ÿ$  €rkcCs&g}|D] }|d d¡std|dƒ‚|d d¡s%td|dƒ‚|d d¡s4|dd d vr5q|dd d vra|dd d kra|dd d…dkra|ddd…dkratd|dƒ‚t ¡dkr˜t|ƒ}|ddd …|dd |d<|ddd …|dd |d<|d d|d <| |¡|d d¡dkrt ¡dkrt|ƒ}|d d¡}|dd|d…|d|d…|d<|d d¡}|dd|d…|d|d…|d<t|ƒD]!\}}|dkrðqç| d¡ddkr||dd||d<nqç| |¡q||S)aÜ Add a few sentences with modified punctuation to Telugu_MTG The Telugu-MTG dataset has punctuation separated from the text in almost all cases, which makes the tokenizer not learn how to process that correctly. All of the Telugu sentences end with their sentence final punctuation being separated. Furthermore, all commas are separated. We change that on some subset of the sentences to make the tools more generalizable on wild text. r4rYz3Expected the second line of %s to start with # textrrZz # translitz7Expected the second line of %s to start with # translitz. . .r[)r r]r^r_r`éýÿÿÿNz ..éüÿÿÿz ...z¨Sentence %s does not end with space-punctuation, which is against our assumptions for the te_mtg treebank. Please check the augment method to see if it is still neededçš™™™™™¹?rLú,rWra) rNrQÚendswithr5rbrdrPÚ enumerater)rrfrgÚ new_sentenceÚindexÚidxÚwordrrrÚaugment_teluguÄs@ H    "((þ €rvz ([a-zA-Z]+)[,] ([a-zA-Z]+) 縅ëQ¸ž?cCsàg}|D]ß}t|ƒD] \}}| d¡rnq qt ||¡}|rãt ¡|krãt|ƒD]6\}}| d¡r4q*| d¡d| d¡krAq*||d d¡ddkrOq*||d d¡d| d¡kr`q*|t|ƒdkrjq||d} |  d¡} | ddks}J‚t| dƒ| d<d  | ¡} |d|d…| g||dd…} ||  | d¡d | d¡¡} t| d¡d | d¡ƒ} ||d| …| d¡d| d¡||| | d…}|| |<|  | ¡qt d t|ƒƒ||S) a9Find some fraction of the sentences which match "asdf, zzzz" and squish them to "asdf,zzzz" This leaves the tokens and all of the other data the same. The only change made is to change SpaceAfter=No for the "," token and adjust the #text line, with the assumption that the conllu->txt conversion will correctly handle this change. This was particularly an issue for Spanish-AnCora, but it's reasonable to think it could happen to any dataset. Currently this just operates on commas and ascii letters to avoid accidentally squishing anything that shouldn't be squished. UD_Spanish-AnCora 2.7 had a problem is with this sentence: # orig_file_sentence 143#5 In this sentence, there was a comma smashed next to a token. Fixing just this one sentence is not sufficient to tokenize "asdf,zzzz" as desired, so we also augment by some fraction where we have squished "asdf, zzzz" into "asdf,zzzz". This exact example was later fixed in UD 2.8, but it should still potentially be useful for compensating for typos. rYú#rar4rorZr[Nú, z3Added %d new sentences with asdf, zzzz -> asdf,zzzz) rqrNÚCOMMA_SEPARATED_REÚsearchr5rÚgroupr9rUrcrPrdr )rrerfrgÚtext_idxrhÚmatchrtruÚcommarjriÚ text_offsetÚtext_lenÚnew_textrrrÚaugment_comma_separationsòsF ÿ     &"< €rƒç{®Gáz”?cCs.g}d}|D]}t ¡|kr| |¡qd}t|ƒD]X\}}| d¡r%q|dks1|t|ƒdkr2q| d¡}|ddkrst|dƒss||d} t|  d¡dƒsSq||d} t |  d¡d¡rdqt |  d¡d¡roqd }nq|s|| |¡qt |ƒ} | | d¡}t |dƒ|d<d  |¡| |<| |d d¡}|d} t |dƒ|d<d  |¡| |d<| |d d¡d} t|ƒD]A\} } |  d ¡r| d | }| d | }|   |¡}|dkrìtd |d| ƒ‚| d|…|| |t|ƒd…}|| | <nqÃ| | ¡|d}qtd|ƒ|S)al Move the comma from after a word to before the next word some fraction of the time We looks for this exact pattern: w1, w2 and replace it with w1 ,w2 The idea is that this is a relatively common typo, but the tool won't learn how to tokenize it without some help. Note that this modification replaces the original text. rFrxrZrar4ror[TrYryz ,z[Unexpected #text line which did not contain the original text to be modified. Looking for Ú Nz&Swapped 'w1, w2' for 'w1 ,w2' %d times)r5rdrqrNr9rrKr r~rbrUrcrSrPÚ RuntimeErrorr )rrerfÚnum_operationsrgÚfoundÚword_idxrurjÚ prev_wordÚ next_wordrrr}rhÚ old_chunkÚ new_chunkÚ new_text_linerrrÚaugment_move_comma9sh       €     $ø  rc Cs2d}d}t|ƒD]8\}}t|ƒdkrtd|ƒ‚|D]}| d¡r5| d¡dkr*d}| d¡dkr3d}n qtd||dfƒ‚q|sE|sG|Sg}|D]K}t ¡d krY| |¡qKg}|D]3}| d¡rn| | dd¡¡q]| d ¡ry| |¡q]|  d ¡}|d  dd¡|d <| d   |¡¡q]| |¡qK|S) uå If there are no instances of ’ in the dataset, but there are instances of ', we replace some fraction of ' with ’ so that the tokenizer will recognize it. # TODO: we could do it the other way around as well Frz$Got a blank sentence in position %d!rYú'Tu’z5Cannot find '# text' in sentences %d. First line: %srVrxrar4) rqr9ÚAssertionErrorrNrPrQr5rdrOrrc) rÚhas_unicode_aposÚhas_ascii_aposÚsent_idxÚsentÚlinerfrirjrrrÚ augment_apossB   ûþ       r—c Cs d}d}|D]#}|D]}| d¡rq | d¡}|ddkr d}q |ddkr(d}q q|s.|s0|Sg}d}|D]F}t ¡d krD| |¡q6d}g} |D]&}| d¡rW|  |¡qJ| d¡}|ddkrhd|d<d}|  d |¡¡qJ| | ¡|r||d}q6td |ƒ|S) u1 Replaces a fraction of '...' with '…' Frxrar4z...Tu…rrnz2Changed %d sentences to use fancy unicode ellipses)rNrr5rdrcr ) rÚ has_ellipsesÚhas_unicode_ellipsesr•r–rjrfÚ num_updatedrˆrirrrÚaugment_ellipsesºsJ    €ù        € r›) ú"õ“õâ€õ«õ»õ「õã€õ《õ》õ„õ″z(.?)[rMz](.+)[z](.?)) rœrržrŸr r¡r£r¥r¥r¦) rœrržr rŸr¢r¤ržrr¦ç333333Ã?cCsvttƒttƒks J‚tƒ}g}|D]›}t ¡|kr| |¡qtdd„|Dƒƒ}|dkr2| |¡qt tttƒƒ¡}t|}t|}|||d|||<g} d} |D]3} |   d¡rb|  | ¡qU|   d¡} | dt vrƒ| rt|| d<n|| d<d} |  d  | ¡¡qU|  | ¡qUt | ƒD]\} }|  d ¡r¦d ||f}t ||¡}|| | <q| | ¡qtd  t| ¡ƒ|¡ƒ|S) uÓ Go through the sentences and replace a fraction of sentences with alternate quotes TODO: for certain languages we may want to make some language-specific changes eg Danish, don't add «...» css0|]}| d¡s| d¡dtvrdVqdS)rxrar4N)rNrÚQUOTES©rEÚxrrrrGÿs€ÿþþz!augment_quotes..rZr4FrxraTrYz \1%s\2%s\3zAugmented {} quotes: {})r9Ú START_QUOTESÚ END_QUOTESrr5rdÚsumÚchoiceÚrangerNrr¨rcrqÚ QUOTES_REÚsubr ÚformatÚvalues)rreÚcountsrfr•Ú count_quotesÚ quote_idxÚ start_quoteÚ end_quoteriÚ saw_startr–rjr}rhÚ replacementrŽrrrÚaugment_quotesísL            € r»cCs(t|ƒD] \}}| d¡r|SqdS)z3 Return the index of the # text line or -1 rYr[)rqrN)rgrtr–rrrÚ find_text_idx-s  ÿr¼z[0-9]cCs| d¡r|S| d¡}t |d¡r5|d d¡}dt|dƒ|t|dƒ|f|d<d |¡}|St |d¡rV|djddd}d t|dƒ||df|d<nt |d¡setd |dƒ‚t t|dƒ|ƒ|d<|d d kr‹t|d ƒ}|dkr‹t t|ƒ|ƒ|d <|d d krÁ|d jddd}t   |d¡r§td|ƒ‚t|dƒdkrÁt t|dƒ|ƒd|d|d <d |¡}|S)zf Adjust all indices in the given sentence by delta. Useful when removing a word, for example rxrarú-z%d-%dr4r )Úmaxsplitz%d.%szUnknown index type: %séréú:z+Need to handle multiple additional deps: %s) rNrrr~r8rcr rÚNotImplementedErrorÚstrÚDIGIT_REr{)r–ÚdeltarjÚindicesÚ index_piecesÚdepÚ dep_piecesrrrÚchange_indices8s4  ( "    $ rÊçš™™™™™É?c Csg}|D]m}t ¡|kr qt|ƒ}||}| d¡dkrqt|ƒD] \}}| d¡r+q!|t|ƒdkr8tdƒ‚| d¡}|ddkrDqt|dƒrMd} nd} |d|…||dd…} |  | d ¡| |<d d „| Dƒ} |  | ¡qt|ƒd kr€t d t|ƒƒ||S)uþ If a sentence starts with certain punct marks, occasionally use the same sentence without the initial punct. Currently this just handles ¿ This helps languages such as CA and ES where the models go awry when the initial ¿ is missing. õ¿r4rxz+Unexpectedly an entire sentence is commentsrar[u¿ NrMcSsg|]}t|dƒ‘qS)r[)rÊr©rrrÚ €sz)augment_initial_punct..ru.Added %d sentences with the leading ¿ removed) r5r¼ÚcountrqrNr9rQrrKrOrdr ) rrerfr•r}rhrtr–rjÚ replace_textrirrrÚaugment_initial_punct[s6       rÐrnc Cs:g}|D]}t|ƒ}||}| d¡dks| d¡dkr |Sq|D]g}t ¡|kr,q#t|ƒ}||}| d¡dkrC| d¡dkrCq#| dd¡ dd¡}t|ƒ}|||<t|ƒD]+\}}| d¡rcqY| d¡} | ddkrsd| d<n | ddkr}d| d<d | ¡||<qY|  |¡q#t |ƒdkr™t d t |ƒƒ||S) zF If there are no sentences with [], transform some () into [] ú[rú]ú(ú)rxrar4zddg}|tjjur| ddg¡td |d |¡¡ƒg}|D]}tj|||d d d }| t|ƒ¡q(|St d||d ¡}t|ƒ}|S) NÚUDBASErézUD_Italian-ISDTzUD_Italian-VITzUD_Italian-TWITTIROzUD_Italian-PoSTWITAzBuilding {} dataset out of {}r`rÞT©Úfail) r(r)Ú TOKENIZERÚextendr r²rcràr) r.r-râráÚ treebanksrr,Ú conllu_fileÚ istd_conllurrrÚbuild_combined_italian_datasets&þ þýröcCs&t d|dd¡}t |¡rtdƒ‚dS)NúUD_English-GUMRedditrérÞzìCannot process UD_English-GUMReddit in its current form. There should be a download script available in the directory which will help integrate the missing proprietary values. Please run that script to update the data, then try again.)r(ràÚmostly_underscoresrQ)ráÚ gum_conllurrrÚcheck_gum_ready-s ÿrúc CsÔ|d}t|ƒ|dkr\gd¢}ddg}g}|D]}tj||dddd}t|ƒ} td t| ƒ|fƒ| | ¡q|D]}tj||d ddd}t|ƒ} td t| ƒ|fƒ| | ¡q:|St d ||d¡} t| ƒ}|S) zJ en_combined is currently EWT, GUM, PUD, Pronouns, and handparsed rîré)úUD_English-EWTúUD_English-GUMr÷zUD_English-PUDzUD_English-PronounsrÞTrïúRead %d sentences from %srërû)rúr(ràrr r9rò) r.r-râráÚtrain_treebanksÚtest_treebanksrr,rôrfÚ ewt_conllurrrÚbuild_combined_english_dataset2s(  ýrc Csàg}|D]i}d}d}d}|D]+}| d¡rq| d¡}t |d¡r#q|ddkr-|d}t|dƒ}|dd k}q|sht|ƒ}|d  d¡}t|d ƒ|d <d |¡|d <| d |d ||f¡| |¡q| |¡q|S) a Add a period to the end of a sentence with no punct at the end. The next-to-last word has SpaceAfter=No added as well. Possibly English-specific because of the xpos. Could be upgraded to handle multiple languages by passing in the xpos as an argument NFrxrarr¿Ú0rXÚPUNCTr[z$%d . . PUNCT . _ %s punct %s:punct _r4) rNrr r~r8rbrUrcrd) Úhandparsed_sentencesrfr•Úroot_idÚmax_idÚ last_punctr–rjrirrrÚ&add_english_sentence_final_punctuationQs0       rcCsZ|d}g}|dkr+|tjjur+tj |dd¡}t|ƒ}tdt|ƒ|fƒ|  |¡|S)z[ Extra sentences we don't want augmented for French - currently, handparsed lemmas ÚHANDPARSED_DIRréz french-lemmaszfr_lemmas.conlluúLoaded %d sentences from %s) r(r)ÚLEMMArÚpathrcrr r9rò©r.r-râÚhandparsed_dirrÚhandparsed_pathrrrrÚ#build_extra_combined_french_datasetts  rcCs˜|d}g}|dkrJtj |dd¡}t|ƒ}t|ƒ}| |¡tdt|ƒ|fƒ|tj j urJtj |dd¡}t|ƒ}tdt|ƒ|fƒ| |¡|S)z1 Extra sentences we don't want augmented r rézenglish-handparsedz english.conllr zenglish-lemmaszen_lemmas.conllu) rr rcrrròr r9r(r)r r rrrÚ$build_extra_combined_english_datasetƒs   rcCs¤|d}|dkr gStj |dd¡}tj |¡std |¡ƒ‚t|ƒ}|D] }|d d¡r5t  |d¡s9t dƒ‚|dd d …d |d<q%t d t |ƒ|fƒ|S) z/ Extra data - the MWT data for Italian r réz italian-mwtz italian.mwtzgCannot find the extra dataset 'italian.mwt' which includes various multi-words retokenized, expected {}rZrzhUnexpected format of the italian.mwt file. Has it already be modified to have SpaceAfter=No everywhere?Nr[rCr ) rr rcÚexistsÚFileNotFoundErrorr²rrprr~r‘r r9)r.r-rârÚ extra_italianÚ extra_sentsrgrrrÚ$build_extra_combined_italian_dataset—s rcCsªg}d}|D]F}t|ƒD] \}}| d¡rnq tdƒ‚| d¡s'| |¡q|d}t|ƒ}|dd…d||<|d dd¡|d<|d}| |¡qtd |ƒ|S) af Spanish GSD and AnCora have different standards for semicolons. GSD has semicolons at the end of sentences, AnCora has them in the middle as clause separators. Consecutive sentences in GSD do not seem to be related, so there is no combining that can be done. The easiest solution is to replace sentence final semicolons with "." in GSD rrYz5Expected every sentence in GSD to have a # text fieldú;r4Nr[r z7Updated %d sentences to replace sentence-final ; with .)rqrNrQrprdrbrOr )Ú sentencesrfrÎrgr}rhrirrrÚreplace_semicolons«s& ÿ    rcCsbg}|D]*}g}|D]}| d¡r| |¡q | d¡}d||<| d |¡¡q | |¡q|S)z Removes a specified column from the given dataset Particularly useful when mixing two different POS formalisms in the same tagger rxrar)rNrdrrc)rÚcolumnrfrgrirurjrrrÚ strip_columnÇs    rcCó t|dƒS)z† Removes all xpos from the given dataset Particularly useful when mixing two different POS formalisms in the same tagger rW©r©rrrrÚ strip_xposÚó rcCr)zŠ Removes all features from the given dataset Particularly useful when mixing two different POS formalisms in the same tagger érrrrrÚ strip_featsâr r"c Cs`|d}|d}|d}ddg}|dkrR|tjjkrRi}tj|d|ddd d }t|ƒ} | ||d<tj|d |d dd d }t|ƒ} t| ƒ} t| ƒ} | ||d <|S|dkrž|tjjuržg} tj|d|ddd d }t|ƒ} td t | ƒ|fƒ|   | ¡tj|d |d dd d }t|ƒ} td t | ƒ|fƒ|   | ¡| Stj|d||dd d }t|ƒ} | S)zI sq_combined is STAF as the base, with TSA added for some things rîÚ UDBASE_GITr zUD_Albanian-STAFzUD_Albanian-TSArérrÞTrïr4rërý) r(r)r*ràrrr"r+r r9rò) r.r-râráÚudbase_git_dirrróÚ documentsrôrfrrrrÚbuild_combined_albanian_datasetês:    r&c Csd|d}|d}ddg}|dkr:|tjjkr:i}|D]}tj|||ddd}t|ƒ} | d ¡s3t| ƒ} | ||<q|S|dkr¢g} |D](}tj|||ddd}t|ƒ} td t| ƒ|fƒ| d ¡ret | ƒ} |   | ¡qB|tjj tjj tjj fvr tj |d d ¡} tj | ¡std t¡ƒ‚t| ƒ} td t| ƒ| fƒ|   | ¡| Stjd||ddd}t|ƒ} | S)aô es_combined is AnCora and GSD put together For POS training, we put the different datasets into a zip file so that we can keep the conllu files separate and remove the xpos from the non-AnCora training files. It is necessary to remove the xpos because GSD and PUD both use different xpos schemes from AnCora, and the tagger can use additional data files as training data without a specific column if that column is entirely blank TODO: consider mixing in PUD? rîr zUD_Spanish-AnCorazUD_Spanish-GSDrérÞTrïÚAnCorarýÚGSDz spanish-mwtzadjectives.conlluzjCannot find the extra dataset 'handpicked.mwt' which includes various multi-words retokenized, expected {})r(r)r*ràrrprr r9rròrñÚMWTr rr rcrrr²r) r.r-rârárrór%r,rôrfrÚ extra_spanishrrrrÚbuild_combined_spanish_datasets@       ýr+c CsØ|d}|d}|dkr^gd¢}g}|D]}tj||dddd}t|ƒ} tdt| ƒ|fƒ| | ¡qtj |d d ¡} tj  | ¡sIt d   t ¡ƒ‚t| ƒ} tdt| ƒ| fƒ| | ¡|St d ||d¡} t| ƒ}|S) Nrîr ré)ú UD_French-GSDzUD_French-ParisStorieszUD_French-RhapsodiezUD_French-SequoiarÞTrïrýzfrench-handparsedzhandparsed_deps.conlluzkCannot find the extra dataset 'handparsed_deps.conllu' which includes various dependency fixes, expected {}r,) r(ràrr r9ròrr rcrrr²r) r.r-rârárrþrr,rôrfÚ extra_frenchrrãrrrÚbuild_combined_french_datasetKs(   ýr.c Cs|d}|d}ddg}|dkrqg}|D]}tj|||ddd}t|ƒ} td t| ƒ|fƒ| | ¡qtj |d ¡} tj  | ¡sGt d | ƒ‚tj | d ¡}tj  |¡s\t d | |fƒ‚t|ƒ} td t| ƒ|fƒ| | ¡|Stj|d||ddd}t|ƒ}|S)a# Combines the IAHLT treebank with an updated form of HTB where the annotation style more closes matches IAHLT Currently the updated HTB is not in UD, so you will need to clone git@github.com:IAHLT/UD_Hebrew.git to $UDBASE_GIT dev and test sets will be those from IAHLT rîr#zUD_Hebrew-IAHLTwikizUD_Hebrew-IAHLTknessetrérÞTrïrýÚ UD_HebrewzOPlease download git@github.com:IAHLT/UD_Hebrew.git to %s (based on $UDBASE_GIT)zhe_htb-ud-train.conlluz)Found %s but inexplicably there was no %sr) r(ràrr r9ròrr rcrr) r.r-rârár$rórr,rôrfÚhebrew_git_dirrrrÚbuild_combined_hebrew_datasetcs.      ýr1)Ú en_combinedÚ es_combinedÚ fr_combinedÚ he_combinedÚ it_combinedÚ sq_combined)r2r4r6c Cs€t d¡|d}t|}t |d¡}dD]¨}t |||¡}||||ƒ} t| tƒr¡|dkr@|r@t |   ¡ƒD] } t | | ƒ| | <q5t j  |¡dd} t | d¡H} t |   ¡ƒD]9} |  | dd¡'} tj| d d d }t|| | ƒWdƒn1s|wYWdƒn1s‹wYqWWdƒn1s›wYq|dkr«|r«t | ƒ} |dur¸|  ||||ƒ¡t|| ƒqdS) Nr3rrèrérz.zipÚwr zutf-8rM)ÚencodingÚnewline)r5r6Ú COMBINED_FNSÚCOMBINED_EXTRA_FNSÚgetr(rìÚ isinstancer"rbÚkeysrÖrr ÚsplitextÚzipfileÚZipFileÚopenÚioÚ TextIOWrapperrròr)r.rr-r%rÚbuild_fnÚextra_fnrârØrÚfilenameÚ output_zipÚzoutÚzfoutÚfoutrrrÚbuild_combined_dataset˜s<     ÿ€ÿ€ÿÿ€  îrM)Úen_craftÚen_geniaÚen_mimicc Cs®t d¡| d¡\}}|dksJ‚dD]@} t ||| ¡} | dkr2t||| ƒ} | dkr1|r1t| ƒ} ng} tj  |dd|  ¡d|  ¡| f¡} |   t | ƒ¡t| | ƒqd S) zw Process the en bio datasets Creates a dataset by combining the en_combined data with one of the bio sets r3rÚenrèréÚ BIO_UD_DIRz UD_English-%szen_%s-ud-%s.conlluN)r5r6rr(rìrrÖrr rcÚupperÚlowerròrr) r.rárrrr-r%ÚnameÚ bio_datasetrârØrÚbio_filerrrÚbuild_bio_dataset³s    €( örXc Cszt|ƒt d¡t |||¡}ddg}g}|D]}tj|||ddd} | t| ƒ¡q|dkr6|r6t|ƒ}t ||ƒdS) z„ Build the GUM dataset by combining GUMReddit It checks to make sure GUMReddit is filled out using the included script r3rür÷rÞTrïréN) rúr5r6r(rìràròrrÖr) rárrrâr%rØrórr,rôrrrÚ"build_combined_english_gum_datasetÈs  rYcCsdD] }t|||||ƒqdSrç)rY)rárrr%rârrrÚbuild_combined_english_gumÞsÿrZc CsÞ|durtj|||ddd}|durt |||¡}td||fƒ|dkr3|dkr3|r3t||tƒdS|dkrE|dkrE|rEt||tƒdS| d¡rV| d ¡rVt ||ƒdS|dkrd|rdt||t ƒdSt |ƒ} t || ƒdS) NrÞTrïz!Reading from %s and writing to %sÚte_mtgréÚar_padtÚko_rß) r(ràrìr rÚrvrkrNrprÝrÖrr) r,rárrr0râr%r×rØrrrrÚprepare_ud_datasetâs r^cCs@t|||||d|ƒt|||||d|ƒt|||||d|ƒdS)zš Process a normal UD treebank with train/dev/test splits SL-SSJ and other datasets with inline modifications all use this code path as well. rérêrëN)r^)r,rárrr0r%rrrÚprocess_ud_treebankõsr_c Cs¢t ||dd¡}t ||dd¡}t ||d¡}t ||d¡}t ||d¡} t |¡dkr8t |¡dkr8||}}t||||dsBdSt|||||dd || d dS) a… Process a UD treebank with only train/test splits For example, in UD 2.7: UD_Buryat-BDT UD_Galician-TreeGal UD_Indonesian-CSUI UD_Kazakh-KTB UD_Kurmanji-MG UD_Latin-Perseus UD_Livvi-KKPP UD_North_Sami-Giella UD_Old_Russian-RNC UD_Sanskrit-Vedic UD_Slovenian-SST UD_Upper_Sorbian-UFAL UD_Welsh-CCG rérÞrërêièiˆ)r,r;r<r=NF)r%r×rØ)r(ràrìÚnum_words_in_filerBr^) r,rárrr0r;Útest_input_conllur<r=Útest_output_conllurrrÚprocess_partial_ud_treebanks ýrccCs@|jdddddd|jddddd dt |¡t |¡dS) Nz --no_augmentÚ store_falser%Tz#Augment the dataset in various ways)ÚactionÚdestÚdefaultÚhelpz--no_prepare_labelsr&z`Prepare tokenizer and MWT labels. Expensive, but obviously necessary for training those models.)Ú add_argumentÚconvert_th_lst20Úadd_lst20_argsÚconvert_vi_vlspÚ add_vlsp_args)ÚparserrrrÚadd_specific_args*s ÿ ÿ roc Cs |d}|d}|d}t|ƒ}| d¡d}tj|dd|dkr+t |d |¡n³|d kr9t |d ||¡n¥|d krFt |d |¡n˜|d krTt  |d ||¡nŠ|dkrat  |d |¡n}|dkrnt  |d |¡np|  d¡rzt |||ƒnd|tvr‡t||||jƒnW|tvr—t|||||||jƒnG|  d¡r®td|||fƒt||||jƒn0tj||dddd} td|||fƒtj||ddddsÔt|||||ƒn t||||||jƒ|tjjusë|tjjur|dvrõt ||¡|jrt ||¡dSdSdS)a& Processes a single treebank into train, dev, test parts Includes processing for a few external tokenization datasets: vi_vlsp, th_orchid, th_best Also, there is no specific mechanism for UD_Arabic-NYUAD or similar treebanks, which need integration with LDC datsets rîrr rrTrÚmy_altÚCONSTITUENCY_BASEÚvi_vlspÚSTANZA_EXTERN_DIRÚ th_orchidÚth_lst20Úth_bestÚ ml_cochinÚ ko_combinedÚen_gumzPreparing data for %s: %s, %srérÞrïrêF)rtruN)rrrrÚconvert_my_altrlÚconvert_th_orchidÚmainrjÚconvertÚconvert_th_bestÚconvert_ml_cochinrNrír;rMr%Ú BIO_DATASETSrXr rZr(ràrcr_r)rñr)Úconvert_conllu_to_txtr&Ú!prepare_tokenizer_treebank_labels) r,r-r.r1rárrrr0Útrain_conllu_filerrrr'3sN    ûr'cCst ttjjt¡dS)N)r(r|r'r)rñrorrrrr|osr|Ú__main__)NT)T)rV)rw)r„)r§)rË)rn)TNN)aÚ__doc__r#ÚglobrDrr5Úrer rAÚ collectionsrÚstanza.models.common.constantrÚstanza.utils.datasets.commonÚutilsÚdatasetsr(rrrrrr Ú4stanza.utils.datasets.tokenization.convert_ml_cochinÚ tokenizationrÚ1stanza.utils.datasets.tokenization.convert_my_altrzÚ2stanza.utils.datasets.tokenization.convert_vi_vlsprlÚ2stanza.utils.datasets.tokenization.convert_th_bestr~Ú3stanza.utils.datasets.tokenization.convert_th_lst20rjÚ4stanza.utils.datasets.tokenization.convert_th_orchidr{rr2rBrKrSrUrkrvÚcompilerzrƒrr—r›r¨rcr°r«r¬r»r¼rÄrÊrÐrÕrÖrÚrÜrÝrærírörúrrrrrrrrr"r&r+r.r1r;r<rMr€rXrYrZr^r_r:rcror'r|Ú__name__rrrrÚs²    "  / -  GT--& @  # .)  #-4&ú ý   ( <  ÿ