o h)@s`ddlZddlmZddlZddlZddlZddlZddlZddlZddl m Z ddl m m mZddlm mZedZedZedZedZejejeddZGd d d eZGd d d eZd/ddZddZ ddZ!ddZ"ddZ#ddZ$ddZ%ddZ&ddZ'd0d"d#Z(d$d%Z)d&d'Z*d1d)d*Z+d+d,Z,d2d-d.Z-dS)3N)Enum)canonical_treebank_namestanzaz^[0-9]+[-][0-9]+z^[0-9]+[-.][0-9]+z^[0-9]+$zconllu_to_text.plc@s eZdZdZdZdZdZdZdS) ModelTypeN)__name__ __module__ __qualname__ TOKENIZERMWTPOSLEMMADEPPARSErrW/var/www/html/env_mimamsha/lib/python3.10/site-packages/stanza/utils/datasets/common.pyrs rcseZdZfddZZS)UnknownDatasetErrorcst|||_dSN)super__init__dataset)selfrtext __class__rrr$s  zUnknownDatasetError.__init__)r r r r __classcell__rrrrr#srtraindevtestcCsr|D]4}|d|d|d}|d|d|d}tj|s&td|tjdtd|d|d d qd S) z Uses the udtools perl script to convert a conllu file to txt TODO: switch to a python version to get rid of some perl dependence /. .gold.conllu.txtz-Cannot convert %s as the file cannot be foundzperl  z > T)shellN)ospathexistsFileNotFoundError subprocess check_outputCONLLU_TO_TXT_PERL) tokenizer_dir short_nameshardsr output_conllu output_txtrrrconvert_conllu_to_txt(s  "r5cCtj||d|dS)N-ud-z -mwt.jsonr)r*joinbase_dirr1rrrrmwt_name8r<cCr6)Nr$r%r8r:rrrtokenizer_conllu_name;r=r>cCs<|d|d|d}t|||}t||d|d|gdS)Nr#r7z .toklabelsz-oz-m)r<prepare_tokenizer_datamain) input_txt input_conllur0r1rlabels_filename mwt_filenamerrr prepare_tokenizer_dataset_labels>s  rEc CsxdD]7}|d|d|d}|d|d|d}z t|||||Wqttfy.td||fdS)z] Given the txt and gold.conllu files, prepare mwt and label files for train/dev/test rr#r$r&r%zFailed to convert %s to %sN)rEKeyboardInterrupt SystemExitprint)r0r1rr4r3rrr!prepare_tokenizer_treebank_labelsFsrIcCsg}g}t|dd>}|D]}|}t|dkr't|dkr&||g}q ||q t|dkr@||Wd|SWd|S1sKwY|S)zh Reads a conllu file as a list of list of strings Finding a blank line separates the lists utf-8encodingrN)openstriplenappend)filenamesentscacheinfilelinerrrread_sentences_from_conlluUs(           rVcCs g}d}d}t|D]W\}}|dr||q |d}t|dr+||q t|d}|ddkrE|ddkr?|}||q |dkrQ|}||q d |d<d |d <|d|q |dur|durwd||d<d ||d <nt|||d<d ||d <d||||<|S) a Possibly add fake dependencies in columns 6 and 7 (counting from 0) The conllu scripts need the dependencies column filled out, so in the case of models we build without dependency data, we need to add those fake dependencies in order to use the eval script etc lines: a list of strings with 10 tab separated columns comments are allowed (they will be skipped) returns: the same strings, but with fake dependencies added if columns 6 and 7 were empty N# r_0r1deproot) enumerate startswithrPsplitMWT_OR_COPY_REmatchintr9str)lines new_linesroot_idx first_idxline_idxrUpieces token_idxrrrmaybe_add_fake_dependenciesjs<           rncCs8|D]}t|}|D]}t||dq td|dqdS)N)file)rnrH)outfilerRrgrUrrrwrite_sentences_to_files rrcCs>t|ddd}t||WddS1swYdS)NwrJrK)rMrr)rQrRrqrrrwrite_sentences_to_conllus "rtFUDBASEc Cs|dr|dr|dd}tj||d|d|}t|}t|dkr5|r3td||dSt|d kr?|dSt d |d |d |d|d )a@ For a given treebank, dataset, extension, look for the exact filename to use. Sometimes the short name we use is different from the short name used by UD. For example, Norwegian or Chinese. Hence the reason to not hardcode it based on treebank set fail=True to fail if the file is not found UD_Korean_segNz*-ud-r$rzCould not find any treebank files which matched {} If you have the data elsewhere, you can change the base directory for the search by changing the {} environment variablerz$Unexpected number of files matched 'r#z/*-ud-') raendswithr)r*r9globrOr,format RuntimeError)treebank udbase_dirr extensionfailenv_varrQfilesrrrfind_treebank_dataset_files    "rcCsfd}d}t|D]"}|}|sq |drq |d}|d}|ddvr,|d}q ||dkS)z Certain treebanks have proprietary data, so the text is hidden For example: UD_Arabic-NYUAD UD_English-ESL UD_English-GUMReddit UD_Hindi_English-HIENCS UD_Japanese-BCCWJ rrWrrX)rZ-g?)rM readlinesrNrarb)rQunderscore_count total_countrUrlrrrmostly_underscoress     rcCs^d}t|}|D]}|}|sq |drq |d}q Wd|S1s(wY|S)z> Count the number of non-blank lines in a conllu file rrWrN)rMrNra) conllu_filecountfinrUrrrnum_words_in_files    rTcslttd}dd|D}dd|D}|r4fdd|D}fdd|D}fdd|D}|S)zX Looks in udbase_dir for all the treebanks which have both train, dev, and test z/UD_*cSsg|] }tj|dqS)r)r)r*rb.0trrr sz$get_ud_treebanks..cSsg|]}|dkr|qS)zUD_English-GUMRedditrrrrrrscs,g|]}t|ddrt|ddr|qS)r conllur")rrrrrrs    cs"g|] }tt|dds|qS)r r)rrrrrrrs csJg|]!}t|dds!tt|dddks!tt|dddkr|qS)r!rr ir"i)rrrrrrrs )sortedr{)rfiltered treebanksrrrget_ud_treebankssrcCst}|jdtddd|S)Nr+zEWhich treebanks to run on. Use all_ud or ud_all for all UD treebanks)typenargshelp)argparseArgumentParser add_argumentrf)parserrrrbuild_argparsesrc Cstddtjt}|dur|||}t}g}|j D]}| dvr7t |d}| |q#t |}||q#|D] }|||||qCdS)NzDatasets program called with: r')ud_allall_udru)loggerinfor9sysargvr parse_args default_pathsget_default_pathsrlowerrextendrrP) process_treebank model_typeadd_specific_argsrargspathsrr~ ud_treebanksrrrr@ s      r@)r)Fru)Tr).renumrr{loggingr)rer-r+stanza.models.common.short_name_to_treebankr,stanza.utils.datasets.prepare_tokenizer_datautilsdatasetsr?stanza.utils.default_pathsr getLoggerrcompileMWT_RErcINT_REr*r9rb__file__r/r ValueErrorrr5r<r>rErIrVrnrrrtrrrrrr@rrrrsB       4