o h?P@sddlZddlZddlZddlZddlZddlZddlZddlmZddl m Z m Z m Z m Z mZmZddlmZddlmZddlmZmZmZddlmZddlmmZddlmZe d Z!Gd d d eZ"Gd d d ej#Z$d6ddZ%ddZ&d7ddZ'd6ddZ(ddZ)ddZ*ddZ+ddZ,ddefddZ-efd d!Z.d"efd#d$Z/d%d&Z0d'd(Z1d)d*Z2d+d,Z3d8d.d/Z4d"efd0d1Z5d"efd2d3Z6d"efd4d5Z7dS)9N)Enum)default_charlms lemma_charlms pos_charlmsdepparse_charlms TRANSFORMERSTRANSFORMER_LAYERS)treebank_to_short_name) ud_scores)downloadDEFAULT_MODEL_DIRUnknownLanguageError)common)conll18_ud_evalstanzac@seZdZdZdZdZdZdS)ModeN)__name__ __module__ __qualname__TRAIN SCORE_DEV SCORE_TEST SCORE_TRAINrrW/var/www/html/env_mimamsha/lib/python3.10/site-packages/stanza/utils/training/common.pyrs rcs6eZdZfddZdfdd ZfddZZS) ArgumentParserWithExtraHelpcstj|i|||_dSN)super__init__ sub_argparse)selfr#argskwargs __class__rrr"s z$ArgumentParserWithExtraHelp.__init__Ncstj|ddS)N)file)r! print_help)r$r)r'rrr*!z&ArgumentParserWithExtraHelp.print_helpcst}|jdur@|jd}d}t|D]\}}|dr&|}q|dkr2|s2|}nq|dd||d}|S)N zusage:rz model arguments:)r! format_helpr#split enumeratestrip startswithjoin)r$ help_textsub_text first_lineline_idxliner'rrr.$s  z'ArgumentParserWithExtraHelp.format_helpr )rrrr"r*r. __classcell__rrr'rrs rcCst|tjd}|jdddddd|jdtd d d |jd d tjdtjdd|jdd dtjdd|jdd dtjdd|jdd dtj dd|jdtddd|jdtddd|jdddd d!|jd"ddd#d!|jd$d%ddd&d'|S)(N)r#formatter_classz --save_output temp_outputT store_falsez1Save output - default is to use a temp directory.)destdefaultactionhelp treebanks+zEWhich treebanks to run on. Use all_ud or ud_all for all UD treebanks)typenargsr@z--trainmode store_constzRun in train mode)r=r>r?constr@z --score_devzScore the dev setr=r?rGr@z --score_testzScore the test setz --score_trainzNScore the train set as a test set. Currently only implemented for some models --save_dirzGRoot dir for saving models. If set, will override the model's default.)rCr>r@ --save_namezHBase name for saving models. If set, will override the model's default.z --charlm_only store_trueFz:When asking for ud_all, filter the ones which have charlms)r?r>r@z--transformer_onlyzPWhen asking for ud_all, filter the ones for languages where we have transformersz--forceforcezRetrain existing models)r=r?r>r@) rargparseArgumentDefaultsHelpFormatter add_argumentstrrrrrr)r#parserrrrbuild_argparse3srRcCs*|jddtdd|jdddddd dS) N--charlmr>zWhich charlm to run on. Will use the default charlm for this language/model if not set. Set to None to turn off charlm for languages with a default charlm)r>rCr@z --no_charlmcharlmrFzCDon't use a charlm, even if one is used by default for this packagerH)rOrP)rQrrradd_charlm_argsIsrUc s|durtddtjtjdd}n tdd|t}t|} |dur1|| dtjvrPtjd} tj| dd} | tjd| } n| j |d\} } | j rc| d| j gt |rk|| }| j} g}| jD]S}|dr|dd }|d vrt|d }dur| jrtd fd d|D}| jrtddd|D}td|d|| |qs||qst|D]\}}|dkrtdt|}td||fg}|dkr| jr| j}t|dkrtj|\}}d||f}tj||}td|||d|g}n|dur3d||f}td||d|g}ng}| tj kr| j!s|durL|||| | }n"| j rYtj| j |}ntjd|}| d|gtj||}|durtntj"|rtd||fqtd||f| j#r|dkrt$%}|| ||||j&| | |Wdn 1swYq|| |||d| | |qdS)a A main program for each of the run_xyz scripts It collects the arguments and runs the main method for each dataset provided. It also tries to look for an existing model and not overwrite it unless --force is provided model_name can be a callable expecting the args - the charlm, for example, needs this feature, since it makes both forward and backward models NzTraining program called with:  rz --extra_args)r%rI/r-)ud_allall_udUDBASEzLFiltering ud_all treebanks to only those which can use charlm for this modelcs2g|]}gt|dddRdur|qS)_rr>N)r r/.0xchoose_charlm_methodrr s"zmain..zSFiltering ud_all treebanks to only those which can use a transformer for this modelcSs&g|]}t|ddtvr|qS)r[r)r r/rr\rrrras&zExpanding %s to %srz)=========================================z%s: %setez%s_%sz!Save file for %s model for %s: %srJz%s_%s.ptzSave file for %s model: %s saved_modelsz%s: %s exists, skipping!z)%s: %s does not exist, training new model)'loggerinfor3sysargv default_pathsget_default_pathsrRindex parse_argsparse_known_argssave_dirextendcallablerErAendswithlowerrget_ud_treebanks charlm_onlytransformer_onlyappendr0r debug save_namelenospathr/rrrLexistsr;tempfileNamedTemporaryFilename) run_treebank model_dir model_nameadd_specific_argsr#build_model_filenamer`r%pathsrQidx extra_args command_argsrErAtreebank ud_treebanks treebank_idx short_namesave_name_argsrw save_name_dirsave_name_filename model_pathrmtemp_output_filerr_rmainMs                         rcst|||durtjddddSfdd|D}tdtdd |Dd fd d |D}d fd d |D}|d |S)z Wrapper for lemma scorer. NTF)verbosecountsenhancedcsg|]}|jqSr)f1)r]key) evaluationrrrasz#run_eval_script..css|]}t|VqdSr )rxr]errr sz"run_eval_script..rVc3s|] }d|VqdS)z{:>%d}Nformatrmax_lenrrrsc3s"|] }dd|VqdS)z{:%d.2f}dNrr\rrrrs r,)r ud_evalbuild_evaluation_tablemaxr3)gold_conllu_filesystem_conllu_fileevalsresults evals_stringresults_stringr)rrrrun_eval_scripts  rcCt||gddS)N)Tokens SentencesWordsrr eval_gold eval_predrrrrun_eval_script_tokensr+rcCst||dgdS)Nrrrrrrrrun_eval_script_mwtsrcCr)N)UPOSXPOSUFeatsAllTagsrrrrrrrun_eval_script_posr+rcCr)N)UASLASCLASMLASBLEXrrrrrrrun_eval_script_depparser+rc Cs||d}|dur|dur||i||}|dur{d|||}tj|sGtd|z t|dd|i|dWn tyFYnwtj|r{|durp|durp||vrp|||vrptd|d|d|d|Std |d|Sd ||}t |}t |d krt d |d zt||dWnty} z t d|d| d} ~ wwt |}t |d krt d|d|dt |dkrt d|d|d } td| d| S)Nz{}/{}/pretrain/{}.ptz5Default pretrain should be {} Attempting to downloadpretrainlangpackage processorsrzUsing default pretrain for :z , found in z> To use a different pretrain, specify --wordvec_pretrain_filez.Using default pretrain for language, found in z{}/{}/pretrain/*.ptrz-Cannot figure out which pretrain to use for 'z;'. Will download the default package and hope for the best)rrzCannot find any pretrains in z No pretrains in the system for this language. Please prepare an embedding as a .pt and use --wordvec_pretrain_file to specify a .pt file to usez Try 'stanza.download("zY")' to get a default pretrain or use --wordvec_pretrain_file to specify a .pt file to userz%Too many pretrains to choose from in z9 Must specify an exact path to a --wordvec_pretrain_filezUsing pretrain found in ) getrryrzr{rdrer r globrxwarningFileNotFoundError) languagedefault_pretrainsdataset_pretrainsdatasetr default_ptdefault_pt_path pretrain_path pretrainsrptrrrfind_wordvec_pretrainsH    $      rc Csd|||}tj|rtd|d|d|Sd||||}tj|r7td|d|d|Sz$t|d|d|i|dtj|rZtd |d|d|WSWntyy}ztd |d |d |d |d |d}~wwtd |d |d |d |d )z Return the path to the forward or backward charlm if it exists for the given package If we can figure out the package, but can't find it anywhere, we try to download it z&saved_models/charlm/{}_{}_{}_charlm.ptz Using model z for z charlmz{}/{}/{}_charlm/{}.ptN_charlmrzDownloaded model, using model z Cannot find z charlm in either z or z Attempted downloading z but that did not work) rryrzr{rdrer ValueErrorr) directionrrTr saved_path resource_pathrrrrfind_charlm_file s&   $"rTc Cs|rztd|||d}td|||d}WnTtyh}zH||dr]|t|dd}ztd|||d}td|||d}WntyT}z td||f|d}~wwtd||nWYd}~nd}~wwd |d |g} |ss| Sd d |d|g| SgS) z? If specified, return forward and backward charlm args forward)rbackwardr[rNzYTried to find charlm %s, which doesn't exist. Also tried %s, but didn't find that eitherzFWas asked to find charlm %s, which does not exist. Did find %s thoughz--charlm_forward_filez--charlm_backward_filerSz--charlm_shorthand)rrr2rxrdr) rrT base_argsrrrr short_charlme2 char_argsrrrbuild_charlm_args's< rcCsX||d}||i|d}|durdS|dkr|S|||ivr&|S|r*|SdS)s charlm == "default" means the default charlm for this dataset or language charlm == None is no charlm Nr>)r)rrrTlanguage_charlmsdataset_charlmsdefault_charlmspecific_charlmrrr choose_charlmGs rcCt|||ttSr)rrrshort_languagerrTrrrchoose_pos_charlm\rcCrr)rrrrrrrchoose_depparse_charlmcrrcCrr)rrrrrrrchoose_lemma_charlmjrrFcCsvg}|dur9|jr9d|vr9|tvr0dt|g}|r.|tvr.d|vr.|dtt|g|S|r9td||S)zJ Choose a transformer using the default options for this language Nz --bert_modelz--bert_hidden_layerszXTransformer requested, but no default transformer for %s Specify one using --bert_model)use_bertrrrrnrPrderror)rrrwarnlayers bert_argsrrrchoose_transformerqsrcCt|||}t||||}|Sr )rrrrrTrr charlm_argsrrrbuild_pos_charlm_args rcCrr )rrrrrrbuild_lemma_charlm_argsrrcCrr )rrrrrrbuild_depparse_charlm_argsrrr )NNNNN)TF)8rMrloggingrypathlibrfr|enumr!stanza.resources.default_packagesrrrrrrstanza.models.common.constantr stanza.models.common.utilsr stanza.resources.commonr r r stanza.utils.datasetsrstanza.utils.default_pathsutilsrh stanza.utilsrr getLoggerrdrArgumentParserrrRrUrrrrrrrrrrrrrrrrrrrrrsJ          v  0