o h@sddlmZmZddlZddlZddlZddlZddlZddlZddlm Z ddl m Z ddl m Z mZddlmZddlmZmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddlmZddl m!Z!ddl"m#Z#ddl$m%Z%ddl&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/ddl0m1Z1m2Z2ddl3m4Z4e4Z5e6dZ7edgdZ8GdddedgdZ9ddZ:ddZ;dd Zd%d&Z?d'd(Z@d)d*ZAd+d,ZBd-d.ZCd/d0ZDd3d1d2ZEdS)4)Counter namedtupleN)nn)utils)FoundationCacheNoTransformerFoundationCache)LargeMarginInSoftmaxLoss)sort_with_indicesunsort)parse_transitions)transition_sequence) tree_reader)InOrderCompoundOracle) InOrderOracle) LSTMModel)TransitionScheme)Tree) TopDownOracle)Trainer) retag_treesbuild_optimizerbuild_schedulerverify_transitionsget_open_nodescheck_constituentscheck_root_labelsremove_duplicate_treesremove_singleton_trees)EvaluateParser ParseResult)get_tqdmzstanza.constituency.trainer TrainItem)tree gold_sequence preterminalsc@seZdZddZdS) EpochStatscCsZ|j|j}|j|j}|j|j}|j|j}|j|j}|j|j}t||||||SN)transitions_correcttransitions_incorrect repairs_usedfake_transitions_used epoch_lossnansr%)selfotherr'r(r)r*r+r,r/e/var/www/html/env_mimamsha/lib/python3.10/site-packages/stanza/models/constituency/parser_training.py__add__&s      zEpochStats.__add__N)__name__ __module__ __qualname__r1r/r/r/r0r%%s r%)r+r'r(r)r*r,cCsf|ddkr |dd}nd}t|d}|r|djnt}|d|d|d|d d }tj|||d }|d r>|t|d }t dt ||} |duro|j j } | dk} t d| |dt ||| } t d|drw|t|j | |||\} } }t d|d | | durt d|d | WddSWddS1swYdS)z Loads the given model file and tests the eval_file treebank. May retag the trees using retag_pipeline Uses a subprocess to run the Java EvalB code num_generaterNkbestwordvec_pretrain_filecharlm_forward_filecharlm_backward_filedevicer9r:r;r<)argsfoundation_cache log_shapes eval_filezRead %d trees for evaluationxpos8Retagging trees using the %s tags from the %s package... retag_packageRetagging finished log_normszF1 score on %s: %fzKBest F1 score on %s: %f)rr?rrloadr@r read_treebanktloggerinfolenmodel retag_methodrrF run_dev_set)r> model_fileretag_pipeliner8 evaluatorr? load_argstrainertreebankretagged_treebankrM retag_xposf1kbestF1_r/r/r0evaluate/s>    "rZcCs<|d|d|d|dd}tj||dd}||dS) zn A utility method to remove the optimizer from a save file Will make the save file a lot smaller r9r:r;r<r=F)r>load_optimizerN)rrGsave)r>model_save_filemodel_load_filerRrSr/r/r0remove_optimizer_s r_cs8dur|jD]}|jr|fddq dSdS)zP Adds a torch.clamp hook on each parameter if grad_clipping is not None Ncst| Sr&)torchclamp)grad grad_clippingr/r0xsz#add_grad_clipping..)rL parameters requires_grad register_hook)rSrdpr/rcr0add_grad_clippingqsrjc Cst|}td||dr#t||d|ddt||d|ddt|}td|t|}d|vr;td td |t|D] } | |vrRtd | qFttd d |Dtdd |Dd} |rvt| tdd |D} td| t |d|d|d\} } t |d|d|d\} }t |d|d|d\}}tdt | tdd t t| t| dd| D}|drt||dt||dt|}t||dt||dtd|t|| |d| |dd|t|| |d| |dd|t|}t||d}|r8|dr8tdtt|dd t |}tt||}t||d}td!d t t|t|| ||||||| || }||t||d"|| || fS)#zd Builds a Trainer (with model) and the train_sequences and transitions for the given trees. z'Unique constituents in training set: %scheck_valid_statesdevstrict_check_constituentsfailsilverzConstituent node counts: %sNz8Fatal problem: the tagger put None on some of the nodes!zUnique tags in training set: %szIFound tag in dev set which does not exist in train set: %s Continuing...cs|]}|VqdSr&count_unary_depth.0tr/r/r0 z build_trainer..csrqr&rrrtr/r/r0rwrxr6csrqr&rrrtr/r/r0rwrxzUnary limit: %dtrainingtransition_schemereversedz)Total unique transitions in train set: %dz(Unique transitions in training set: %s cSsg|] }|D]}|qqSr/) components)rutransxr/r/r0 sz!build_trainer..zRoot labels in treebank: %strainrare_word_thresholduse_silver_wordsz2Getting silver words to add to the delta embeddingz Silver wordspostfixz$Using the following open nodes: %srd)rget_unique_constituent_labelsrIrJrget_constituent_countsget_unique_tags RuntimeErrormaxr convert_trees_to_sequencesrKjoinmapstrsetr check_transitionsget_root_labelsrrget_unique_wordsget_rare_wordsget_common_wordstqdmsortedrr build_trainerlog_num_words_knownrj)r> train_trees dev_trees silver_treesr?r^train_constituentsconstituent_countstagstag unary_limittrain_sequencestrain_transitions dev_sequencesdev_transitionssilver_sequencessilver_transitionsexpanded_train_transitions root_labelswords rare_words silver_words open_nodesrSr/r/r0rzsd                rc Cst|t|ddkr|dd}nd}|dr:ddla|dr%|dnd|d}tj||d tjjd d d t|d }t|dt |d}t dt ||dr_t |d}t|}t |d}t dt |t |d}g}|drt |d}t dt ||drt |d}|durt d|d|dt|||d}t|||d}t|||d}t d|r|djnt} t||||| |\} } } } |dr| t|| || | ||| | | } Wdn1swY|drt| S) zG Build a model, train it using the requested train & dev files r5rr6Nwandb wandb_namez%s_constituency shorthand)nameconfig dev_scorer)summaryr7save_dir train_filez"Read %d trees for the training settrain_remove_duplicatesrrAzRead %d trees for the dev setrl silver_filez)Read %d trees for the silver training setsilver_remove_duplicatesrprCrMrDrVrEr@)rlog_training_argsrIrinitrun define_metricr ensure_dirr rHrJrKrrrr?rrr@iterate_trainingfinish)r>r^rPr8rrQrrrr?rSrrrr/r/r0rsP        "rcCs(dd|D}ddt|||D}|S)NcSsg|] }dd|DqS)cSs(g|]}t|jt|jdjddqS)r)label)rchildren)rrr)ru preterminalr/r/r0rs z1compose_train_data...)yield_preterminals)rur"r/r/r0rs  z&compose_train_data..cSsg|]}t|qSr/)r!rurr/r/r0r )zip)trees sequencespreterminal_listsdatar/r/r0compose_train_datas rcCs\|sggfS|}t||krt|||t||ks||d}|d|}||fS)a Return the next epoch_size trees from the training data, starting with leftover data from the previous epoch if there is any The training loop generally operates on a fixed number of trees, rather than going through all the trees in the training set exactly once, and keeping the leftover training data via this function ensures that each tree in the training set is touched once before beginning to iterate again. N)rKrandomshuffleextend)leftover_training_data train_data epoch_size epoch_datar/r/r0next_epoch_data s      rcCs|jD] }|ddkr nqtd|jD]a}|ddkrv|d}|ddur2||dkr2d|d<n2|d durC||d krCd|d<n!|d rZ||d d krZ|d|d |d<n |d|d|d<|d|krvtd|d||dqdS)zI Update the learning rate for the bert finetuning, if applicable param_group_namebasez-There should always be a base parameter groupbertlrbert_finetune_begin_epochNbert_finetune_end_epoch multistageepochsstage1_bert_learning_ratebert_learning_ratez(Setting %s finetuning rate from %f to %f) param_groupsAssertionErrorrIrJ)r> optimizerepochs_trainedbase_param_group param_groupold_lrr/r/r0update_bert_learning_rate$s&      rc +s|ddkrtddd} tjdd} nL|ddkrEzd d lm} Wn ty/td wtd |d dd} | d|d d} n|ddkrZtddd} tdd} ntd|d|j | fddt |j D} | t||}t||}|dst||d<|r|ds|d|d<|dri}|ddf||dd<t|r|ddf||ddd<d }|d!tjurt|j|d"|d#|d$}n,|d!tjurt|j|d"|d#|d$}n|d!tjurt|j|d"|d#|d$}g}g}|jd krtd%|j|j|j|d&d kr1|jd kr1|j|d'|jdd(t|jd)|dd)D]B|_| td*|jt||j |j|d+r]|!t"|||d\}}t"|||d\}}||}|j#d,dd-t$|j|| | | |||}t%|j&|||| \}}}||jks|jd kr|jd.krtd/||j||_|j|_|j|d0dd(|j'd krt(d1|j'd2|jd3|j)d4|j*d5|j+d6|j|fd7|j|jfg}td8,||j j-d d9}|j./||j j-d d9}||krtd:|||d;rNt0j1|j+|d<|jd=|d>rNt23|d>}|j&4D]\}}|5|rLt01|t6j78|iq7|d?d kr|j|d?krt9d@dA|j&j:j;|j&j} ||\}!}"t?@|j&jA}#|#Bdd |#BdCd tCjD|d0|#ddD}|j&}$tdE|tdF|jtE|}#tdG|!|"rdHndI|!|#d<|"sd |#dC<|F|dJ}%|G|dK}&|G|dL}'t|%|&|'|$jH|$jI|$jJ|$jK|$j |$jL|$jM|$jN|$jO|$j|$jP|$Q|#}(|( |(R|$tS|#|(d})tT|#|)}*tC|(|)|*|| |j|j}tU||dM|dNrX|dOrX|j|dOdd(|d&d ur|d&|jkr|j|dPd kr|j|d'|j|dQd(q=|S)Ra Given an initialized model, a processed dataset, and a secondary dev dataset, train the model The training is iterated in the following loop: extract a batch of trees of the same length from the training set convert those trees into initial parsing states repeat until trees are done: batch predict the model's interpretation of the current states add the errors to the list of things to backprop advance the parsing state for each of the trees losscrosszBuilding CrossEntropyLoss(sum)cS|Sr&r/rr/r/r0reXz"iterate_training..sum) reductionfocalr) FocalLossz^focal_loss not installed. Must `pip install focal_loss_torch` to use the --loss=focal featurezBuilding FocalLoss, gamma=%floss_focal_gammacSstj|ddS)Nr6)dim)r`softmaxrr/r/r0re`s)rgamma large_marginz&Building LargeMarginInSoftmaxLoss(sum)cSrr&r/rr/r/r0redrzUnexpected loss term: %scs(i|]\}}|tj|dddqS)F)rgr<r)r`tensor unsqueeze)ruyrr<r/r0 ksz$iterate_training..rsilver_epoch_sizerpattn_num_layersFrrTNrz oracle_leveladditional_oracle_levelsdeactivated_oracle_levelszLRestarting trainer with a model trained for %d epochs. Best epoch %d, f1 %fsave_each_startsave_each_name)save_optimizerr6zStarting epoch %drFcSs t|dS)Nr6)rKrr/r/r0res )keyrzNew best dev score: %.5f > %.5f save_namez!Had to ignore %d batches with NaNzEpoch %d finishedzTransitions correct: %szTransitions incorrect: %szTotal loss for epoch: %.5fzDev score (%5d): %8fzBest dev score (%5d): %8fr|rz$Updating learning rate from %f to %fr)r+r)stepwandb_norm_regex early_dropoutcss|]}|dkVqdS)rNr/rr/r/r0rwrxz#iterate_training..z"Setting dropout to 0.0 at epoch %d lattn_d_proj)r[z1Finished stage at epoch %d. Restarting optimizerz#Previous best model was at epoch %dz5Switching to a model with %d pattn layers and %slattnzNO r9r:r;rd checkpointcheckpoint_save_namesave_each_frequencysave_each_optimizer)VrIrJrCrossEntropyLossfocal_loss.focal_lossr ImportErrorr ValueErrorr<to enumerate transitionsrrrKr uses_lattnrIN_ORDERrrIN_ORDER_COMPOUNDrTOP_DOWNr best_epochrbest_f1r\rangerrrFrsorttrain_model_one_epochrNrLr,warningr'r(r+rr schedulerrrlogrecompilenamed_parameterssearchr`linalgnormany word_dropoutripredict_dropoutlstm_input_dropoutbatches_trainedcopydeepcopyr>poprrGdict load_pretrain load_charlm bert_modelbert_tokenizerforce_bert_saved peft_name constituentsr delta_wordsrconstituent_opensrcopy_with_new_structurerrrj)+r>rSrrrrrrr?rQprocess_outputsmodel_loss_functionrtransition_tensorsr silver_datamultistage_splitsoraclerleftover_silver_datarepoch_silver_data epoch_statsrWrYstats_log_linesrnew_lr watch_regexnrirr-stage_pattn_layersstage_uses_lattn temp_argsrLptforward_charlmbackward_charlm new_modelrr!r/rr0rFs                  $       ,            2rc Cs`ttdt||d}t||j} tdtttdd} tt |d|dD]3\} } || | |d} t || |j | ||||| }|j d7_ |j dkrU| | | |} q*tdd| jD}td d| jD}td |t| jtd |t| jt| jdkrtd d dd| jD| jdkrtd| j| S)Nrtrain_batch_sizerzEpoch %drr6cs|]\}}|VqdSr&r/rurYvr/r/r0rw*rxz(train_model_one_epoch..csrQr&r/rRr/r/r0rw+rxzTransitions correct: %d %szTransitions incorrect: %d %szOracle repairs: %sr|css&|]\}}d|j|j|fVqdS)z %s (%s): %dN)rvalue)rurrr/r/r0rw/s$zFake transitions used: %d)listrrKrrrr%rrrtrain_model_one_batchrLr-r,r zero_gradrr'itemsr(rIrJrr)r most_commonr*)epochrSr>r<r=rrAr>interval_startsrrD batch_idxinterval_startbatch batch_stats total_correcttotal_incorrectr/r/r0rs*   " rc & s|dd|Ddd|Ddd|D} t} t} t} d} g}g}t| dkrH|j| dd\}}}dd| D}fd d|D}||||g}g}t||| D]\}}}||kr| |d 7<|jd t|j kr|d ur||d krt |d krt |j }| ||r||||\}}|d ur||j|d||| d } qZ||||qZ| ||fd 7<|jd t|j krqZ|d us||d ks| ||s||||qZ||||\}}|jrJ| |d 7<|d ur(t |dkr(||j|d||qZ||||qZt| dkrB|j||dd} t| dks,t|}t|}||}|||}||drd} td|d||t|d}!tjD]F\}"}#|!|"rd} |#jr|#jd urtd|"tj |#tj |#jq||#jrtd|"tj |#q|td|"tj |#q|| stdt!t"|rd}$d }%n|#}$d}%t$|$| | | | |%S)aB Train the model for one batch The model itself will be updated, and a bunch of stats are returned It is unclear if this refactoring is useful in any way. Might not be ... although the indentation does get pretty ridiculous if this is merged into train_model_one_epoch and then iterate_training cSg|]}|jqSr/)r$rr/r/r0rAz)train_model_one_batch..cSrbr/)r"rr/r/r0rBrccSrbr/r#rr/r/r0rCrcrF)is_legalcSsg|]}|j|jqSr/)r#num_transitionsrr/r/r0rYscsg|]}|qSr/r/)rugold_transitionr>r/r0rZrr6Noracle_initial_epochoracle_forced_errorsrdoracle_frequencyTrnrGz#Watching %s ... epoch %d batch %dz %s norm: %f grad: %fz) %s norm: %f grad required, but is None!z %s norm: %f grad not requiredz (none found!)r)%initial_state_from_preterminalsrrKpredictappendrr short_namerfr#rchoicerre fix_error_replace is_correct bulk_applyr`catbackwardrIrJr#r$rSrLr%r&rgrbr'r(r)isnanitemr%)&rZr\rLtraining_batchr>r<r=rAr> current_batchr'r(r)r* all_errors all_answersoutputspred_transitionsrYgold_transitions trans_tensor new_batchupdate_transitionspred_transitionrgstatefake_transition new_sequence repair_typeerrorsanswers tree_lossmatchedrGrHri batch_lossr,r/rhr0rV5s     $             ;    & rVc Cstdt||d||dd}|dk}t|tdd\}}tt|} |j| |j |d|j |d} t | |} | } |dkrztd |d| g} tt |D]} tt|d d | d } | |j| |j |d|j|dqRd dt| D} t| t|krtdt| t|n ddt| |D} |dddkr|drtj|dd dtj|d|dd}tj|d|dd}tj|rtd|ntj|rtd|nt|d-}| D]"}|jdj}|dr||j}||d||dqWdn 1swYt |D]S} tj|d|dd| }t|d3}| | d D]#}|jdj}|drW||j}||d||dqDWdn 1sswYq&t|d}| D]}||d|j|dqWdn 1swYt| dkrd S|dur|dkrtd!d"| D}nd}t|d# }| | }Wdn 1swYn| | }|!d$r|j"nd}|j#||j$fS)%z This reparses a treebank and executes the CoreNLP Java EvalB code. It only works if CoreNLP 4.3.0 or higher is in the classpath. zProcessing %d trees from %srAr5rT)rreverseeval_batch_size) keep_scoreszGenerating %d random analysesFztb%03d)leavercSs*g|]}t|djdd|DddqS)rcSsg|]}|jdqS)r) predictions)rurir/r/r0rsz*run_dev_set...N)rgold)ruparsesr/r/r0rs"zrun_dev_set..z&Only evaluating %d trees instead of %dcSsg|] \}}|j|dqS))r)rr)rurrr/r/r0rsmodeNrm predict_file predict_dir)verbosez .pred.mrgz .orig.mrgz!Cowardly refusing to overwrite {}wpredict_output_gold_tagspredict_format z.%03d.pred.mrgr6)rrcss|]}t|jVqdSr&)rKr)rufrr/r/r0rwszrun_dev_set..r7rX)%rIrJrKevalgetr iterrparse_sentences_no_gradbuild_batch_from_treesrmr rrnweighted_choicerr rrospathrexistsformatopenrr" replace_tagsrwriterrprocessHasFieldrXrWtreeF1)rLretagged_treesoriginal_treesr>rQr5r sorted_treesoriginal_indices tree_iteratorrT full_resultsgenerated_treebanksi pred_file orig_filefoutr" output_treer8responserXr/r/r0rNs   $                  rNr&)F collectionsrrr.loggingrrr#r`rstanza.models.commonr%stanza.models.common.foundation_cacherr&stanza.models.common.large_margin_lossrstanza.models.common.utilsr r stanza.models.constituencyr r r 3stanza.models.constituency.in_order_compound_oracler*stanza.models.constituency.in_order_oracler%stanza.models.constituency.lstm_modelr,stanza.models.constituency.parse_transitionsr%stanza.models.constituency.parse_treer*stanza.models.constituency.top_down_oracler"stanza.models.constituency.trainerr stanza.models.constituency.utilsrrrrrrrrrstanza.server.parser_evalrrstanza.utils.get_tqdmr r getLoggerrIr!r%rZr_rjrrrrrrrrVrNr/r/r/r0sT             ,   0 N>"L${