o hX@s"dZddlZddlZddlZddlZddlZddlZddlmZddlm Z ddl m Z m Z ddl mZddl mZddlmZmZmZdd lmZdd lmZdd lmZmZmZmZmZmZmZm Z dd l!m"Z"e#d Z$e#dZ%ddZ&ddZ'dddZ(dddZ)e*dkre)dSdS)a8!A command line interface to a shift reduce constituency parser. This follows the work of Recurrent neural network grammars by Dyer et al In-Order Transition-based Constituent Parsing by Liu & Zhang The general outline is: Train a model by taking a list of trees, converting them to transition sequences, and learning a model which can predict the next transition given a current state Then, at inference time, repeatedly predict the next transition until parsing is complete The "transitions" are variations on shift/reduce as per an intro-to-compilers class. The idea is that you can treat all of the words in a sentence as a buffer of tokens, then either "shift" them to represent a new constituent, or "reduce" one or more constituents to form a new constituent. In order to make the runtime a more competitive speed, effort is taken to batch the transitions and apply multiple transitions at once. At train time, batches are groups together by length, and at inference time, new trees are added to the batch as previous trees on the batch finish their inference. There are a few minor differences in the model: - The word input is a bi-lstm, not a uni-lstm. This gave a small increase in accuracy. - The combination of several constituents into one constituent is done via a single bi-lstm rather than two separate lstms. This increases speed without a noticeable effect on accuracy. - In fact, an even better (in terms of final model accuracy) method is to combine the constituents with torch.max, believe it or not See lstm_model.py for more details - Initializing the embeddings with smaller values than pytorch default For example, on a ja_alt dataset, scores went from 0.8980 to 0.8985 at 200 iterations averaged over 5 trials - Training with AdaDelta first, then AdamW or madgrad later improves results quite a bit. See --multistage A couple experiments which have been tried with little noticeable impact: - Combining constituents using the method in the paper (only a trained vector at the start instead of both ends) did not affect results and is a little slower - Using multiple layers of LSTM hidden state for the input to the final classification layers didn't help - Initializing Linear layers with He initialization and a positive bias (to avoid dead connections) had no noticeable effect on accuracy 0.8396 on it_turin with the original initialization 0.8401 and 0.8427 on two runs with updated initialization (so maybe a small improvement...) - Initializing LSTM layers with different gates was slightly worse: forget gates of 1.0 forget gates of 1.0, input gates of -1.0 - Replacing the LSTMs that make up the Transition and Constituent LSTMs with Dynamic Skip LSTMs made no difference, but was slower - Highway LSTMs also made no difference - Putting labels on the shift transitions (the word or the tag shifted) or putting labels on the close transitions didn't help - Building larger constituents from the output of the constituent LSTM instead of the children constituents hurts scores For example, an experiment on ja_alt went from 0.8985 to 0.8964 when built that way - The initial transition scheme implemented was TOP_DOWN. We tried a compound unary option, since this worked so well in the CoreNLP constituency parser. Unfortunately, this is far less effective than IN_ORDER. Both specialized unary matrices and reusing the n-ary constituency combination fell short. On the ja_alt dataset: IN_ORDER, max combination method: 0.8985 TOP_DOWN_UNARY, specialized matrices: 0.8501 TOP_DOWN_UNARY, max combination method: 0.8508 - Adding multiple layers of MLP to combine inputs for words made no difference in the scores Tried both before the LSTM and after A simple single layer tensor multiply after the LSTM works well. Replacing that with a two layer MLP on the English PTB with roberta-base causes a notable drop in scores First experiment didn't use the fancy Linear weight init, but adding that barely made a difference 260 training iterations on en_wsj dev, roberta-base model as of bb983fd5e912f6706ad484bf819486971742c3d1 two layer MLP: 0.9409 two layer MLP, init weights: 0.9413 single layer: 0.9467 - There is code to rebuild models with a new structure in lstm_model.py As part of this, we tried to randomly reinitialize the transitions if the transition embedding had gone to 0, which often happens This didn't help at all - We tried something akin to attention with just the query vector over the bert embeddings as a way to mix them, but that did not improve scores. Example, with a self.bert_layer_mix of size bert_dim x 1: mixed_bert_embeddings = [] for feature in bert_embeddings: weighted_feature = self.bert_layer_mix(feature.transpose(1, 2)) weighted_feature = torch.softmax(weighted_feature, dim=1) weighted_feature = torch.matmul(feature, weighted_feature).squeeze(2) mixed_bert_embeddings.append(weighted_feature) bert_embeddings = mixed_bert_embeddings It seems just finetuning the transformer is already enough (in general, no need to mix layers at all when finetuning bert embeddings) The code breakdown is as follows: this file: main interface for training or evaluating models constituency/trainer.py: contains the training & evaluation code constituency/ensemble.py: evaluation code specifically for letting multiple models vote on the correct next transition. a modest improvement. constituency/evaluate_treebanks.py: specifically to evaluate multiple parsed treebanks against a gold. in particular, reports whether the theoretical best from those parsed treebanks is an improvement (eg, the k-best score as reported by CoreNLP) constituency/parse_tree.py: a data structure for representing a parse tree and utility methods constituency/tree_reader.py: a module which can read trees from a string or input file constituency/tree_stack.py: a linked list which can branch in different directions, which will be useful when implementing beam search or a dynamic oracle constituency/lstm_tree_stack.py: an LSTM over the elements of a TreeStack constituency/transformer_tree_stack.py: attempts to run attention over the nodes of a tree_stack. not as effective as the lstm_tree_stack in the initial experiments. perhaps it could be refined to work better, though constituency/parse_transitions.py: transitions and a State data structure to store them constituency/transition_sequence.py: turns ParseTree objects into the transition sequences needed to make them constituency/base_model.py: operates on the transitions to turn them in to constituents, eventually forming one final parse tree composed of all of the constituents constituency/lstm_model.py: adds LSTM features to the constituents to predict what the correct transition to make is, allowing for predictions on previously unseen text constituency/retagging.py: a couple utility methods specifically for retagging constituency/utils.py: a couple utility methods constituency/dyanmic_oracle.py: a dynamic oracle which currently only operates for the inorder transition sequence. uses deterministic rules to redo the correct action sequence when the parser makes an error. constituency/partitioned_transformer.py: implementation of a transformer for self-attention. presumably this should help, but we have yet to find a model structure where this makes the scores go up. constituency/label_attention.py: an even fancier form of transformer based on labeled attention: https://arxiv.org/abs/1911.03875 constituency/positional_encoding.py: so far, just the sinusoidal is here. a trained encoding is in partitioned_transformer.py. this should probably be refactored to common, especially if used elsewhere. stanza/pipeline/constituency_processor.py: interface between this model and the Pipeline stanza/utils/datasets/constituency: various scripts and tools for processing constituency datasets Some alternate optimizer methods: adabelief: https://github.com/juntang-zhuang/Adabelief-Optimizer madgrad: https://github.com/facebookresearch/madgrad N)constant)utils) add_peft_argsresolve_peft_args)parser_training) retagging)ConstituencyCompositionSentenceBoundary StackHistory)TransitionScheme)load_model_parse_text)DEFAULT_LEARNING_EPSDEFAULT_LEARNING_RATESDEFAULT_MOMENTUMDEFAULT_LEARNING_RHODEFAULT_WEIGHT_DECAY NONLINEARITYadd_predict_output_argspostprocess_predict_output_args)DEFAULT_MODEL_DIRstanzazstanza.constituency.trainerc Cs> t}|jdtddd|jdtddd|jdtd d d|jd td d d|jdtdd|jdtd dd|jdtd dd|jdtd dd|jdddd dd|jdtddd|jddd d d!d"|jd#d$d%d&d'|jd(d)d*d+d,|jd-d td.d/|jd0d td1d/|jd2d td3d/|jd4d5td6d/|jd7d td8d/|jd9d:td;d/|jdd?d*d@d,t||jdAtdBdCd|jdDtdEdFd|jdGtd dHd|jdIdJd*dKdLdM|jdNtd dOd|jdPd$d%dQd'|jdRtd dHd|jdStd dTd|jdUtd dVd|jdWtd dXd|jdYdZgd[d\|jd]td^d_dt||jd`tdadb|jdctdddb|jdetdBdfd|jdgtdBdhd|jditj djdkdl dm dndotDd/|jdpdtdqd/|jdrtj dsdkdl dm dtdotDd/|jdudvtdqd/|jdwtdxdyd|jdztd{d|jd|td}d~d|jdtd dd|jddJd%dd'|jddd*dd,|jdtddd|jdtddd|jdtddd|jdtd dd|jdtd dd|jdtd dd|jdtddd|jdtddd|jdtddd|jdtddd|jdtd dd|jdtd dd|jdtddd|jdddJd*dd|jdtdd|jddJd*dddM|jddJd*dddMt ||jdd td td/|jdd td td/|jdd td td/|jdd tdd/|jdttdd/|jddtdd/|jdd dd|jdd tdĠ tdd/|jdd^tdd/|jddtdd/|jddtdd/|jddtdd/|jdd tdd/|jdd tdd/|jdd tdd/|jddd ddd"|jdddd|jddtdd/|jddtdd/|jddtdd/|jddtdd/|jddtdd/|jddtdd/|jdtjddkd dm ddotDd/|jdd$d%dd'|jddJd%dd'|jddd*dd,|jddtdd|jdd tdd/|jddJdd%dd|jddJdd*dd|jddtdd/|jddtdd/|jddtd d/|jd dtd d/|jd d td d/|jddtdd/|jdtjddkd dm ddotDd/|jdtjddkd dm ddotDd/|jddvtdd/|jdd tdd/|jdd%dd|jd d%d!d|jd"td d#d|jd$d%d*d&d,|jd'td d(d|jd)td d*dt||jd+d,td-d/|jd.dtd/d/|jd0dxtd1d/|jd2dvtd3d/|jd4d5td6d/|jd7d8td9d/|jd:d;tdd/|jd?dtd@d/|jdAd^tdBd/|jdCd$d%dDd'|jdEdFdGdFgdHd|jdId tdJd/|jdKd5tdLd/|jdMd5tdNd/|jdOdJd%dPd'|jdQdJd%dRd'|jdSd$d%dTd'|jdUdJd%dVd'|jdWdJd*dXdVdM|jdYd$d%dZd'|jd[d\td]d/|jd^dtd_d/|jd`d8tdad/|jdbdtdcd/|jdddtded/|jdfdJd%dgd'|jdhd$d%did'|jdjdkd*did,|jdldmd*dnd,|jdod$d%dpd'|jdqd$d%drd'|jdsd dtd|jdud%dvd|jdwd dxd|jdyd dzd|S({z Adds the arguments for building the con parser For the most part, defaults are set to cross-validated values, at least for WSJ z --data_dirzdata/constituencyzDirectory of constituency data.)typedefaulthelpz --wordvec_dirzextern_data/wordveczDirectory of word vectorsz--wordvec_filezFile that contains word vectorsz--wordvec_pretrain_fileNz'Exact name of the pretrain file to readz--pretrain_max_vocabi)rrz--charlm_forward_filez$Exact path to use for forward charlmz--charlm_backward_filez%Exact path to use for backward charlmz --bert_modelz>Use an external bert model (requires the transformers package)z--no_bert_model bert_model store_constzDon't use bert)destactionconstrz--bert_hidden_layersz;How many layers of hidden state to use from the transformerz--bert_hidden_layers_originalbert_hidden_layersz&Use layers 2,3,4 of the Bert embedding)rrrrz--bert_finetuneF store_truez(Finetune the bert (or other transformer))rrrz--no_bert_finetune bert_finetune store_falsez.Don't finetune the bert (or other transformer))rrrz--bert_finetune_layersz3Only finetune this many layers from the transformer)rrrz--bert_finetune_begin_epochz/Which epoch to start finetuning the transformerz--bert_finetune_end_epochz.Which epoch to stop finetuning the transformerz--bert_learning_rateg;On?z?Scale the learning rate for transformer finetuning by this muchz--stage1_bert_learning_ratez^Scale the learning rate for transformer finetuning by this much only during an AdaDelta warmupz--bert_weight_decayg-C6?z>Scale the weight decay for transformer finetuning by this muchz--stage1_bert_finetunezuFinetune the bert (or other transformer) during an AdaDelta warmup, even if the second half doesn't use bert_finetunez--no_stage1_bert_finetunestage1_bert_finetunez{Don't finetune the bert (or other transformer) during an AdaDelta warmup, even if the second half doesn't use bert_finetunez--tag_embedding_dimz2Embedding size for a tag. 0 turns off the featurez--delta_embedding_dimdz$Embedding size for a delta embeddingz --train_filezInput file for data loader.z--no_train_remove_duplicatesTtrain_remove_duplicateszlDo/don't remove duplicates from the training file. Could be useful for intentionally reweighting some trees)rrrrz --silver_filezSecondary training file.z--silver_remove_duplicateszsDo/don't remove duplicates from the silver training file. Could be useful for intentionally reweighting some treesz --eval_filez--xml_tree_filez?Input file of VLSP formatted trees for parsing with parse_text.z--tokenized_filez9Input file of tokenized text for parsing with parse_text.z--tokenized_dirz>Input directory of tokenized text for parsing with parse_text.z--modetrain)r) parse_textpredictremove_optimizer)rchoicesz--num_generaterzLWhen running a dev set, how many sentences to generate beyond the greedy onez--langLanguage)rrz --shorthandzTreebank shorthandz--transition_embedding_dimzEmbedding size for a transitionz--transition_hidden_sizez#Embedding size for transition stackz--transition_stackcS t|SNr upperxr5\/var/www/html/env_mimamsha/lib/python3.10/site-packages/stanza/models/constituency_parser.pye z build_argparse..z*How to track transitions over a parse. {}z, cs|]}|jVqdSr0name.0r4r5r5r6 fz!build_argparse..z--transition_headszCHow many heads to use in MHA *if* the transition_stack is Attentionz--constituent_stackcSr/r0r1r3r5r5r6r7ir8csr9r0r:r<r5r5r6r>jr?z--constituent_headsz --hidden_sizeiz?Size of the output layers for constituency stack and word queuez--epochsiz --epoch_sizeizRuns this many trees in an 'epoch' instead of going through the training dataset exactly once. Set to 0 to do the whole training setz--silver_epoch_sizezNRuns this many trees in a silver 'epoch'. If not set, will match --epoch_sizez --multistagez^1/2 epochs with adadelta no pattn or lattn, 1/4 with chosen optim and no lattn, 1/4 full modelz--no_multistage multistagez don't do the multistage learningz--oracle_initial_epochz_Epoch where we start using the dynamic oracle to let the parser keep going with wrong decisionsz--oracle_frequencyg?zHHow often to use the oracle vs how often to force the correct transitionz--oracle_forced_errorsgMbP?z`Occasionally have the model randomly walk through the state space to try to learn how to recoverz--oracle_levelziRestrict oracle transitions to this level or lower. 0 means off. None means use all oracle transitions.z--additional_oracle_levelszpAdd some additional experimental oracle transitions. Basically for A/B testing transitions we expect to be bad.z--deactivated_oracle_levelszhTemporarily turn off a default oracle level. Basically for A/B testing transitions we expect to be bad.z--train_batch_sizez7How many trees to train before taking an optimizer stepz--eval_batch_size2z)How many trees to batch when running evalz --save_dirzsaved_models/constituencyzRoot dir for saving models.z --save_namez2{shorthand}_{embedding}_{finetune}_constituency.ptzFile name to save the modelz--save_each_namez@Save each model in sequence to this pattern. Mostly for testingz--save_each_startzWhen to start saving each modelz--save_each_frequencyz!How frequently to save each modelz--no_save_each_optimizersave_each_optimizerz1Don't save the optimizer when saving 'each' model)rrrrz--seediz--no_check_valid_statescheck_valid_stateszDon't check the constituents or transitions in the dev set when starting a new parser. Warning: the parser will never guess unknown constituentsz--no_strict_check_constituentsstrict_check_constituentsz`Don't check the constituents between the train & dev set. May result in untrainable transitionsz--learning_ratezLearning rate for the optimizer. Reasonable values are 1.0 for adadelta or 0.001 for SGD. None uses a default for the given optimizer: {}z--learning_epszSeps value to use in the optimizer. None uses a default for the given optimizer: {}z--learning_momentumz:Momentum. None uses a default for the given optimizer: {}z--learning_weight_decayz1Weight decay (eg, l2 reg) to use in the optimizerz--learning_rhozRho parameter in Adadeltaz--learning_beta2g+?zBeta2 argument for AdamWz--optimz8Optimizer type: SGD, AdamW, Adadelta, AdaBelief, Madgrad)rrz--stage1_learning_ratezTLearning rate to use in the first stage of --multistage. None means use default: {}adadeltaz--learning_rate_warmupzNumber of epochs to ramp up learning rate from 0 to full. Set to 0 to always use the chosen learning rate. Currently not functional, as it didn't do anythingz--learning_rate_factorg333333?z-Plateau learning rate decreate when plateauedz--learning_rate_patiencezPlateau learning rate patiencez--learning_rate_cooldown zPlateau learning rate cooldownz--learning_rate_min_lrzPlateau learning rate minimumz--stage1_learning_rate_min_lrz'Plateau learning rate minimum (stage 1)z--grad_clippingzPClip abs(grad) to this amount. Use --no_grad_clipping to turn off grad clippingz--no_grad_clipping grad_clippingz0Use --no_grad_clipping to turn off grad clippingz--losscrosszMcross, large_margin, or focal. Focal requires `pip install focal_loss_torch`z--loss_focal_gammazgamma value for a focal lossz--early_dropoutzWhen to turn off dropoutz--word_dropoutg?zDropout on the word embeddingz--predict_dropoutz%Dropout on the final prediction layerz--lstm_layer_dropoutgzDropout in the LSTM layersz--lstm_input_dropoutzDropout on the input to an LSTMz--transition_schemecSr/r0)r r2r3r5r5r6r7cr8zTransition scheme to use. {}csr9r0r:r<r5r5r6r>dr?z --reversedz#Do the transition sequence reversedz--combined_dummy_embeddingzWUse the same embedding for dummy nodes and the vectors used when combining constituentsz--no_combined_dummy_embeddingcombined_dummy_embeddingz]Don't use the same embedding for dummy nodes and the vectors used when combining constituentsz--nonlinearityreluzMNonlinearity to use in the model. relu is a noticeable improvement over tanh)rr-rz --maxout_kzAUse maxout layers instead of a nonlinearity for the output layersz--use_silver_wordsuse_silver_wordszCTrain/don't train word vectors for words only in the silver dataset)rrrrz--no_use_silver_wordsz--rare_word_unknown_frequency{Gz?z7How often to replace a rare word with UNK when trainingz--rare_word_thresholdzEHow many words to consider as rare words as a fraction of the datasetz--tag_unknown_frequencyz1How often to replace a tag with UNK when trainingz--num_lstm_layersz#How many layers to use in the LSTMsz--num_tree_lstm_layerszHow many layers to use in the TREE_LSTMs, if used. This also increases the width of the word outputs to match the tree lstm inputs. Default 2 if TREE_LSTM or TREE_LSTM_CX, 1 otherwisez--num_output_layersz.How many layers to use at the prediction levelz--sentence_boundary_vectorscSr/r0)r r2r3r5r5r6r7r8z5Vectors to learn at the start & end of sentences. {}csr9r0r:r<r5r5r6r>r?z--constituency_compositioncSr/r0)rr2r3r5r5r6r7r8z5How to build a new composition from its children. {}csr9r0r:r<r5r5r6r>r?z--reduce_headszhNumber of attn heads to use when reducing children into a parent tree (constituency_composition == attn)z--reduce_positionzDimension of position vector to use when reducing children. None means 1/4 hidden_size, 0 means don't use (constituency_composition == key | untied_key)z--relearn_structurezStarting from an existing checkpoint, add or remove pattn / lattn. One thing that works well is to train an initial model using adadelta with no pattn, then add pattn with adamw)rrz --finetunez=Load existing model during `train` mode from `load_name` pathz--checkpoint_save_namez,File name to save the most recent checkpointz--no_checkpoint checkpointzDon't save checkpointsz --load_namezKModel to load when finetuning, evaluating, or manipulating an existing filez--load_packagezIDownload an existing stanza package & use this for tests, finetuning, etcz--pattn_d_modeliz*Partitioned attention model dimensionalityz--pattn_morpho_emb_dropoutzFDropout rate for morphological features obtained from pretrained modelz--pattn_encoder_max_lenz?Max length that can be put into the transformer attention layerz--pattn_num_headsz5Partitioned attention model number of attention headsz --pattn_d_kv@z Size of the query and key vectorz --pattn_d_ffiz=Size of the intermediate vectors in the feed-forward sublayerz--pattn_relu_dropoutg?z1ReLU dropout probability in feed-forward sublayerz--pattn_residual_dropoutz9Residual dropout probability for all residual connectionsz--pattn_attention_dropoutzAttention dropout probabilityz--pattn_num_layerszENumber of layers for the Partitioned Attention. Currently turned offz --pattn_biasz(Whether or not to learn an additive biasz--pattn_timingsinlearnedz*Use a learned embedding or a sin embeddingz--lattn_d_input_projzNIf set, project the non-positional inputs down to this size before proceeding.z --lattn_d_kvz!Dimension of the key/query vectorz--lattn_d_projz=Dimension of the output vector from each label attention headz--lattn_resdropz&Whether or not to use Residual Dropoutz --lattn_pwffz8Whether or not to use a Position-wise Feed-forward Layerz--lattn_q_as_matrixzNWhether or not Label Attention uses learned query vectors. False means it doesz--lattn_partitionedz Whether or not it is partitionedz--no_lattn_partitionedlattn_partitionedz--lattn_combine_as_selfz@Whether or not the layer uses concatenation. False means it doesz --lattn_d_l zNumber of labelsz--lattn_attention_dropoutzDropout for attention layerz --lattn_d_ffz#Dimension of the Feed-forward layerz--lattn_relu_dropoutz$Relu dropout for the label attentionz--lattn_residual_dropoutz(Residual dropout for the label attentionz--lattn_combined_inputz4Combine all inputs for the lattn, not just the pattnz --use_lattnz+Use the lattn layers - currently turned offz--no_use_lattn use_lattnz--no_lattn_combined_inputlattn_combined_inputz:Don't combine all inputs for the lattn, not just the pattnz --log_normsz=Log the parameters norms while training. A very noisy optionz --log_shapesz*Log the parameters shapes at the beginningz --watch_regexzs>!      (     X5