o h37@sdZddlZddlmZddlZddlZddlZddlZddlZddl Z ddl m Z ddl m Z ddlmZmZddlmZmZmZmZmZddlmZed Zd d Zdd d ZddZdddZddZddZ e!dkrtedSdS)a Entry point for training and evaluating a neural tokenizer. This tokenizer treats tokenization and sentence segmentation as a tagging problem, and uses a combination of recurrent and convolutional architectures. For details please refer to paper: https://nlp.stanford.edu/pubs/qi2018universal.pdf. Updated: This new version of tokenizer model incorporates the dictionary feature, especially useful for languages that have multi-syllable words such as Vietnamese, Chinese or Thai. In summary, a lexicon contains all unique words found in training dataset and external lexicon (if any) is created during training and saved alongside the model after training. Using this lexicon, a dictionary is created which includes "words", "prefixes" and "suffixes" sets. During data preparation, dictionary features are extracted at each character position, to "look ahead" and "look backward" to see if any words formed found in the dictionary. The window size (or the dictionary feature length) is defined at the 95-percentile among all the existing words in the lexicon, this is to eliminate the less frequent but long words (avoid having a high-dimension feat vector). Prefixes and suffixes are used to stop early during the window-dictionary checking process. N)copy)utils)Trainer) DataLoaderTokenizationDataset) load_mwt_dict eval_modeloutput_predictions load_lexiconcreate_dictionary)_training_loggingstanzacCsxt}|jdtdd|jdtddd|jdtdd d|jd tdd d|jd td d|jdtddd|jdtddd|jdtdd|jdtdd|jddddgd|jdddd|jdtdd d|jd!td"d#d|jd$td%d&d|jd'd(d)d*d+|jd,d-d)d.d+|jd/td0d1d|jd2dd3d|jd4tdd5d|jd6td7d8d|jd9dd:d|jd;tdtd?d@d|jdAtdBdCd|jdDtdEdFd|jdGtdHdId|jdJtdHdKd|jdLtdMdNd|jdOtdHdPd|jdQtdRdSd|jdTtdUdVd|jdWtdUdXd|jdYtdZd[d|jd\td]d^d|jd_tdd`d|jdatdbdcd|jddtdedfd|jdgtdhdid|jdjtd]dkd|jdltdmdnd|jdotdpdqd|jdrtddsd|jdttddud|jdvtdwdxdt||jdytdzd{|jd|d}ddd~d|jdd}d)dd+|jdddd|jdddd|S)z4 If args == None, the system args are used. z --txt_filezInput plaintext file)typehelpz --label_fileNzCharacter-level label file)rdefaultrz--mwt_json_filezJSON file for MWT expansionsz --conll_filezCoNLL file for outputz--dev_txt_filez1(Train only) Input plaintext file for the dev setz--dev_label_filez7(Train only) Character-level label file for the dev setz--dev_conll_goldz<(Train only) CoNLL-U file for the dev set for early stoppingz--langLanguagez --shorthandzUD treebank shorthandz--modetrainpredict)rchoicesz--skip_newline store_truez\Whether to skip newline characters in input. Particularly useful for languages like Chinese.)actionrz --emb_dim zDimension of unit embeddingsz --hidden_dim@zDimension of hidden unitsz--conv_filtersz1,9zbConfiguration of conv filters. ,, separates layers and , separates filter sizes in the same layer.z --no-residualresidual store_falsezAdd linear residual connections)destrrz--no-hierarchical hierarchicalz"Hierarchical" RNN tokenizerz--hier_invtemp?zSInverse temperature used in propagating tokenization predictions between RNN layersz--input_dropoutz Dropout input embeddings as wellz --conv_resz)Convolutional residual layers for the RNNz --rnn_layerszLayers of RNN in the tokenizerz--use_dictionarya(Use dictionary feature. The lexicon is created using the training data and external dict (if any) expected to be found under the same folder of training dataset, formatted as SHORTHAND-externaldict.txt where each line in this file is a word. For example, data/tokenize/zh_gsdsimp-externaldict.txtz--max_grad_normg?z Maximum gradient norm to clip toz--annealg+?zHAnneal the learning rate by this amount when dev performance deterioratez--anneal_afteriz2Anneal the learning rate no earlier than this stepz--lr0gMb`?zInitial learning ratez --dropoutgQ?zDropout probabilityz--unit_dropoutzUnit dropout probabilityz--feat_dropoutg?z?Features dropout probability for each element in feature vectorz--feat_unit_dropoutz.The whole feature of units dropout probabilityz --tok_noiseg{Gz?z:Probability to induce noise to the input of the higher RNNz--sent_drop_probg?z|Probability to drop sentences at the end of batches during training uniformly at random. Idea is to fake paragraph endings.z--last_char_drop_probzProbability to drop the last char of a block of text during training, uniformly at random. Idea is to fake a document ending w/o sentence final punctuation, hopefully to avoid the tokenizer learning to always tokenize the last character as a periodz--weight_decaygz Weight decayz --max_seqlendz-Maximum sequence length to consider at a timez --batch_sizezBatch size to usez--epochs z#Total epochs to train the model forz--stepsiPz7Steps to train the model for, if unspecified use epochsz--report_stepsz#Update step interval to report lossz--shuffle_stepsz8Step interval to shuffle each paragraph in the generatorz --eval_stepszEStep interval to evaluate the model on the dev set for early stoppingz--max_steps_before_stopizJEarly terminates after this many steps if the dev scores are not improvingz --save_namezFile name to save the modelz --load_namezFile name to load a saved modelz --save_dirzsaved_models/tokenizezDirectory to save models inz--seedi)rrz --use_mwtuse_mwtz}Whether or not to include mwt output layers. If set to None, this will be determined by examining the training data for MWTs)rrrrz --no_use_mwtz+Whether or not to include mwt output layersz--wandbzStart a wandb session and write the results of training. Only applies to training. Use --wandb_name instead to specify a namez --wandb_namezWName of a wandb session to start when training. Will default to the dataset short name)rr)argparseArgumentParser add_argumentstrintfloatradd_device_args)parserr,R/var/www/html/env_mimamsha/lib/python3.10/site-packages/stanza/models/tokenizer.pybuild_argparse"sh r.cCs*t}|j|d}|jrd|_t|}|S)NargsT)r. parse_args wandb_namewandbvars)r0r+r,r,r-r1`s  r1cCs^|ddur |d}n|dd}tjtj|d|s&tj|r&|Stj|d|S)N save_name shorthandz _tokenizer.ptsave_dir)ospathexistsjoin)r0r5r,r,r-model_file_namejs   &r<cCst|d}t|dtd|dgd|d<t|d|d<t||d<tt j |dd |dd krCt |dSt |dS) Nr/seedzRunning tokenizer in {} modemode) space_before capitalizednumeric end_of_para start_of_para feat_funcsfeat_dimr5rr)r1rset_random_seedloggerinfoformatlenr< ensure_dirr8r9splitrevaluater/r,r,r-maints      rNcCs|drt|\}|d<t|}|d|dd7<nd|d<d}d}t|d}|d|dd }t|||d }|j}t||d <|d |d d }t|||d|d}|ddurs||d<t d |drldnd|dt |||||dd} |ddurt j |d|d} | | | |dt|} |ddur|dn t| |d|dd} |d} d}d}d}|drddl}|dr|dnd |d!}|j||d"|jjd#d$d%|jjd&d'd%td(| d(D]}|j|d)|d*d+}| |}||d,dkr(t d- || ||dr(|jd#|i|d.|d/dkr<||d/dkr<|||d0dkrt|| |||}|dr[|jd&|i|d.d1 |d2g}||d3kr||kr|d4 | | |d5g7}| |d59} | | |}||kr|d6g7}|}|}| |d7n"|dkr|||d8kr|d9 ||g7}t d:|n t d:|q|dr||dkrt d; ||dSt d<| |d7dS)=Nuse_dictionary num_dict_featrEr mwt_json_filetxt_file label_filetxtlabel) input_files dictionary vocab_size dev_txt_filedev_label_fileTrXvocab evaluationrYr#z9Found {}mwts in the training data. Setting use_mwt to {}zno device)r0r^lexiconrYra load_namer7lr0stepsepochs batch_sizerr3r2z %s_tokenizerr6)nameconfig train_lossmin)summary dev_scoremaxr unit_dropoutfeat_unit_dropout)rprq report_stepszStep {:6d}/{:6d} Loss: {:.3f})step shuffle_steps eval_stepszDev score: {:6.3f}r anneal_afterzlr: {:.6f} -> {:.6f}annealzNew best dev score!r5max_steps_before_stopz4Stopping training after {} steps with no improvement zBest dev score={} at step {}z,Dev set never evaluated. Saving final model)r r rrr^rJrhas_mwtrGrHrIrr8r9r;load change_lrr(r3initrun define_metricrangenextupdatelogshufflersavefinish)r0rbrYmwt_dicttrain_input_files train_batchesr^dev_input_files dev_batchestrainerrcNrelrprev_dev_scorebest_dev_score best_dev_stepr3r2rsbatchlossrnreportsr,r,r-rs    "  0           rc Cst|d}t|dp|d|dd}|j|j}}|D]}|ds.|dvr.||||<q|d|d d }t|||d |jd }t|d |||||d\}} } } t d || d|| dS)NrRrcr5ra) model_filera_file)rar>r7rcr5rSrTrUTr] conll_file max_seqlenz OOV rate: {:6.3f}% ({:6d}/{:6d})r) rrr0r^endswithrrYr rGrHrI) r0rr loaded_argsr^keval_input_filesbatches oov_countr_r,r,r-rMs  " rM__main__)N)"__doc__r$rloggingrandomnumpynpr8torchjsonstanza.models.commonr"stanza.models.tokenization.trainerrstanza.models.tokenization.datarr stanza.models.tokenization.utilsrrr r r stanza.modelsr getLoggerrGr.r1r<rNrrM__name__r,r,r,r-s0      > f