o –h¸&ã@sÄdZddlZddlmZddlZddlZddlZddlZddl m Z ddl m Z ddl mZddlmZmZddlmZmZddlmZe d ¡Ze ej¡d Zd Zd d „Zedkr`eƒdSdS)a# Loads NER models & separates out the word vectors to base & delta The model will then be resaved without the base word vector, greatly reducing the size of the model This may be useful for any external users of stanza who have an NER model they wish to reuse without retraining If you know which pretrain was used to build an NER model, you can provide that pretrain. Otherwise, you can give a directory of pretrains and the script will test each one. In the latter case, the name of the pretrain needs to look like lang_dataset_pretrain.pt éN)Ú defaultdict)ÚPipeline)Úlang_to_langcode)ÚPretrainÚPretrainedWordVocab)ÚPAD_IDÚ VOCAB_PREFIX)ÚTrainerÚstanzaFg-Cëâ6?c(sdt ¡}|jdtddd|jdtddd|jdtd d d| ¡}tj |j¡rA|j}t  |¡}t |ƒd kr@t d   |j¡ƒ‚ntj  |j¡sPt d   |j¡ƒ‚tj |j¡\}}|g}tj |j¡r|j}t  |¡‰ttƒ}ˆD]‰|ˆ d¡d  ˆ¡qqntj |¡\}‰ˆg‰t‡fdd„ƒ}|j}tj|ddg}g}g} |D]Ä} tj || ¡} d} |  | ¡s¾td  | ¡ƒ‚| dt | ƒ …} | jddd\}}tdƒtd||fƒt|ddd|i| d}|jd}td  |¡ƒ|jd }|jj}|d‰|jjjj d }t!|j"dƒ}||kr*|dkr|d ks*td!  ||j"d|¡ƒ‚t#t$||||ƒƒ}|D]l}tj ||¡}td"  |¡ƒt%|d#‰td$  ˆj&j ¡ƒtd%  |jjjj ¡ƒˆj&j d|jjjj dkrwtd&ƒq8t'ˆj&j d |jjjj d ƒ}ˆj&j d |jjjj d krÄt(‡fd'd(„t)|ƒDƒƒr¨td)ƒntd*ƒq8ˆj&j d |jjjj d krÄtd+  |¡ƒt*|j +¡ƒj,}|jjjd|…dd…fˆj& -|¡d|…dd…f‰ˆ .¡‰t/j0j1ˆdd, 2¡ 3¡}t4 5|d k¡d krtd-ƒ‚t4 5|t6k¡}||d.krAtd/|||fƒˆj&j d |jjjj d kr?td0ƒˆj‰ˆ|d<ˆj&j d|j"d1<nrtd2|||fƒt5‡fd3d(„ˆDƒƒ}td4|ƒt7r¤d }ˆD];}|ˆjvrlqbˆ 8|¡} |jjj| dd…f}!ˆj 8|¡}"ˆj&|"dd…f}#|! .¡ 2¡|# 1¡t6krœ|d7}qbtd5|ƒq8td6  |¡ƒ| | ¡q¦d7| 9¡vs¼J‚‡fd8d9„t)d:ƒDƒ}$g}%t)d:t |ƒƒD]}&||&d;krê|% ˆ :|&¡¡|$ ˆ|& 2¡¡qÐ|jj; d<¡t |%ƒd krtd=ƒ|  | ¡nTtd>t |%ƒƒtd?t |$ƒƒt4 <|$¡}$t/ =|$¡}$|$j d t |%ƒt t>ƒks.J‚t|$j ƒt?|%ˆj@ˆjAd@}%|%|d7<tB C|$j d |$j dtD¡|j_E|jjEjjF G|$¡tj || ¡}'| H|'¡| | |f¡q¦tƒt |ƒd kr„tdAƒ|D]}&t|&ƒq|t |ƒd kr™tdBƒ|D]}&t|&ƒq‘t | ƒd kr®tdCƒ| D] }&t|&ƒq¦dSdS)DNz --input_pathzsaved_models/nerz*Where to find NER models (dir or filename))ÚtypeÚdefaultÚhelpz --output_pathzsaved_models/shrunkz&Where to write shrunk NER models (dir)z--pretrain_pathzsaved_models/pretrainz)Where to find pretrains (dir or filename)rzNo ner models found in {}zNo ner model found at path {}Ú_csˆS©N©r)Ú pretrainsrúf/var/www/html/env_mimamsha/lib/python3.10/site-packages/stanza/utils/training/separate_ner_pretrain.pyÚBszmain..T)Úexist_okz _nertagger.ptzUnexpected name: {}é)Úmaxsplitz/===============================================zProcessing lang %s package %sz tokenize,nerÚner)Ú processorsÚtokenize_pretokenizedÚpackageÚner_model_pathzLoaded NER processor: {}ÚwordÚlangÚzhzzh-hansz#lang not as expected: {} vs {} ({})zAttempting pretrain: {})Úfilenamez" pretrain shape: {}z" embedding in ner model shape: {}z% DIMENSION DOES NOT MATCH. SKIPPINGc3s$|] }ˆ |¡ˆ |¡kVqdSr)Úid2unit©Ú.0Úx)Ú word_vocabrrÚ {s€"zmain..z= Attempting to use pt vectors to replace ner model's vectorsz: NUM VECTORS DO NOT MATCH. WORDS DO NOT MATCH. SKIPPINGzR WARNING: if any vectors beyond {} were fine tuned, that fine tuning will be lost)Údimz,This should not be - a norm was less than 0!éz* Accepted! %d of %d vectors match for %sz+ Setting model vocab to match the pretrainÚ word_emb_dimz, %d of %d vectors matched for %s - SKIPPINGc3s|]}|ˆjvVqdSr)Úvocabr!)Úptrrr%“s€z %d words were in both vocabsz1 %d vectors were close when ignoring id orderingz COULD NOT FIND A MATCHING PT: {}Údeltacsg|]}ˆ| ¡‘qSr)Úcpu)r"Úi)r+rrÚ «szmain..égÚword_embzJNo vectors were changed! Perhaps this model was trained without finetune.z%d delta vocabz%d vectors in the delta set)rÚlowerzFinal pretrain mappings:zMISSING EMBEDDINGS:zNOT FINE TUNED:)IÚargparseÚArgumentParserÚ add_argumentÚstrÚ parse_argsÚosÚpathÚisdirÚ input_pathÚlistdirÚlenÚFileNotFoundErrorÚformatÚisfileÚsplitÚ pretrain_pathrÚlistÚappendÚ output_pathÚmakedirsÚjoinÚendswithÚ ValueErrorÚprintrrÚtrainersÚmodelr)r0ÚweightÚshaperÚargsÚsortedÚsetrÚembÚminÚallÚrangeÚnextÚ parametersÚdeviceÚtoÚdetachÚtorchÚlinalgÚnormr,ÚnumpyÚnpÚsumÚEPSÚDEBUGÚunit2idÚkeysr Úunsaved_modulesÚstackÚ from_numpyrrrr1ÚnnÚ EmbeddingrÚ delta_embÚdataÚcopy_Úsave)(ÚparserrNÚ ner_model_dirÚnersÚ pt_model_dirÚlang_to_pretrainÚnew_dirÚfinal_pretrainsÚmissing_pretrainsÚ no_finetuneÚ ner_modelÚner_pathÚexpected_endingÚ short_namerrÚpipeÚ ner_processorÚtrainerr)Ú num_vectorsÚlcodeÚ ner_pretrainsÚpt_modelÚpt_pathÚNrWÚ delta_normsÚ num_matchingÚ vocab_sameÚrearranged_countr#Úx_idÚx_vecÚpt_idÚpt_vecÚ delta_vectorsÚ delta_vocabr-Únew_pathr)r+rr*r$rÚmain%s  ÿ ÿ       6    € € €         ýrŽÚ__main__)Ú__doc__r2Ú collectionsrÚloggingr7r]r^rZÚtorch.nnrgr rÚstanza.models.common.constantrÚstanza.models.common.pretrainrrÚstanza.models.common.vocabrrÚstanza.models.ner.trainerr Ú getLoggerÚloggerÚsetLevelÚERRORrar`rŽÚ__name__rrrrÚs,       2 ÿ