o h_@sddlZddlZddlZddlZddlmZddlmmZ ddl m Z m Z m Z mZedZddiddidZGdddeZd d Zd)d d Zd)ddZddZdejfddZddZd*ddZd*ddZd*ddZdZddZd*dd Z d!d"Z!d#d$Z"d%d&Z#d+d'd(Z$dS),N)pad_packed_sequencepack_padded_sequence pack_sequencePackedSequencestanzause_fastT)zvinai/phobert-basezvinai/phobert-largecs eZdZdZfddZZS)TextTooLongErrorzF A text was too long for the underlying model (possibly BERT) cs&td|||f||_||_dS)NzoFound a text of length %d (possibly after tokenizing). Maximum handled length is %d Error occurred at line %d)super__init__line_numtext)selflengthmax_lenr r  __class__^/var/www/html/env_mimamsha/lib/python3.10/site-packages/stanza/models/common/bert_embedding.pyr s zTextTooLongError.__init__)__name__ __module__ __qualname____doc__r __classcell__rrrrrsrcCs|dvr d|_dSdS)N)zhf-internal-testing/tiny-bertgoogle/muril-base-casedzgoogle/muril-large-casedz-airesearch/wangchanberta-base-att-spm-uncasedzcamembert/camembert-largez,hfl/chinese-electra-180g-large-discriminatorz*NYTK/electra-small-discriminator-hungariani)model_max_length) model_name tokenizerrrrupdate_max_lengths rFcCs|rAzddlm}Wn tytdwt|t}|ds&d|d<|r-||||d<|j|fi|}t |||SdS)Nr) AutoTokenizerUPlease install transformers library for BERT support! Try `pip install transformers`. vinai/phobertTadd_prefix_spacelocal_files_only) transformersr ImportError BERT_ARGSgetdict startswithupdatefrom_pretrainedr)rtokenizer_kwargsr"r bert_argsbert_tokenizerrrrload_tokenizer$s     r.cCsT|r(zddlm}Wn tytdw|j||d}t|||d}||fSdS)Nr) AutoModelr)r")r+r")NN)r#r/r$r*r.)rr+r"r/ bert_modelr-rrr load_bert6s r1cCs`|dr dd|Dndd|D}d|}||}||}|jg||jg}||fS)zV Tokenize a sentence manually, using for checking long sentences and PHOBert. r cSs g|] }|ddddqS) _ replace.0wordrrr G z#tokenize_manual..cSg|]}|ddqS)r2r4r5r7rrrr:Gr4)r(jointokenizeconvert_tokens_to_ids bos_token_id eos_token_id)rsentr tokenizedsentencesent_idstokenized_sentrrrtokenize_manualBs &   rHc Cs~|durt|}g}|D] }dd|D}t|||\}}t||jdkr'q ||q t|dt|t|t||S)zI Filter out the (NER, POS) data that is too long for BERT model. NcSs"g|] }t|tr |n|dqS)r) isinstancestrr7rrrr:`s"zfilter_data..zWEliminated %d of %d datapoints because their length is over maximum size of BERT model.)r.rHlenrappendloggerlog) rdatar log_level filtered_datarCrEr3rGrrr filter_dataWs  rScCs&d|vsd|vr dS|drdSdS)zw TODO: we were lazy and didn't implement any form of length fudging for models other than bert/roberta/electra bartxlnetTr F)r()rrrrneeds_length_filterls  rVcCsX|durtj|ddddjddd}n tj|| ddd}|r*|S|S)z Clone & detach the feature, keeping the last N layers (or averaging -2,-3,-4 if not specified) averaging 3 of the last 4 layers worked well for non-VI languages Naxis)torchstacksumclonedetach)feature num_layersrarrrcloned_featurevs  $ rdc Cs<g}dd|D} || dddd} | d|} | d|} tttt| dD]o} | d}t|dt| }| ||} | ||} |rlt|| | dd }t |j ||}Wd n1sfwYn|| | dd }t |j ||}t ||D]\}}|d t|d }|s|d d }| |qq,|S)zu Handles vi-bart. May need testing before using on other bart https://github.com/VinAIResearch/BARTpho cSs g|] }ddd|DqS)r4cSr<)r4r3r5r7rrrr:r=z;extract_bart_word_embeddings...)r>r8rErrrr:r;z0extract_bart_word_embeddings..ptT)return_tensorspaddingreturn_attention_mask input_idsattention_maskrkoutput_hidden_statesNrKrX) torangeintmathceilrLminr]no_gradrddecoder_hidden_statesziprM)rrmodelrPdevicekeep_endpointsrcra processed sentencesrDrjrkistart_sentence end_sentencefeaturesrbrErrrextract_bart_word_embeddingss2     rc s>g}g} gt|D]E\} } t|| |\} } | t| |jkr>td|jt| || tt| |j| d|| | t |  g}||q t| }t j j jj| d|jd}g}ttt|dD]~}|d|d|d}|d}||jd}t j|||jd|d}t| ||D]\}} d||d t| f<q|rt || ||dd }|t|j||7}Wd n1swYqm||||dd }|t|j||7}qmt||ksJt|t|ksJfd d t|D}|rd d |D}dd t||D}|S)a# Extract transformer embeddings using a method specifically for phobert Since phobert doesn't have the is_split_into_words / tokenized.word_ids(batch_index=0) capability, we instead look for @@ to denote a continued token. data: list of list of string (the text tokens) %Invalid size, max size: %d, got %d %sr4T) batch_first padding_valuerlrrorzNrmcs,g|]\}fddtDqS)cs>g|]\}}|dkr|ddr|dkr|dqS)rroz@@)endswith)r8idx2r3)idxlist_tokenizedrrr:s>z9extract_phobert_embeddings...) enumerater8rCr)rrr:s z.extract_phobert_embeddings..cSsg|] }dg|dgqS)rrXr)r8offrrrr:scSsg|]\}}||qSrr)r8rboffsetrrrr:r=)rrHrMrLrrNerrorrr>r]tensorrannutilsrnn pad_sequence pad_token_idrqrrrsrtshapezerosrvr`rprd hidden_statesrx)rrryrPrzr{rcrar|tokenized_sentsrrCrDrGprocessed_sentsizetokenized_sents_paddedrr~ padded_inputrrrksent_idxrboffsetsrrrextract_phobert_embeddingssN    r) zbert-base-german-casedzdbmdz/bert-base-german-casedz!dbmdz/bert-base-italian-xxl-casedzdbmdz/bert-base-italian-casedz2dbmdz/electra-base-italian-xxl-cased-discriminatorz avichr/heBERTzonlplab/alephbert-basez#imvladikon/alephbertgimmel-base-512zcahya/bert-base-indonesian-1.5Gzindolem/indobert-base-uncasedrzl3cube-pune/marathi-robertacCs>g}|D]}||ddj}ddt||D}||q|S)aPatch bert tokenizers with missing characters There is an issue that some tokenizers (so far the German ones identified above) tokenize soft hyphens or other unknown characters into nothing If an entire word is tokenized as a soft hyphen, this means the tokenizer simply vaporizes that word. The result is we're missing an embedding for an entire word we wanted to use. The solution we take here is to look for any words which get vaporized in such a manner, eg `len(token) == 2`, and replace it with a regular "-" Actually, recently we have found that even the Bert / Electra tokenizer can do this in the case of "words" which are one special character long, so the easiest thing to do is just always run this function F)is_split_into_wordscSs$g|]\}}t|dkr|ndqS)rK-rL)r8r9tokenrrrr: s$z$fix_blank_tokens..)rjrxrM)rrPnew_datarErD new_sentencerrrfix_blank_tokens s  rc s|dddd}dd|D} tt|D]q} |j| d} d| | d<t| D]\} } | dur1n | d| | | d<q'| | d d| | d <td d | | Dratd || | | | |t| jdkrt djt| || t t| j| d || qg}tt t t|dD]}fdd|dd|d|dD}tdd |D}tjt||tj|d}t|D]!\} }d|| dt|f<t||kr|jg|t|q|rttj||d}|||dd}|t|j||7}Wdn 1swYqtj||d}|||dd}|t|j||7}qg}|s. batch_indexrrorXcs|]}|duVqdSNrr8xrrr 5z+extract_xlnet_embeddings..YOOPS, hit None when preparing to use Bert data[idx]: {} offsets: {} list_offsets[idx]: {}rKrr4rlcs(g|]}jg|ddjgqS)Nr)rArBrrrrr:Hs(rjcss|]}t|VqdSrrrrrrrIr)dtyperzrrmcSg|]}|ddqSrorXrrrrrr:cr=)rqrLword_idsrany ValueErrorformatrrNrrr>rrrsrtmaxr]rlongextendrrvrrdrrxrM)rrryrPrzr{rcrarD list_offsetsrrposrrr~rjrrk input_row id_tensorrbr|new_sentrrrextract_xlnet_embeddings$sV  *   rcCs4|jd|jkr|||dd}t|j||}|Sg}t|jd|jd} |j| } | dkr2td|} |} | ddd|jf} | ddd|jf}||| dd}t|j||}t|d krn|dd| dddf}||| jd|jkr|n| dd| df} | dd| df} q7tj |dd }|S) a4 Extract an embedding from the given transformer for a certain attention mask and tokens range In the event that the tokens are longer than the max length supported by the model, the range is split up into overlapping sections and the overlapping pieces are connected. No idea if this is actually any good, but at least it returns something instead of horribly failing TODO: at least two upgrades are very relevant 1) cut off some overlap at the end as well 2) use this on the phobert, bart, and xln versions as well roTrmrKzReally tiny tokenizer!NrrZ) rrrdrr RuntimeErrorrLrMr]cat)ryrattention_tensorrrcrarzrslices slice_len prefix_lenremaining_attention remaining_idsattention_sliceid_slicerrrbuild_cloned_featuresjs4    rcCstdgt|d}t|D]\}}|durq |||d<q d|d<|dddD]}|dur7|d|d<|Sq(|S)z[ Convert a transformers-tokenized sentence's offsets to a list of word to position NrKrorrrX)rLr)rErrrrrrrconvert_to_position_lists rc Cs||ddddd}g} tt|D]} t|| |j| d} | | qtdd| DrVt||}||ddddd}g} tt|D]} t|| |j| d} | | qCtdd| Drmtd || | | | |g} tt t t|d D]W}t j |d d |d |d |d }t j |d d |d |d |d }|rt | t|||||||7} Wdn1swYq|| t|||||||7} q|g}|sdd| D} t| | D] \}} || }||q|S)NlongestTF)rhrrrircs"|] }tdd|DVqdS)csrrrrrrrrr4extract_base_embeddings...Nrr8converted_offsetsrrrr z*extract_base_embeddings..csr)csrrrrrrrrrrNrrrrrrrrrlrkrrjcSrrrrrrrr:r=z+extract_base_embeddings..)rqrLrrrMrrrrrrrsrtr]rrvrrx)rrryrPrzr{rcrarDrrrrrr~rrr|rbrrrrextract_base_embeddingss>   &&  rc Cs|dur |jr |n ||||dr%t||||||||Sd|vr4t||||||||St|tr=t |}d|vrLt ||||||||St ||||||||S)z Extract transformer embeddings using a generic roberta extraction data: list of list of string (the text tokens) num_layers: how many to return. If None, the average of -2, -3, -4 is returned Nr rTrU) _hf_peft_config_loadeddisable_adaptersenable_adapters set_adapterr(rrrItuplelistrr) rrryrPrzr{rcra peft_namerrrextract_bert_embeddingss    r)NF)T)NTN)%rsloggingnumpynpr]torch.nnrtorch.nn.functional functionalFtorch.nn.utils.rnnrrrr getLoggerrNr%rrrr.r1rHDEBUGrSrVrdrrBAD_TOKENIZERSrrrrrrrrrrs8      $I F*4