o h@sdZddlZddlZddlZddlmmmZddl m Z m Z m Z e dZddZejfddZd d Zd d ZdddZddZddZeedfddZdS)z- Utility functions for data transformations. N)HEADIDUPOSstanzacsfdd|D}|S)Ncs"g|] }|vr |ntjqS)constantUNK_ID.0tvocabrT/var/www/html/env_mimamsha/lib/python3.10/site-packages/stanza/models/common/data.py s"zmap_to_ids..r)tokensr idsrr r map_to_idssrcCsg}|}t|dtr%|tdd|Ddd|D}t|dts tj|g|R|}t|D]\}}t|||dt|f<q5|S)z4 Convert (list of )+ tokens to a padded LongTensor. rcs|]}t|VqdSNlen)r yrrr z"get_long_tensor..cSsg|] }|D]}|qqSrr)r rzrrrrsz#get_long_tensor..N) isinstancelistappendmaxtorch LongTensorfill_ enumerater) tokens_list batch_sizepad_idsizesxrisrrrget_long_tensorsr*cCs|dus |ddur dStdd|D}t|dd}t|||}t|D]\}}t|||dt|ddf<q*|S)Nrcsrrr)r r'rrrr$rz#get_float_tensor..)rrr FloatTensorzero_r") features_listr$seq_len feature_lenfeaturesr(frrrget_float_tensor!s"r2cCsb|ggkr gggfS|gtt|gt|}ddttt|ddD}|dd|dfS)zO Sort all fields by descending order of lens, and return the original indices. cSsg|]}t|qSr)rr rrrr0szsort_all..T)reverseN)rangerrzipsorted)batchlens unsorted_all sorted_allrrrsort_all+s  r=皙??c st|}tfdd|D}tfdd|D}tfdd|D}|dkr.td|dkr.c3r@rrrA)can_augment_predicaterrrErc3s"|] }|o | VqdSrrrArDrCrrrFsrzOcan_augment_predicate allowed sentences not allowed by should_augment_predicatez9Found no sentences which matched can_augment_predicate {}g)rsumAssertionErrorloggerwarningformat) train_datarCrD desired_ratio max_ration_datan_should_augment n_can_augmentn_errorn_neededratiorrErget_augment_ratio3s$rTcCs|d}|tddkS)NPUNCT)getr)rB last_wordrrr should_augment_nopunct_predicateXsrYcsN|dtddkrdSttdkrdStfdd|Dr%dSdS) zo Check that the sentence ends with PUNCT and also doesn't have any words which depend on the last word rUNrVFr5c3s4|]}t|tdko|ttdkVqdS)r5rN)rrr)r wordrXrrrfs2z0can_augment_nopunct_predicate..T)rWrrrany)rBrr[rcan_augment_nopunct_predicate\sr]TcCst|dkrgS|durt|||}|dkr|rt|SgSg}|D]'}||rIt|krBt|dkrBt|dd}||q"|rI||q"|S)a Adds extra training data to compensate for some models having all sentences end with PUNCT Some of the models (for example, UD_Hebrew-HTB) have the flaw that all of the training sentences end with PUNCT. The model therefore learns to finish every sentence with punctuation, even if it is given a sentence with non-punct at the end. One simple way to fix this is to train on some fraction of training data with punct. Params: train_data: list of list of dicts, eg a conll doc augment_ratio: the fraction to augment. if None, a best guess is made to get to 10% should_augment_predicate: a function which returns T/F if a sentence already ends with not PUNCT can_augment_predicate: a function which returns T/F if it makes sense to remove the last PUNCT TODO: do this dynamically, as part of the DataLoader or elsewhere? One complication is the data comes back from the DataLoader as tensors & indices, so it is much more complicated to manipulate rNr5rU)rrTrrandomr)rK augment_ratiorCrDkeep_original_sentencesnew_datarB new_sentencerrr augment_punctjs$    rc)r>r?)__doc__loggingr^r%stanza.models.common.seq2seq_constantmodelscommonseq2seq_constantrstanza.models.common.docrrr getLoggerrHrPAD_IDr*r2r=rTrYr]rcrrrrs$  %