o h+@sdZddlZddlmZedZdZeeedgZ ddZ d d Z d d Z d dZ ddZddZddZddZddZdS)z1 Utility functions for dealing with NER tagging. N)EMPTYstanza)_-NOcCs0|D]}t|dkr|dddvrdSqdS)z Check if a basic tagging scheme is used. Return True if so. Args: all_tags: a list of NER tags Returns: True if the tagging scheme does not use B-, I-, etc, otherwise False N)B-I-S-E-B_I_S_E_FTlenall_tagstagrR/var/www/html/env_mimamsha/lib/python3.10/site-packages/stanza/models/ner/utils.pyis_basic_schemes  rcCs:|D]}|tvr qt|dkr|dddvrqdSdS)z Check if BIO tagging scheme is used. Return True if so. Args: all_tags: a list of NER tags Returns: True if the tagging scheme is BIO, otherwise False rN)r r r rFT)EMPTY_OR_O_TAGrrrrr is_bio_schemes rcCsg}t|D]G\}}|tvr||q|ddkrH|dks6||ddks6||ddd|ddkrB|d|ddq||q||q|S)a Convert the original tag sequence to BIO2 format. If the input is already in BIO2 format, the original input is returned. Args: tags: a list of tags in either BIO or BIO2 format Returns: new_tags: a list of tags in BIO2 format rIrNB enumeraterappendtagsnew_tagsirrrrto_bio21s   8  r%cCstg}t|D]1\}}|tvr||q|dks(||ddks(||d|kr0|d|q|d|q|S)a Convert a basic tag sequence into a BIO sequence. You can compose this with bio2_to_bioes to convert to bioes Args: tags: a list of tags in basic (no B-, I-, etc) format Returns: new_tags: a list of tags in BIO format rrrr r rr!rrr basic_to_bioIs  (r&cCs$g}t|D]\}}|tvr||qt|dkr!td||dddvrU|dt|krI||ddddvrI|d|ddq|d|ddq|dddvr|dt|kr}||ddddvr}|d |ddq|d |ddqtd ||S) z Convert the BIO2 tag sequence into a BIOES sequence. Args: tags: a list of tags in BIO2 format Returns: new_tags: a list of tags in BIOES format rzInvalid BIO2 tag found: N)r rrr r )r r r r zInvalid IOB tag found: )rrr r Exceptionr!rrr bio2_to_bioes_s    ((r(cCsdd|D}|S)z If any tags are None, _, -, or blank, turn them into EMPTY The input should be a list(sentence) of list(word) of tuple(text, list(tag)) which is the typical format for the data at the time data.py is preprocessing the tags cSg|] }dd|DqS)cSs*g|]}|dtdd|dDfqS)rcss |] }|tvr tn|VqdSN) EMPTY_TAGr.0xrrr sz=normalize_empty_tags....rtupler-wordrrr s*z3normalize_empty_tags...rr-sentencerrrr4sz(normalize_empty_tags..r) sentences new_sentencesrrrnormalize_empty_tagssr9csNg}g}d}t|D]C\}}t|\}}|||s0tdd|Dr0|dkr.td|d}|rHtdd|DsAtd|d d |D}||q td d|Dt|D]\}}tfd d|Drptd |q[g} g} tD]Gd} d} fdd |D} t| }| ot | }|r| dkrd} t dn|r| dkrd} t d| | | | qyg}t||D]R\}}dd |D}tt| | D]2\\} } fdd |D} | rt t| } n t| } | rt | } t| D] \}}|||<qq|dd t||Dq|r%dd |D}|S)z Convert tags in these sentences to bioes We allow empty tags ('_', '-', None), which will represent tags that do not get any gradient when training Fcs"|] }|dup t|tVqdSr* isinstancestrr-rrrrr/ zprocess_tags..rzGGot a mix of tags and lists of tags. First non-list was in sentence %dTcsr:r*r;r>rrrr/r?zLGot a mix of tags and lists of tags. First tag as a list was in sentence %dcSsg|]}|fqSrrr>rrrr4sz process_tags..css"|] }|D]}t|VqqdSr*r)r-r"r.rrrr/r?c3s|] }t|kVqdSr*rr,) max_columnsrrr/szQNER tags not uniform in length at sentence %d. TODO: extend those columns with Ocsg|] }|D]}|qqSrr)r-sentr. column_idxrrr4sbioeszOBIO tagging scheme found in input at column %d; converting into BIOES scheme...zQBasic tagging scheme found in input at column %d; converting into BIOES scheme...cSr))cSsg|]}|qSrrr,rrrr4s+process_tags...r)r- sentence_tagsrrrr4csg|]}|qSrrr,rBrrr4scSsg|] \}}|t|fqSrr0)r-wtrrrr4scSr))cSs g|] }|d|ddfqS)rrrr2rrrr4s rErr5rrrr4rG)rzipr any ValueErrorallmaxrangerrlowerloggerdebugr(r&r%)r7scheme all_wordsrconverted_tuplessent_idxrAwordsr"all_convert_bio_to_bioesall_convert_basic_to_bioesconvert_bio_to_bioesconvert_basic_to_bioes tag_columnis_biois_basicresulttag_idxrr)rCr@r process_tagssj         racsggdfdd}t|D]c\}}|durd}|dkr&|gq|dr8||g|ddq|drI||ddq|dr_||dd|gq|d ru||g|dd|gq|S) z Decode from a sequence of BIOES tags, assuming default tag is 'O'. Args: tags: a list of BIOES tags Returns: A list of dict with start_idx, end_idx, and type values. Ncs.tdkrddddSdS)Nr)startendtype)rr rcur_typeent_idxsresrrflushs  z decode_from_bioes..flushrr rr r r )r startswithr )r"rjidxrrrfrdecode_from_bioess>         rmcGst|d}|ddD]}d}|t|kr||dkr!|d7}q||dr9||dkr4||||<|d7}q||dsHtd|||}|d}|t|kr||dd||ddkrltd ||||d rtn||d std |||d7}|t|ksT|t|krtd ||d}td d|||Dr||||||<|}|t|ksq |S)z~ Merge multiple sequences of NER tags into one sequence Only O is replaced, and the earlier tags have precedence rrNrr r z)Got unexpected tag sequence at idx {}: {}rz%Unexpected tag sequence at idx {}: {}r r z'Got a sequence with an unclosed tag: {}css|]}|dkVqdS)rNrr,rrrr/6szmerge_tags..)listrrkrLformatrM) sequencesr"sequencerl start_idxend_idxrrr merge_tags sD         &rt)__doc__loggingstanza.models.common.vocabr getLoggerrQr+r1rnrrrr%r&r(r9rarmrtrrrrs    P 1