o hu$@sdZddlZddlZddlmZddlmZddlmZmZm Z m Z m Z m Z m Z mZmZmZmZmZmZddlmZmZGdddeZGd d d ZdS) zI Utility functions for the loading and conversion of CoNLL-format files. N)ZipFile)Document) IDTEXTLEMMAUPOSXPOSFEATSHEADDEPRELDEPSMISCNER START_CHAREND_CHAR) FIELD_TO_IDX FIELD_NUMc@s eZdZdS) CoNLLErrorN)__name__ __module__ __qualname__rrM/var/www/html/env_mimamsha/lib/python3.10/site-packages/stanza/utils/conll.pyr src@seZdZedddZeddZeddZedd Zedd d Zedd dZ edddZ eddZ edddZ d S)CoNLLTc Csgg}}gg}}t|D]a\}}|d}t|dkr4t|dkr3||g}||g}q|dr?||q|d}|rMd|dvrMqt|tkrjtd|ddtd t|d |d | ||g7}qt|dkr||||||fS) a Load the file or string into the CoNLL-U format data. Input: file or string reader, where the data is in CoNLL-U format. Output: a tuple whose first element is a list of list of list for each token in each sentence in the data, where the innermost list represents all fields of a token; and whose second element is a list of lists for each comment in each sentence in the data. z r# .zCannot parse CoNLL line z : expecting z fields, z found at line z ) enumeratelstriprstriplenappend startswithsplitrr) fignore_gappingdocsent doc_comments sent_commentsline_idxlinearrayrrr load_conlls0         .    zCoNLL.load_conllc Csg}g}t|D]p\}}g}g}t|D]Y\}}zt|} Wnty7} z td||t| f| d} ~ wwd| tvrVtdd| tjdddD| t<| | qtdd| tjd ddD| t<| | q| || |q||fS) ai Convert the CoNLL-U format input data to a dictionary format output data. Input: list of token fields loaded from the CoNLL-U format data, where the outmost list represents a list of sentences, and the inside list represents all fields of a token. Output: a list of list of dictionaries for each token in each sentence in the document. z*Could not process sentence %d token %d: %sNrcs|]}t|VqdSNint.0xrrr Ez&CoNLL.convert_conll..rmaxsplitcsr/r0r1r3rrrr6Hr7-) rrconvert_conll_token ValueErrorrstrrtupler$r") doc_conlldoc_dict doc_emptysent_idx sent_conll sent_dict sent_empty token_idx token_conll token_dicterrr convert_conll4s( $ $   zCoNLL.convert_conllcCs.t|}d|}|d}dd|D}|S)a Convert the dictionary format input data to the CoNLL-U format output data. This is the reverse function of `convert_conll`, but does not include sentence level annotations or comments. Can call this on a Document using `CoNLL.convert_dict(doc.to_dict())` Input: dictionary format data, which is a list of list of dictionaries for each token in each sentence in the data. Output: CoNLL-U format data as a list of list of list for each token in each sentence in the data. z{:c}z cSs g|] }dd|dDqS)cSsg|]}|dqS)rr$r3rrr \sz1CoNLL.convert_dict... rK)r4sentencerrrrL\s z&CoNLL.convert_dict..)rformatr$)r@r'text sentencesr?rrr convert_dictNs   zCoNLL.convert_dictcCsti}tD]3}|t|}|dkr|tkrt|||<n|||<|ttdkr7|tt|t<|tt|t<q|S)z Convert the CoNLL-U format input token to the dictionary format output token. Input: a list of all CoNLL-U fields for the token. Output: a dictionary that maps from field name to value. _)rr r2rr)rGrHfieldvaluerrrr;_s zCoNLL.convert_conll_tokenNc Cst||gr t||grJd|r|sJd|r(t|}t||\}}nW|rat|*}||}ttj|dd|\}}Wdn1sLwYWdn1s[wYnt|dd}t||\}}Wdn1szwYt |\} } | || fS)zV Load the CoNLL-U format data from file or string into lists of dictionaries. z%either use input file or input stringz*must provide input_file if zip_file is setutf-8encodingN) anyallioStringIOrr.ropen TextIOWrapperrJ) input_file input_strr&zip_fileinfiler?r)zinfinr@rArrr conll2dictss$     zCoNLL.conll2dictcCs(tj||||d\}}}t|d||dS)NrarPcommentsempty_sentences)rrer)r_r`r&rar@r)rArrr conll2docszCoNLL.conll2doccCsNtj||||d\}}}g}g}g} g} d} t|||D]d\} } }| D]M}|ds/|drp|jdddd}t|dkrA|} n$|| krdt|d| | d}| dkr\|jD]}| |_ qT| ||} nq#| g}| g} |g} nq#| | | | | |qt|dkrt|d| | d}| dkr|jD]}| |_ q| ||} |S) Nrfz # doc_id =z # newdoc id ==rr8rrg) rrezipr#r$r!rrQstripdoc_idr")r_r`r&rar@r)rAdocs current_doccurrent_comments current_emptycurrent_doc_idr'rhemptycommentrnnew_docirrrconll2multi_docssJ           zCoNLL.conll2multi_docscCst|}t||dS)zq Convert the dictionary format input data to the CoNLL-U format output data and write to a file. N)rrwrite_doc2conll)r@filenamer'rrr dict2conllszCoNLL.dict2conllwrVcCsbt|dr|d|dSt|||d}|d|WddS1s*wYdS)a Writes the doc as a conll file to the given file. If passed a string, that filename will be opened. Otherwise, filename.write() will be called. Note that the output needs an extra at the end to be a legal output file writez{:C} rWN)hasattrr}rOr])r'rzmoderXoutfilerrrrys "zCoNLL.write_doc2conll)T)NNTN)r|rV) rrr staticmethodr.rJrRr;rerjrxr{ryrrrrrs& "      ) r)__doc__osr[zipfilerstanza.models.common.docrrrrrrr r r r r rrrrrr<rrrrrrs  <