o h@sddlZddlZddlZddlZddlZddlmZddlmZm Z m Z  de fddZ GdddZ d d d Zed krAedSdS) N)utils)ListTupleAnypathcCstjjj|S)zh" loads in a Stanza document object from a path to a CoNLL file containing annotated sentences. )stanzarconllCoNLL conll2doc)rr i/var/www/html/env_mimamsha/lib/python3.10/site-packages/stanza/models/lemma_classifier/prepare_dataset.pyload_doc_from_conll_filesr c@s^eZdZdedeedefddZddZdeefd d Ze d d Z d eddfddZ dS) DataProcessor target_word target_uposallowed_lemmascCs(||_t||_||_t||_dSN)rrecompiletarget_word_regexrr)selfrrrr r r __init__s zDataProcessor.__init__cCs0|jD]}|j|jr|j|jvrdSqdS)NTF)wordsr fullmatchtextuposr)rsentencewordr r r keep_sentence s zDataProcessor.keep_sentencereturncCs@g}t|jD]\}}|j|jr|j|jvr||q|S)zt Finds all occurrences of self.target_word in tokens and returns the index(es) of such occurrences. ) enumeraterrrrrrappend)rr occurrencesidxtokenr r r find_all_occurrences&s  z"DataProcessor.find_all_occurrencescCst|ddd@}|d|dt||dd}|D]}|s,|dd }n|d |t|q |d WddS1sJwYdS) Nzw+zutf-8)encodingz{ z "upos": %s, z "sentences": [Fz Tz, z ] } )openwritejsondumps) save_namer sentencesoutput_fwrote_sentencerr r r write_output_file0s     "zDataProcessor.write_output_filer+Nc sg}|jD]C|rHddjD}|}|D],}|jj|jrGfddttjD}t|}| |||j|jdqq|rS| ||j ||S)a Takes any sentence from `doc` that meets the condition of `keep_sentence` and writes its tokens, index of target word, and lemma to `save_name` Sentences that meet `keep_sentence` and contain `self.target_word` multiple times have each instance in a different example in the output file. Args: doc (Stanza.doc): Document object that represents the file to be analyzed save_name (str): Path to the file for storing output cSsg|]}|jqSr )r).0r$r r r Osz2DataProcessor.process_document..csg|]}j|jqSr )rr)r0irr r r1Vs)r upos_tagsindexlemma) r,rrr%rrr6rangelenr!r/r) rdocr+r,tokensindexesr#r4 num_tokensr r3r process_document@s&    zDataProcessor.process_document) __name__ __module__ __qualname__strrrrintr% staticmethodr/r=r r r r rs rc Cst}|jdttjtjtddd|jdtddd|jdtd d d|jd td d d|jdtddd| |}|j }|j }|j }|j }t|}|D] }t|d||qQt|}t|d|g|d}|||S)Nz --conll_pathzen_gum-ud-train.conlluz#path to the conll file to translate)typedefaulthelpz --target_wordz'szToken to classify on, e.g. 's.z --target_uposAUXzupos on target tokenz --output_pathztest_output.txtzPath for output filez--allowed_lemmasz.*z?A regex for allowed lemmas. If not set, all lemmas are allowedz: r)rrr)argparseArgumentParser add_argumentrAosrjoindirname__file__ parse_args conll_pathr output_pathrvarsprintr rr=) argsparserrPrrQrargr9 processorr r r maincs"$  rX__main__r)rHr)rKrrstanza.models.lemma_classifierrtypingrrrrAr rrXr>r r r r s  K