o h@s`ddlZddlmZddlmZddlmZmZddZd dd Z d d Z e d kr.e dSdS)N)doc)TokenizationDataset)predictdecode_predictionscCsbt|jD])\}}t|jD]\}}t|jdkr-ddd|jD}|j|kr-dSqqdS)zf Return True/False if the MWTs in the doc are all exactly composed of the text in their words css|]}|jVqdS)N)text).0xr R/var/www/html/env_mimamsha/lib/python3.10/site-packages/stanza/models/mwt/utils.py sz)mwts_composed_of_words..FT) enumerate sentencestokenslenwordsjoinr)rsent_idxsentence token_idxtokenexpectedr r r mwts_composed_of_wordss rTc Csrd|jvr tdd|jvrtd|jd}|jd}ddd|D}t|j||jd|jjd }t|j||jj d |j d |j d|j d d d\}}|rt ||D];\} } d } | D]2} t | d krrd | | | t | d<| | t | dd krd| | t | d<| t | d7} q^qVt|jd|||d|jj d|jj ddkd\} } }t||}|||S)a Uses the tokenize processor and the mwt processor in the pipeline to resplit tokens into MWT tokens: a list of list of string pipeline: a Stanza pipeline which contains, at a minimum, tokenize and mwt keep_tokens: if True, enforce the old token boundaries by modify the results of the tokenize inference. Otherwise, use whatever new boundaries the model comes up with. between running the tokenize model and breaking the text into tokens, we can update all_preds to use the original token boundaries (if and only if keep_tokens == True) This method returns a Document with just the tokens and words annotated. tokenizez/Need a Pipeline with a valid tokenize processormwtz*Need a Pipeline with a valid mwt processorz css|]}d|VqdS) N)r)r rr r r r +szresplit_mwt..T) input_textvocab evaluation dictionary batch_size max_seqlen num_workersr)trainerdata_generatorr!r"use_regex_tokensr#rN skip_newline shorthandla_ittb)rmwt_dict orig_textall_raw all_preds no_ssplitr'use_la_ittb_shorthand) processors ValueErrorrrconfigrr$r rargsgetMAX_SEQ_LENGTH_DEFAULTziprrrDocumentprocess)rpipeline keep_tokenstokenize_processor mwt_processor fake_textbatchesr-r,rpredchar_idxword_documentr r r resplit_mwtsT          rDcCsLtjdddd}gdddgg}t||}t|t||dd }t|dS) Nenz tokenize,mwtgum)r0package)Izcan'tbelieveitzI can'tsleepF)r:)stanzaPipelinerDprint)piperrr r r mainRs   rP__main__)T) rLstanza.models.commonrstanza.models.tokenization.datar stanza.models.tokenization.utilsrrrrDrP__name__r r r r s   >