o h'@szddlZddlZddlZddlZddlZddlmZ edZddZ ddZ dd Z e d kr;e ej d ddSdS) N)Counterz\n\s*\ncCs6||dkrt||}|rt|d}d|fSdS)zb Detect if a paragraph break can be found, and return the length of the paragraph break sequence.  rT)Fr)PARAGRAPH_BREAKmatchlengroup)indextext para_break break_lenr g/var/www/html/env_mimamsha/lib/python3.10/site-packages/stanza/utils/datasets/prepare_tokenizer_data.py is_para_breaks  rcCsd}d}|t|kr|t|krt||\}}|r:t|dkr.td|s,Jd|d}|d||d7}n5td||rQtd||sQ|||7}n|||7}||dd ||kskJd ||f|d7}|d7}|t|kr|t|ks||fS) zu Locate the next word in the text. In case a paragraph break is found, also write paragraph break to labels. rz^\s+$zSFound non-empty string at the end of a paragraph that doesn't match any token: |{}|z z^\s$r zECharacter mismatch: raw text contains |%s| but the next word is |%s|.)rrrerformatwritereplace)rr wordoutputidx word_sofarr r r r r find_next_words$    (rc Cst}|jdtdd|jdtdd|jdddtdd |jd d dtd d |j|d }t|jddd}d|}Wdn1sHwYt |}|j durZt j }nt j|j d}t j|ddt|j d}d}g}t|jddd}d} d} d} g} d} |D]}|}t |rF|ddkrt | dkr|} q|d}d|dvrq|d}d|dvrdd|ddD\} } |}g} nS| t|dkr| krnn| |g7} qt|d| kr| |g7} dd| D} ||t| fg7}|dr| ddstd| t jd d} d} d}qt | r)|| t||||\}}d!t |dd|dvrBd"nd#} qt | rlt| ddksVJ|| ddd$t| ddd} d} qWdn 1szwYd}|j r|d%|j }t|}|jdurtd&|dSt|jd}tjt |!|d'd(Wdn 1swY|d)t ||j}t|dS)*Nplaintext_filez'Plaintext file containing the raw input)typehelp conllu_filez2CoNLL-U file containing tokens and sentence breaksz-oz--outputzFOutput file name; output to the console if not specified (the default))defaultrrz-mz --mwt_outputzYOutput file name for MWT expansions; output to the console if not specified (the default))argsrzutf-8)encodingrrT)exist_okw# .r-cSsg|]}t|qSr )int.0xr r r izmain..cSsg|]}|qSr )lowerr+r r r r.qr/z0Sentence ID with potential wrong MWT expansion: )file013z{}z!Tokenizer labels written to %s zMWTs:)indentz1{} unique MWTs found in data. MWTs written to {})"argparseArgumentParser add_argumentstr parse_argsopenrjoin readlinesrrsysstdoutospathsplitmakedirsrstripr*tupleislowerprintstderrrrrcloser mwt_outputjsondumplistitems)r parserfr textlenroutdirrmwt_expansionsbufmwtbeginmwtendexpanded last_commentslinerlastmwt word_found status_linemwtsr r r main:s               ( (5   r___main__r)r7rLrArr? collectionsrcompilerrrr___name__argvr r r r s   \