o hJ @sZdZddlZddlmZddlmZddlmZmZddZ eedGd d d eZ dS) z. Processors related to Jieba in the pipeline. N)doc)TOKENIZE)ProcessorVariantregister_processor_variantcCs&zddl}WdStytdw)zI Import necessary components from Jieba to perform tokenization. rNzuJieba is used but not installed on your machine. Go to https://pypi.org/project/jieba/ for installation instructions.T)jieba ImportError)rrY/var/www/html/env_mimamsha/lib/python3.10/site-packages/stanza/pipeline/external/jieba.py check_jieba s  r rc@seZdZddZddZdS)JiebaTokenizercCs:|ddvr tdtddl}||_|dd|_dS)z Construct a Jieba-based tokenizer by loading the Jieba pipeline. Note that this tokenizer uses regex for sentence segmentation. lang)zhzzh-hanszzh-hantz[Jieba tokenizer is currently only allowed in Chinese (simplified or traditional) pipelines.rN no_ssplitF) Exceptionr rnlpgetr)selfconfigrrrr __init__s zJiebaTokenizer.__init__c Cst|tjr |j}n|}t|tstd|jj|dd}g}g}d}|D]A}t d|r4|t |7}q%tj |tj tj d|dtjd|t |i}|||t |7}|jsf|dvrf||g}q%t |dkrr||t||S) z^ Tokenize a document with the Jieba tokenizer and wrap the results into a Doc object. zFMust supply a string or Stanza Document object to the Jieba tokenizer.F)cut_allrz\s+=|)u。u！u？!?) isinstancerDocumenttextstrrrcutrematchlenTEXTMISC START_CHAREND_CHARappendr) rdocumentrtokens sentencescurrent_sentenceoffsettoken token_entryrrr process&s2    (      zJiebaTokenizer.processN)__name__ __module__ __qualname__rr.rrrr r s r ) __doc__rstanza.models.commonrstanza.pipeline._constantsrstanza.pipeline.processorrrr r rrrr s