o he @sZdZddlZddlmZddlmZddlmZmZddZ eedGd d d eZ dS) zo Processors related to SudachiPy in the pipeline. GitHub Home: https://github.com/WorksApplications/SudachiPy N)doc)TOKENIZE)ProcessorVariantregister_processor_variantcCs.z ddl}ddl}WdStytdw)zM Import necessary components from SudachiPy to perform tokenization. rNzBoth sudachipy and sudachidict_core libraries are required. Try install them with `pip install sudachipy sudachidict_core`. Go to https://github.com/WorksApplications/SudachiPy for more information.T) sudachipysudachidict_core ImportError)rrr ]/var/www/html/env_mimamsha/lib/python3.10/site-packages/stanza/pipeline/external/sudachipy.pycheck_sudachipy s  r rc@seZdZddZddZdS)SudachiPyTokenizercCsR|ddkr tdtddlm}ddlm}||_|dd|_d S) zx Construct a SudachiPy-based tokenizer. Note that this tokenizer uses regex for sentence segmentation. langjaz:SudachiPy tokenizer is only allowed in Japanese pipelines.r) tokenizer) dictionary no_ssplitFN) Exceptionr rrr Dictionarycreategetr)selfconfigrrr r r __init__s   zSudachiPyTokenizer.__init__c Cst|tjr |j}n|}t|tstd|j|}g}g}|D];}|}| r,q!| }| } tj |tj tjd|dtjd| i} || |js\|dvr\||g}q!t|dkrh||t||S)zb Tokenize a document with the SudachiPy tokenizer and wrap the results into a Doc object. zJMust supply a string or Stanza Document object to the SudachiPy tokenizer.=|)u。u！u？!?r) isinstancerDocumenttextstrrrtokenizesurfaceisspacebeginendTEXTMISC START_CHAREND_CHARappendrlen) rdocumentrtokens sentencescurrent_sentencetoken token_textstartr% token_entryr r r process-s2         zSudachiPyTokenizer.processN)__name__ __module__ __qualname__rr4r r r r r s r ) __doc__restanza.models.commonrstanza.pipeline._constantsrstanza.pipeline.processorrrr r r r r r s