o h @sRdZddlmZddlmZddlmZmZddZeedGdd d eZ d S) z. Processors related to spaCy in the pipeline. )doc)TOKENIZE)ProcessorVariantregister_processor_variantcCs&zddl}WdStytdw)zI Import necessary components from spaCy to perform tokenization. rNzlspaCy is used but not installed on your machine. Go to https://spacy.io/usage for installation instructions.T)spacy ImportError)rrY/var/www/html/env_mimamsha/lib/python3.10/site-packages/stanza/pipeline/external/spacy.py check_spacy s  r rc@seZdZddZddZdS)SpacyTokenizercCs|ddkr tdz ddl}ddlm}Wn ty!tdw||_|jdr7|j|j d n|jd | d d |_ dS) zJ Construct a spaCy-based tokenizer by loading the spaCy pipeline. langenz>spaCy tokenizer is currently only allowed in English pipeline.rN)EnglishzqspaCy 2.0+ is used but not installed on your machine. Go to https://spacy.io/usage for installation instructions.z2. sentencizer no_ssplitF) Exceptionr spacy.lang.enrrnlp __version__ startswithadd_pipe create_pipegetr)selfconfigrrrrr __init__s    zSpacyTokenizer.__init__c Cst|tjr |j}n|}t|tstd||}g}|jD]0}g}|D]$}tj|jtj tj d|j dtj d|j t |ji}||q%||q|jr[dd|Dg}t||S)z^ Tokenize a document with the spaCy tokenizer and wrap the results into a Doc object. zFMust supply a string or Stanza Document object to the spaCy tokenizer.=|cSsg|] }|D]}|qqSrr).0strrr Hsz*SpacyTokenizer.process..) isinstancerDocumenttextstrrrsentsTEXTMISC START_CHARidxEND_CHARlenappendr) rdocumentr$ spacy_doc sentencessenttokenstok token_entryrrr process0s$    .   zSpacyTokenizer.processN)__name__ __module__ __qualname__rr5rrrr r s r N) __doc__stanza.models.commonrstanza.pipeline._constantsrstanza.pipeline.processorrrr r rrrr s