o h*@sdZddlmZmZddlmZddlmZmZm Z Gddde Z GdddeZ Gd d d eZ Gd d d e ZGd dde ZddZddZdS)z Base classes for processors )ABCabstractmethod)Document)NAME_TO_PROCESSOR_CLASSPIPELINE_NAMESPROCESSOR_VARIANTSc@sXeZdZdZddZeddZeddZedd Zed d Z d d Z ddZ dS)ProcessorRequirementsExceptionzA Exception indicating a processor's requirements will not be met cCs(||_|j||_||_|dSN)_err_processor err_processor mark_inactive_processors_list_provided_reqs build_message)selfprocessors_listr provided_reqsrT/var/www/html/env_mimamsha/lib/python3.10/site-packages/stanza/pipeline/processor.py__init__ s   z'ProcessorRequirementsException.__init__cC|jS)z) The processor that raised the exception )r rrrrr z,ProcessorRequirementsException.err_processorcCs t|jjSr )typer __name__rrrrprocessor_types z-ProcessorRequirementsException.processor_typecCrr )r rrrrrz.ProcessorRequirementsException.processors_listcCrr )rrrrrr"rz,ProcessorRequirementsException.provided_reqsc CsNd|jdd|jd|jjd|jj|jd|jj|jd |_dS)Nz---- Pipeline Requirements Error! Processor: z Pipeline processors list: ,z Processor Requirements: z - fulfilled: z - missing: z The processors list provided for this pipeline is invalid. Please make sure all prerequisites are met for every processor. )rjoinrr requires intersectionrmessagerrrrr&s  z,ProcessorRequirementsException.build_messagecCrr r!rrrr__str__0z&ProcessorRequirementsException.__str__N) r __module__ __qualname____doc__rpropertyr rrrrr#rrrrr s     rc@seZdZdZddZddZeddZdd Zd d Z d d Z ddZ ddZ e ddZe ddZe ddZe ddZddZdS) Processorz Base class for all processors cCs`||_||_||||s|||t|dr,|jj r.|jj |_ dSdSdSN_variant) _config _pipeline_set_up_variants_set_up_variant_requires_set_up_requires_set_up_provides_check_requirementshasattrr+OVERRIDEprocessrconfigpipelinedevicerrrr7s zProcessor.__init__cCs:|jj}d}|jdur|jd}|dur|Sd||S)zB Simple description of the processor: name(model) N model_pathz{}({})) __class__rr,getformat)rnamemodelrrrr#Is   zProcessor.__str__cCdS)z> Process a Document. This is the main method of a processor. Nrrdocrrrr5WszProcessor.processcs(tdr j|Sfdd|DS)h Process a list of Documents. This should be replaced with a more efficient implementation if possible. r+cg|]}|qSrr5.0rBrrr bz*Processor.bulk_process..)r3r+ bulk_processrdocsrrrrJ\s  zProcessor.bulk_processcC|jj|_dS)zf Set up what processor requirements this processor fulfills. Default is to use a class defined list. N)r;PROVIDES_DEFAULT _providesrrrrr1dzProcessor._set_up_providescCrM)zR Set up requirements for this processor. Default is to use a class defined list. N)r;REQUIRES_DEFAULT _requiresrrrrr0hrPzProcessor._set_up_requirescCsTt|dsdSt|jdr|j|jj|_dSt|jjdr(|jjj|_dSdS)z If this has a variant with its own requirements, use those instead Returns True iff the _requires is set from the _variant r+Fr0TrQ)r3r+r0rRr;rQrrrrr/ls     z"Processor._set_up_variant_requirescsdt|jjd}tfddt|Dr0d|_fddt|Dd}t|||_dSdS)Nrc3s"|] }d|dVqdS)with_FNr<rGvariantr7rr s z-Processor._set_up_variants..cs"g|] }d|dr|qS)rSFrTrUrWrrrHs"z.Processor._set_up_variants..)listr;rNanyr_trainerr+)rr7r9processor_name variant_namerrWrr.}s zProcessor._set_up_variantscCr)z" Configurations for the processor )r,rrrrr7rzProcessor.configcCr)z- The pipeline that this processor belongs to )r-rrrrr8rzProcessor.pipelinecCrr )rOrrrrprovidesrzProcessor.providescCrr )rRrrrrrrzProcessor.requirescCs`|jdds dStjdd|jjDtgg}|j|r.dd|jjD}t|||dS)zg Given a list of fulfilled requirements, check if all of this processor's requirements are met or not. check_requirementsTNcSsg|]}|jqSr)r^)rG processorrrrrHsz1Processor._check_requirements..cSsg|]}|dqS)rr)rGitemrrrrHs) r7r<setunionr8loaded_processorsr load_listr)rr load_namesrrrr2s"  zProcessor._check_requirementsN)rr%r&r'rr#rr5rJr1r0r/r.r(r7r8r^rr2rrrrr)4s(      r)c@s(eZdZdZdZeddZddZdS)ProcessorVariantz' Base class for all processor variants FcCr@)a Process a document that is potentially preprocessed by the processor. This is the main method of a processor variant. If `OVERRIDE` is set to True, all preprocessing by the processor would be bypassed, and the processor variant would serve as a drop-in replacement of the entire processor, and has to be able to interpret all the configs that are typically handled by the processor it replaces. NrrArrrr5s zProcessorVariant.processcsfdd|DS)rCcrDrrErFrrrrHrIz1ProcessorVariant.bulk_process..rrKrrrrJszProcessorVariant.bulk_processN)rr%r&r'r4rr5rJrrrrrgs   rgcsteZdZdZfddZeddZddZdd Ze d d Z e d d Z e ddZ e ddZddZZS) UDProcessorzb Base class for the neural UD Processors (tokenize,mwt,pos,lemma,depparse,sentiment,constituency) csHt|||d|_d|_d|_t|ds||||||dSr*)superr _pretrainr[_vocabr3 _set_up_model_set_up_final_configr6r;rrrs zUDProcessor.__init__cCsdSr rr6rrrrlszUDProcessor._set_up_modelcCsJ|jdur|jj|jj}|_dd|D}ni}||||_dS)zV Finalize the configurations for this processor, based off of values from a UD model. NcSs i|] \}}t|s||qSr)rhfilter_out_option)rGkvrrr s z4UDProcessor._set_up_final_config..)r[argsvocabrkitemsupdater,)rr7 loaded_argsrrrrms   z UDProcessor._set_up_final_configcCsd|_d|_dS)ze Drop memory intensive resources if keeping this processor around for reasons other than running it. N)r[rkrrrrr s zUDProcessor.mark_inactivecCrr )rjrrrrpretrainrzUDProcessor.pretraincCrr )r[rrrrtrainerrzUDProcessor.trainercCrr )rkrrrrrtrzUDProcessor.vocabcCs0gd}|ds|drdS||vrdSdS)z) Filter out non-processor configurations ) r9cpucudadev_conll_goldepochslangmode save_name shorthand_file_dirTF)endswith)optionoptions_to_filterrrrros zUDProcessor.filter_out_optioncCsht|dr |j|Sdd|D}tg}||_tdd|D|_tdd|D|_|||S)a Most processors operate on the sentence level, where each sentence is processed independently and processors can benefit a lot from the ability to combine sentences from multiple documents for faster batched processing. This is a transparent implementation that allows these processors to batch process a list of Documents as if they were from a single Document. r+cSsg|] }|jD]}|qqSr) sentences)rGrBsentrrrrHsz,UDProcessor.bulk_process..cs|]}|jVqdSr ) num_tokensrFrrrrXz+UDProcessor.bulk_process..csrr ) num_wordsrFrrrrXr) r3r+rJrrsumrrr5)rrLcombined_sents combined_docrrrrJs   zUDProcessor.bulk_process)rr%r&r'rrrlrmr r(rxryrt staticmethodrorJ __classcell__rrrnrrhs         rhc@s(eZdZdZddZddZddZdS) ProcessorRegisterExceptionzB Exception indicating processor or processor registration failure cCs||_||_|dSr )_processor_class_expected_parentr)rprocessor_classexpected_parentrrrr s z#ProcessorRegisterException.__init__cCsd|jd|jd|_dS)NzFailed to register 'z'. It must be a subclass of 'z'.)rrr!rrrrrsz(ProcessorRegisterException.build_messagecCrr r"rrrrr#r$z"ProcessorRegisterException.__str__N)rr%r&r'rrr#rrrrrs  rcsfdd}|S)Ncs*t|ts t|t|t<t|Sr ) issubclassr)rrrappendClsr>rrwrappers   z#register_processor..wrapperr)r>rrrrregister_processors rcsfdd}|S)Ncs$t|ts t|t|t<|Sr )rrgrrrr>rVrrrs   z+register_processor_variant..wrapperr)r>rVrrrrregister_processor_variantsrN)r'abcrrstanza.models.common.docrstanza.pipeline.registryrrr Exceptionrr)rgrhrrrrrrrs *lP