o h@sdZddlZddlZddlZddlZddlmZddlmZm Z ej j ej j gZ ddZddZd d Zd d Zd dZddZdS)zF Test various resource downloading functions from resources/common.py N)common)TEST_MODELS_DIRTEST_WORKING_DIRc CsJtjtd}tj|d}tt t |Wdn1s#wYt |ddd }| dWdn1s?wYd}t |t j ||dtt t j |d dWdn1siwYtt t j |d d d Wdn1swYt j |d |d WddS1swYdS) Ndirztest.txtwzutf-8)encodingzUnban mox opal! 44dbf21b4e89cea5184615a72a825a36)md512345)r alternate_md5)tempfileTemporaryDirectoryrospathjoinpytestraisesFileNotFoundErrorrassert_file_existsopenwrite ValueError)test_dirfilenamefout EXPECTED_MD5r]/var/www/html/env_mimamsha/lib/python3.10/site-packages/stanza/tests/resources/test_common.pytest_assert_file_existss&      "rcCsztjtd-}tjd|ddddtjd|ddd}t|tjs"Jt|jdks+JWddS1s6wYdS) NrentokenizeewtF model_dir processorspackageverbose)r$r%r&) r rrstanzadownloadPipeline isinstancelenloaded_processors)rpipelinerrrtest_download_tokenize_mwt%s "r0c CstjtdZ}tjd|ddddtt|ddgksJtj |d}tt|}|gdks4Jttj |dd gksCJ|D]}t ttj ||d ksWJqEWd d S1scwYd S) z Test the download path for a single file rather than the default zip The expectation is that an NER model will also download two charlm models. If that layout changes on purpose, this test will fail and will need to be updated rr nerontonotes_charlmFr#resources.jsonbackward_charlmforward_charlmr1pretrainontonotes_charlm.ptN r rrr)r*sortedrlistdirrrr-ren_diren_dir_listingirrrtest_download_non_default-s""rAc Cstjtda}tjd|ddddgiddtt|ddgks"Jtj |d}tt|}|gd ks8Jtttj |dd d gksJJ|D]}t ttj ||d ks^JqLWd d S1sjwYd S)a Test the download path for two NER models The package system should now allow for multiple NER models to be specified, and a consequence of that is it should be possible to download two models at once The expectation is that the two different NER models both download a different forward & backward charlm. If that changes, the test will fail. Best way to update it will be two different models which download two different charlms rr r1r2anatemFr#r3r4z anatem.ptr8r(Nr:r=rrrtest_download_two_models?s $""rCcCstjtdy}td|dd\}}}}|dddksJ|dks"Jtd|ddid\}}}}|dddks9J|dks?Jtd|d d id\}}}}|dd dksVJ|dks\Jtd|d d\}}}}|d d dksqJ|dkswJWddS1swYdS) zD Test a few options for specifying which processors to load rr Nz tokenize,posdefault)r!posr!spacyrEr")r rrrprocess_pipeline_parameters)rlangr$r&r%rrr test_process_pipeline_parametersWs   "rIcCstt}d}||vrt|dkr|d}||vrt|dks||vs%Jt||dks/Jd|vs5Jd|dvs=J|dddksGJt|d}d|vsSJd|vsYJd|dvsaJt|d}||ksmJdS) Nzdzhaliaszzh-hansr!Chinese lang_name)rload_resources_jsonrr-get_language_resources) resourcesbad_lang zh_resourceszh_hans_resourcesrrrtest_language_resourcesls       rV)__doc__rrr r)stanza.resourcesr stanza.testsrrmarktravisclient pytestmarkrr0rArCrIrVrrrrs