o h@sddZddlZddlZddlmZddlmZejjejj gZ ej ddddZd d Z d d Z dS) zD Test the MWT resplitting of preexisting tokens without word splits N) resplit_mwt)TEST_MODELS_DIRmodule)scopecCstjdtdddS)z1 A reusable pipeline with the NER module enz tokenize,mwtgum)dir processorspackage)stanzaPipelinerr r V/var/www/html/env_mimamsha/lib/python3.10/site-packages/stanza/tests/mwt/test_utils.pypipelinesrcCs<gdddgg}t||}t|jdksJt|jdjdks"Jt|jdjdjdks1J|jdjdjdjdksAJ|jdjdjdjd ksQJt|jdjdks]Jt|jdjdjd kslJ|jdjdjdjd ks|J|jdjdjdjdksJ|jdjdjdjd ksJd S) z7 Test splitting with enforced token boundaries Izcan'tbelieveitI can'tsleeprcan'trNrlen sentencestokenswordstextrrdocr r rtest_resplit_keep_tokenss     $r$cCs gdddgg}t||dd}t|jdksJt|jdjdks$Jt|jdjd jdks3J|jdjd jdjd ksCJ|jdjd jd jd ksSJt|jd jd ks_Jt|jd jd jdksnJ|jd jd jdjd ks~J|jd jd jd jd ksJd S)z: Test splitting without enforced token boundaries rrrF) keep_tokensrrrrrrrNrr"r r rtest_resplit_no_keep_tokens,s   $r&)__doc__pytestr stanza.models.mwt.utilsr stanza.testsrmarkrtravis pytestmarkfixturer$r&r r r rs