o
–h» ã @ s¤ d Z ddlZddlZddlZddlmZ ddlmZ ddlm Z ej
j ej
j gZ
dZdZdZd Zd
Zd Zd Zd
Zdd„ Zd ¡ Zdd„ Zdd„ Zdd„ Zdd„ Z dS )z
A few tests for Vietnamese parsing, which has some difficulties related to spaces in words
Technically some other languages can have this, too, like that one French token
é N)Úpretrain)Ú tree_reader)Ú
build_traineruŒ (ROOT (S-TTL (NP (" ") (N-H Äảo) (Np Äà i Loan) (" ") (PP (E-H ở) (NP (N-H đồng bằng) (NP (N-H sông) (Np Cá»u Long))))) (. .)))uŒ (ROOT (S-TTL (NP (" ") (N-H Äảo) (Np Äà i_Loan) (" ") (PP (E-H ở) (NP (N-H đồng_bằng) (NP (N-H sông) (Np Cá»u_Long))))) (. .)))u€ (ROOT (S (NP (" ") (N Äảo) (Np Äà i Loan) (" ") (PP (E ở) (NP (N đồng bằng) (NP (N sông) (Np Cá»u Long))))) (. .)))uœ (ROOT (S-TTL (NP (PUNCT -LRB-) (N-H Äảo) (Np Äà i Loan) (PUNCT -RRB-) (PP (E-H ở) (NP (N-H đồng bằng) (NP (N-H sông) (Np Cá»u Long))))) (. .)))uœ
(S-TTL (NP (PUNCT LBKT) (N-H Äảo) (Np Äà i_Loan) (PUNCT RBKT) (PP (E-H ở) (NP (N-H đồng_bằng) (NP (N-H sông) (Np Cá»u_Long))))) (. .))
u¢
(S-TTL (NP (PUNCT LBKT) (N-H Äảo) (Np Äà i_Loan) (PUNCT RBKT) (PP (E-H ở) (NP (N-H đồng_bằng) (NP (N-H sông) (Np Cá»u_Long))))) (. .))
u£
(S-TTL (NP (PUNCT LBKT) (N-H Äảo) (Np Äà i_Loan) (PUNCT RBKT) (PP (E-H ở) (NP (N-H đồng_bằng) (NP (N-H sông) (Np Cá»u_Long))))) (. .))
uÅ (_ROOT (_S (_NP (_" " )_" (_N Äảo )_N (_Np Äà i_Loan )_Np (_" " )_" (_PP (_E ở )_E (_NP (_N đồng_bằng )_N (_NP (_N sông )_N (_Np Cá»u_Long )_Np )_NP )_NP )_PP )_NP (_. . )_. )_S )_ROOTc C sz t d¡d } t | ¡}t|ƒdksJ ‚t|d ƒ| ks J ‚|d jd jd jd }| ¡ s1J ‚|jd jdks;J ‚dS )z`
Test that an individual tree with spaces in the leaves is being processed as we expect
Ú
r é é u
ÄÃ i LoanN) Ú VI_TREEBANKÚsplitr Ú
read_treesÚlenÚstrÚchildrenÚis_preterminalÚlabel)ÚtextÚtreesÚnode© r úd/var/www/html/env_mimamsha/lib/python3.10/site-packages/stanza/tests/constituency/test_vietnamese.pyÚtest_read_vi_tree! s
r u”
4 4
Äảo 0.11 0.21 0.31 0.41
ÄÃ i Loan 0.12 0.22 0.32 0.42
đồng bằng 0.13 0.23 0.33 0.43
sông 0.14 0.24 0.34 0.44
c
C sà t d¡d } t | ¡}t|d ¡ ƒ}t ¡ E}tj
|d¡}tj
|d¡}t |ddd
}| t
¡ W d ƒ n1 s>w Y tj||d d
}| ¡ t|ƒ}|j} W d ƒ n1 s`w Y | |¡d ksnJ ‚dS ) zT
Test that a VI embedding's words are correctly found when processing trees
r r zemb.txtzemb.ptÚwzutf-8)ÚencodingNT)ÚfilenameÚ vec_filenameÚ save_to_fileé )r r r r
ÚsetÚ leaf_labelsÚtempfileÚTemporaryDirectoryÚosÚpathÚjoinÚopenÚwriteÚ VI_EMBEDDINGr ÚPretrainÚloadr ÚmodelÚnum_words_known)
r r ÚwordsÚtempdirÚ emb_filenameÚ pt_filenameÚfoutÚptÚtrainerr( r r r Útest_vi_embedding7 s
ÿ ÷ r1 c C sl t d¡d } t | ¡}t|ƒdksJ ‚t|d ƒ| ks J ‚d |d ¡t ks)J ‚d |d ¡tks4J ‚dS )z^
By default, spaces are left as spaces, but there is a format option to change spaces
r r r z{}z{:_O}N)r r r r
r r ÚformatÚVI_TREEBANK_UNDERSCORE©r r r r r Útest_space_formattingM s
r5 c C s t d¡d } t | ¡}t|ƒdksJ ‚t|d ƒ| ks J ‚d |d ¡tks)J ‚d|d _d |d ¡t ks9J ‚d|d _d |d ¡t
ksIJ ‚t d¡d }t t
¡
d |¡ W d ƒ n1 sew Y t d
¡d }t t
¡ d |¡ W d ƒ d S 1 s‡w Y d S ) Nr r r z{:_V}é2 z{:_Vi}éd z(ROOT)z{:V}z(ROOT (1) (2) (3)))ÚVI_TREEBANK_PARENr r r
r r r2 ÚVI_TREEBANK_VLSPÚtree_idÚVI_TREEBANK_VLSP_50ÚVI_TREEBANK_VLSP_100ÚpytestÚraisesÚ
ValueError)r r ÚemptyÚbranchesr r r Útest_vlsp_formattingY s"
ÿ "ÿrB c C sh t d¡d } t | ¡}dd„ |D ƒ}t|ƒdksJ ‚t|d ƒtks%J ‚d |d ¡} | tks2J ‚dS )z?
Test turning the parse tree into a 'language' for GPT
r r c S s g | ]}| ¡ ¡ ‘qS r )Ú
prune_noneÚsimplify_labels)Ú.0Útr r r Ú
s s z,test_language_formatting..r z{:L}N) r r r r
r r ÚVI_TREEBANK_SIMPLEr2 ÚEXPECTED_LABELED_BRACKETSr4 r r r Útest_language_formattingm s
rJ ) Ú__doc__r r r= Ústanza.models.commonr Ústanza.models.constituencyr Ú&stanza.tests.constituency.test_trainerr ÚmarkÚpipelineÚtravisÚ
pytestmarkr r3 rH r8 r9 r; r< rI r Ústripr% r1 r5 rB rJ r r r r Ú s0 ú