o h*@sdZddlZddlZddlmZddlmZmZmZiddddd d d d d d ddddddddddddddddddddd d!d"d#id$d#d%d&d'd#d(d)d*d+d,d-d.d/d0d1d2d1d3d1d4d5d6d7d8d7d9d:d;d<d=d>d?d-dd@dAdBddCdDZ dEdFZ dGdHZ dIdJZ e egdKee Ze egdLee ZdXdOdPZdYdQdRZdSdTZdUdVZedWkredSdS)ZaF Script for processing the VTB files and turning their trees into the desired tree syntax The VTB original trees are stored in the directory: VietTreebank_VLSP_SP73/Kho ngu lieu 10000 cay cu phap The script requires two arguments: 1. Original directory storing the original trees 2. New directory storing the converted trees N) defaultdict) read_treesMixedTreeErrorUnlabeledTreeErrorz(ADV-MDPz(RP-MDPz(MPDz(MDPz(MP z(NP z(MP(z(NP(z(Np(z(Np (z(NP (z(NLOCz(NP-LOCz(N-P-LOCz(N-p-locz(NPDOBz(NP-DOBz(NPSUBz(NP-SUBz(NPTMPz(NP-TMPz(PPLOCz(PP-LOCz(SBA z(SBAR z(SBA-z(SBAR-z(SBA(z(SBAR(z(SBASz(SBARz(SABRz(SE-SPLz(S-SPLz(SBARRPPADVzPP-ADVz(PR (z(PP (z(PPPz(PPVP0ADVzVP-ADVz(S1z(Sz(S2z(S3zBP-SUBzNP-SUBAPPPDzAP-PPDAPPRDzNp--HzNp-Hz(WPNPz(WHNPz(WHRPPz(WHRPz(PVz(WHRP u(WHPP (Pro-h nào))z(WHRP (Pro-h Sao))z(NP)z(Mpdz(Whadv u(Whpr (Pro-h nào))z(Whpr (Pro-h Sao))z(Tp-tmpz(YpcCs"tD] \}}|||}q|S)N) REMAPPINGitemsreplace)treeoldnewri/var/www/html/env_mimamsha/lib/python3.10/site-packages/stanza/utils/datasets/constituency/vtb_convert.py unify_labelGsrcCs4d}|D]}|dkr|d7}q|dkr|d8}q|S)z} Checks if the tree is properly closed :param tree: tree as a string :return: True if closed otherwise False r()r)r countcharrrrcount_paren_parityNs rcCs|ds |dr dSdS)a Check if a line being read is a valid constituent The idea is that some "trees" are just a long list of words with no tree structure and need to be eliminated. :param line: constituent being read :return: True if it has open OR closing parenthesis. rrTF) startswithendswith)linerrr is_valid_line]s r)WPYPSNPSTCUPCz(TPXpXPWHVPWHPRNOWHADV(SC ((VOC ((Adv ((SP (ADV-MDP(SPL(ADV ((V-MWE ()rrrr r!r"r#r$r%r(r)r*r+r,r-r.r/TFc Cs|rt}nt}tt}t|ddd}t|ddd} |} d} d} d} t| D]w\}}d|}|dkrzzz(ROOTemptyzEmpty tree in {} line {}z) runclosedz#Unclosed tree from {} line {}: |{}| extra_parenszGExtra parens at end of tree from {} line {} for having extra parens: {}RBKTz-RRB-LBKTz-LRB-multiplez4Multiple trees in one xml annotation from {} line {} untagged_leafz2Tree with non-preterminal leaves in {} line {}: {}z"Weird label {} from {} line {}: {} missing_idzMissing ID from {} at line {}z z mixedz1Mixed leaves and constituents from {} line {}: {} unlabeledz+Unlabeled nodes in tree from {} line {}: {}invalidzEInvalid tree error in {} line {}: |{}|, rejected because of line |{}|)WEIRD_LABELS_2023 WEIRD_LABELSrlistopen readlines enumeratejoinsplitrrintstripappendformatrr rlenall_leaves_are_preterminalsrfindwriterrr) orig_filenew_file fix_errorsconvert_bracketsupdated_tagset write_ids weird_labelserrorsreaderwritercontentr tree_id reading_treeline_idxrparityprocessed_trees bad_label weird_labelrrr convert_filers$          *V VVrecCstt}|D]3}tjtj|d\} } tj|| } | d} t|| ||||} | D] }||| |q-qt | dkrHt ddSt dt | }|rn|D]}t d|t d||t qVt |D]}t d|t ||fqpdS) Nr7z.mrgrzAll errors were fixed!zFound the following errors:z--------- %10s -------------z z%s: %d) rrEospathsplitextrJrIreextendrOkeysprintsorted) file_listnew_dirverboserUrVrWrXrZfilename base_name_new_path new_file_path new_errorserjrrr convert_filess,    rwcs*t}fdd|D}t||dS)Ncs.g|]}tj|ddkrtj|qS)rz.raw)rfrgrhrI).0forig_dirrr s.zconvert_dir..)rflistdirrw)r{rnrmrrzr convert_dirs r~cCsJtjdd}|jddd|jddd|}|j}|j}t||dS) z Converts files from the 2009 version of VLSP to .mrg files Process args, loop through each file in the directory and convert to the desired tree format z7Script that converts a VTB Tree into the desired format) descriptionr{z>The location of the original directory storing original trees )helprnz=The location of new directory storing the new formatted treesN)argparseArgumentParser add_argument parse_argsorg_dirrnr~)parserargsrrnrrrmainsr__main__)TFFF)FTFFF)__doc__rrf collectionsr&stanza.models.constituency.tree_readerrrrr rrrrlsetrErjrDrCrerwr~r__name__rrrrs        !$%5  c