o h@sdZddlZddlmZddlZddlZddlZddlmZddl m Z ddl m Z e Z ddZd d Zd d Zd dZedkrFedSdS)a Builds a self-training dataset from an Italian data source and two models The idea is that the top down and the inorder parsers should make somewhat different errors, so hopefully the sum of an 86 f1 parser and an 85.5 f1 parser will produce some half-decent silver trees which can be used as self-training so that a new model can do better than either. The dataset used is PaCCSS, which has 63000 pairs of sentences: http://www.italianlp.it/resources/paccss-it-parallel-corpus-of-complex-simple-sentences-for-italian/ N)deque)FoundationCache) selftrain)get_tqdmcCsPtjdd}t||jdddd|jddd d d |jd d |}|S)NzFScript that converts part of a wikipedia dump to silver standard trees) descriptionz --input_dirz%extern_data/vietnamese/wikipedia/textz is its own item in the list. Lines are separated by to give hints to the stanza tokenizer. The first line after is skipped as it is usually the document title. Nzs"    )