o h@s\dZddlZddlZddlZddZddZddZdd d Zd dZe dkr,edSdS)a From a directory of files with VTB Trees, split into train/dev/test set with a split of 70/15/15 The script requires two arguments 1. org_dir: the original directory obtainable from running vtb_convert.py 2. split_dir: the directory where the train/dev/test splits will be stored NcCstt|}t||S)z This function creates the random order with which we use to loop through the files :param org_dir: original directory storing the files that store the trees :return: list of file names randomly shuffled )sortedoslistdirrandomshuffle)org_dir file_namesr g/var/www/html/env_mimamsha/lib/python3.10/site-packages/stanza/utils/datasets/constituency/vtb_split.pycreate_shuffle_lists r cCs\|sd}n |ds|d}tj|d|}tj|d|}tj|d|}|||fS)z This function creates the necessary paths for the train/dev/test splits :param split_dir: directory that stores the splits :return: train path, dev path, test path _z %strain.mrgz %sdev.mrgz %stest.mrg)endswithrpathjoin) split_dir short_name train_pathdev_path test_pathr r r create_pathss  rc Cstd}|D]3}|ds qtj||}t|ddd}|}|D]}|d7}q!Wdn1s2wYq|S)z Function for obtaining the number of samples :param org_dir: original directory storing the tree files :param file_names: list of file names in the directory :return: number of samples r.mrgrutf-8encodingN)rrrropen readlines)rrcountfilenamefile_dirreadercontentliner r r get_num_samples/s  r%ffffff?333333?c Csztj|dd||dkrtd||t|}t||\}}} t||} td| |t| |} ||dkrO| } | | f} ||f}td| | | n5||dkrut| ||} | | | f} ||| f}td| | | | | nd } | f} | f}td | d }g}|D]:}|d sqt tj ||d d }| }dd|D}dd|D}| |Wdn1swYq|dur|d d krt||d |d}| }||||d|||d}t|}t|| D]B\}}t |dd d .}||kr*t|d}|durtd|||d|d7}||ks Wdn 1s5wYqdS)NT)exist_okg?z>Not making a test slice with the given ratios: train {} dev {}zFound {} total lines in {}zSplitting {} train, {} devgz#Splitting {} train, {} dev, {} testrzCopying all {} lines to testrrrcSsg|]}|qSr )strip.0xr r r szsplit_files..cSsg|]}|r|qSr r r*r r r r-tr.rwz9Ran out of trees before reading all of the expected trees )rmakedirsprintformatr rr%intrrrrrextendleniterzipnext RuntimeErrorwrite)rrr train_sizedev_sizerotationrrrr num_samples stop_trainstop_dev output_limits output_namesrtreesr r" new_treesrotation_start rotation_end tree_iter write_path count_limitwriter next_treer r r split_filesFsj         $      rMcCsTtjdd}|jddd|jddd|}|j}|j}tdt||d S) z Main function for the script Process args, loop through each tree in each file in the directory and write the trees to the train/dev/test split with a split of 70/15/15 zHScript that splits a list of files of vtb trees into train/dev/test sets) descriptionrzMThe location of the original directory storing correctly formatted vtb trees )helprzs   C