zrodlo corpus split
Write a corpus's last documents to a new held-out corpus and the rest to a new training corpus, without encoding them again.
Usage#
zrodlo corpus split --corpus corpus/ --held-out-documents <N> --train-output train/ --held-out-output held-out/Flags#
| Flag | Required | Description |
|---|---|---|
--corpus | yes | The corpus directory to split |
--held-out-documents | yes | How many of the corpus's last documents the held-out corpus takes |
--train-output | yes | The new training corpus directory, holding every document but the held-out ones |
--held-out-output | yes | The new held-out corpus directory, holding the last --held-out-documents |
What it does#
Writes the last documents to a new held-out corpus and the rest to a new training corpus without encoding anything again, so the two streams read one after the other are the source's.
Refusals#
It refuses a corpus corpus inspect refuses, with that command's sentence, and a --held-out-documents of zero as a usage error. A failure while writing removes what was written:
--held-out-documents <n> leaves none of the <n> documents of <path> to train on--train-output <path> and --held-out-output <path> are the same directory or one lies inside the other; a split writes two separate corpora<path> already exists; corpus split writes new directories and overwrites nothing<path> must exist to hold <path>, for an output whose parent does not exist