Usage#

zrodlo corpus prepare --tokenizer tokenizer.json --eos-token '<|endoftext|>' [--text a.txt ...] [--jsonl data.jsonl ... --field text] --output corpus/

Flags#

Flag	Required	Description
`--tokenizer`	yes	The tokenizer.json the documents are encoded with: the model's own
`--eos-token`	yes	The tokenizer's token that ends every document
`--text`	repeatable	A text file read as one document; repeatable
`--jsonl`	repeatable	A JSON Lines file read as one document per line; repeatable, read after every --text
`--field`	with `--jsonl`	The key of each JSON Lines object that holds its document's text
`--output`	yes	The new corpus directory to write; it must not exist, its parent must

What it does#

Encodes every --text file as one document, then every line of every --jsonl file as one document (the string under --field, the layout Hugging Face datasets writes with to_json), ends each with --eos-token, and writes tokens.bin and corpus.json to a new directory. A JSON Lines file is read line by line, so it need not fit in memory. The layout of both files is in What Zrodlo reads and writes.

Refusals#

It creates only the output directory, so a refusal leaves nothing behind:

corpus prepare needs at least one --text or --jsonl file
<path> already exists; corpus prepare writes a new directory and overwrites nothing
<path> cannot be created: its parent <path> is not an existing directory, and Zrodlo creates only the directory it writes, never a missing parent; create <path> first
tokenizer <path> has no token <token> to mark the end of a document
text <path> cannot be read as UTF-8
document <name> encodes to no tokens; remove it or give it content
--jsonl needs --field, the key of each line's object that holds its text
--field <name> names the text key of --jsonl files, and no --jsonl file is given
line <file>:<line> cannot be read as UTF-8, line <file>:<line> is not JSON and line <file>:<line> holds no string field <name>, naming the JSON Lines file and line