zrodlo tokenizer train
Train a byte-level BPE tokenizer on text files and write it as a new tokenizer.json.
Usage#
zrodlo tokenizer train --text a.txt [--text b.txt ...] --vocab-size <N> --min-frequency <N> [--special-token '<|endoftext|>' ...] --output tokenizer.jsonFlags#
| Flag | Required | Description |
|---|---|---|
--text | yes, repeatable | A text file trained on; repeatable |
--vocab-size | yes | The size of the vocabulary, counting the byte symbols and the special tokens |
--min-frequency | yes | The fewest times a pair of symbols must occur to be merged |
--special-token | repeatable | A token kept whole, such as the end-of-sequence token; repeatable |
--output | yes | The new tokenizer.json file to write; it must not exist |
What it does#
Trains a byte-level BPE, GPT-2's tokenization: every byte has a symbol, so no text is out of vocabulary; pairs occurring at least --min-frequency times are merged until the vocabulary holds --vocab-size tokens or no pair is left; each --special-token is kept whole. A special token named twice is added once, and one that is already a byte symbol keeps that symbol's id, so the least --vocab-size accepted is the 256 byte symbols plus the distinct special tokens that are not byte symbols. It writes a new Hugging Face tokenizer.json and reports it as tokenizer inspect does.
Refusals#
Each refusal prints Error: and one of these, with the values it read in place of <…>; nothing is written:
tokenizer train needs at least one --text file<path> already exists; tokenizer train writes a new file and overwrites nothingtext <path> cannot be read: <error>, for a--textthat does not exist or cannot be opened for reading (its mode, or a folder the system guards), before anything is trained--vocab-size <n> is smaller than the <n> byte symbols and <n> distinct special tokens that are not byte symbols a byte-level BPE starts from; pass --vocab-size <n> or more