1. Train a tokenizer#

A byte-level BPE, GPT-2's tokenization, with the end-of-sequence token kept whole. The least --vocab-size accepted is the 256 byte symbols plus each distinct special token that is not a byte symbol.

zrodlo tokenizer train --text train.txt --text more.txt \
  --vocab-size 1024 --min-frequency 2 --special-token '<|endoftext|>' \
  --output tokenizer.json

2. Prepare and split a corpus#

zrodlo corpus prepare --tokenizer tokenizer.json --eos-token '<|endoftext|>' \
  --text train.txt --text more.txt --output corpus/
zrodlo corpus split --corpus corpus/ --held-out-documents 1 \
  --train-output train/ --held-out-output held-out/

3. Initialize a model#

config.json is a Transformers configuration whose model_type is gpt2 or llama. A GPT-2 configuration states n_inner; Zrodlo builds only a width the file states.

zrodlo model init --config config.json --tokenizer tokenizer.json --seed 7 --output base/

4. Train with checkpoints#

zrodlo model train --model base/ --corpus train/ --output run/ --steps 20 \
  --context 32 --batch-size 4 --learning-rate 0.001 --beta1 0.9 --beta2 0.999 \
  --epsilon 1e-8 --weight-decay 0.01 --decay-scope all --seed 7 \
  --precision f32 --checkpoint-every 10 \
  --eval-corpus held-out/ --eval-every 10 --device cpu

Progress on stderr names each step's loss, learning rate, gradient norm and tokens per second, and each held-out score.

5. Resume and compare#

zrodlo checkpoint resume --checkpoint run/step-10/ --corpus train/ --output resumed/ --steps 20 --device cpu

resumed/step-20/ holds the same losses, weights and optimizer state as run/step-20/.

6. Evaluate and generate#

zrodlo model evaluate --model run/step-20/ --corpus held-out/ --context 32 --batch-size 4 --precision f32 --device cpu
zrodlo model generate --model run/step-20/ --prompt 'the cat' --tokens 8 --precision f32 --device cpu

Add --json to any command for one JSON document on stdout instead of field: value lines.