Resumes that drift
A run stopped halfway rarely resumes to the same weights, so the finished model is not the one the plan described.
Wisent Zrodlo
Pretrain your own language model, exactly reproducibly.
Zrodlo is one command-line program that trains every weight of a GPT-2 or Llama decoder: it trains a tokenizer on your text, prepares a token corpus, initializes or loads a model, trains it with AdamW and checkpoints it, and writes a Transformers model directory. A resumed run reaches the same losses, weights and optimizer state as one that was never interrupted.
- GPT-2 and Llama
- Tokenizer to Transformers model
- Exact resume
- bf16 on Metal and CUDA
- One Rust binary
A pretraining run you cannot repeat is a model you cannot explain.
Pretraining your own model usually means stitching a tokenizer, a corpus pipeline, an optimizer and checkpoints together by hand.
Half-written outputs
A failed step leaves directories and files behind that the next run trips over.
Unnamed draws
Shuffled windows and dropout masks come from a generator nobody records, so nobody can say what the model read.
From your text to a Transformers model.
Every run follows the same commands, and every number in them is one you chose.
Train a tokenizer
zrodlo tokenizer train writes a byte-level BPE tokenizer.json from your text files, GPT-2's tokenization.
Prepare a corpus
zrodlo corpus prepare encodes text and JSON Lines files into one token stream read from disk page by page; corpus split sets the last documents aside as held-out text.
Initialize a model
zrodlo model init draws every weight of a GPT-2 or Llama decoder from a seed, from a Transformers config.json.
Train with checkpoints
zrodlo model train updates every weight with AdamW, a warmup and a cosine or linear decay, clipping and held-out scores, and writes checkpoints.
Resume exactly
zrodlo checkpoint resume continues a checkpoint to the same losses, weights and optimizer state as the run that was never interrupted.
A real run, and its exact resume.
Captured output of Zrodlo on the fixture texts of its own repository: a tokenizer, a corpus, a tiny GPT-2, four training steps, and a resume from step 2.
$ zrodlo tokenizer train --text train.txt --text train-more.txt --vocab-size 257 --min-frequency 2 --special-token <|endoftext|> --output tokenizer.json
special_tokens.0.content: <|endoftext|>
vocab_size: 257
$ zrodlo corpus prepare --tokenizer tokenizer.json --eos-token <|endoftext|> --text train.txt --text train-more.txt --output corpus
manifest.token_count: 452
$ zrodlo model init --config config.json --tokenizer tokenizer.json --seed 7 --output base
parameters: 102714
$ zrodlo model train --model base --corpus corpus --output run --steps 4 --context 24 --batch-size 2 ... --checkpoint-every 2 --device cpu
step 1/4 loss 10.821585 learning_rate 0.001 gradient_norm 0.24548672204991842
step 2/4 loss 10.822021 learning_rate 0.001 gradient_norm 0.6257491792928925
checkpoint run/step-2
step 3/4 loss 10.824739 learning_rate 0.001 gradient_norm 0.6193788284506654
step 4/4 loss 10.813206 learning_rate 0.001 gradient_norm 0.7362186539267707
checkpoint run/step-4
$ zrodlo checkpoint resume --checkpoint run/step-2 --corpus corpus --output resumed --steps 4 --device cpu
step 3/4 loss 10.824739 learning_rate 0.001 gradient_norm 0.6193788284506654
step 4/4 loss 10.813206 learning_rate 0.001 gradient_norm 0.7362186539267707
checkpoint resumed/step-4
steps_done: 4
$ cmp run/step-4/model.safetensors resumed/step-4/model.safetensors && echo identical
identicalWhat researchers ask first.
Which models can it train?
GPT-2 and Llama decoders, chosen by the model_type of a Transformers config.json. The Llama model computes RMS norms, rotary embeddings, grouped-query attention and a SiLU-gated feed-forward layer as Transformers does.
What hardware does it need?
The CPU, or a GPU through Metal or CUDA. bf16 mixed precision runs on a GPU, with weights and optimizer moments kept in f32.
What happens when something is wrong?
The command is refused naming the file or flag and the values it read, and nothing is written or removed first. A command that writes a directory creates only that directory and removes it when refused.
Does it need other services?
No. Zrodlo is one Rust binary built on Candle; preparing data, training, resuming and evaluating involve no gateway, vault, fleet or service.
Install Zrodlo and pretrain your first model.
One binary, from Stado or from source, writes the Transformers model directory Ster then posttrains.