zrodlo model train
Train every weight of a model directory, from scratch or continued.
Usage#
zrodlo model train --model base/ --corpus corpus/ --output run/ --steps <N> --context <TOKENS> --batch-size <WINDOWS> \
--learning-rate <LR> --beta1 <B1> --beta2 <B2> --epsilon <EPS> --weight-decay <WD> --decay-scope all|matrices --seed <SEED> \
--precision f32|bf16 [--micro-batch-size <WINDOWS>] [--warmup-steps <N>] [--stable-steps <N>] \
[--decay-steps <N> --decay-shape cosine|linear --min-learning-rate <LR>] [--clip-norm <NORM>] [--shuffle] \
[--eval-corpus held-out/ --eval-every <N>] [--checkpoint-every <N>] [--keep-checkpoints <N>] --device cpu|metal|cuda [--ordinal <N>]Flags#
| Flag | Required | Description |
|---|---|---|
--model | yes | The model directory or checkpoint whose weights are trained |
--corpus | yes | The prepared corpus the windows are read from |
--output | yes | The run directory each step-<n> checkpoint is written into |
--steps | yes | How many optimizer updates the run makes |
--context | yes | Tokens per training window; the model reads all but the last |
--batch-size | yes | Windows per optimizer update |
--micro-batch-size | no | Windows per forward pass, dividing --batch-size; the step's gradient is the sum over its micro-batches |
--learning-rate | yes | AdamW's step size: a positive finite number |
--beta1 | yes | AdamW's decay of the gradient's running mean, between 0 and 1 |
--beta2 | yes | AdamW's decay of the squared gradient's running mean, between 0 and 1 |
--epsilon | yes | AdamW's term added to the denominator: a positive finite number |
--weight-decay | yes | AdamW's decoupled weight decay: a finite number, zero or more |
--decay-scope | yes | The weights --weight-decay pulls toward zero: all, or matrices only (no biases or norm scales) |
--warmup-steps | no | Updates over which the learning rate rises linearly to --learning-rate |
--stable-steps | no | Updates after the warmup at the full rate before the decay starts; needs --decay-steps |
--decay-steps | no | Updates over which the rate falls in --decay-shape to --min-learning-rate |
--decay-shape | with --decay-steps | How the rate falls over --decay-steps: cosine or linear; required with it |
--min-learning-rate | with --decay-steps | The rate the decay ends at and keeps; required with --decay-steps |
--clip-norm | no | The largest global L2 norm a step's gradient keeps; a larger one is scaled down to it |
--checkpoint-every | no | Steps between checkpoints written under --output; the last step is always written |
--keep-checkpoints | no | Keep only the newest this many checkpoints under --output, removing older ones as new ones are written |
--seed | yes | Keys the dropout masks and the shuffled window order; a resumed run keeps it |
--shuffle | switch | Read each pass over the corpus in an order drawn from --seed and the pass |
--precision | yes | The number type every pass computes in: f32, or bf16 for mixed precision; a resumed run keeps it |
--eval-corpus | with --eval-every | A prepared held-out corpus scored during the run; needs --eval-every |
--eval-every | with --eval-corpus | Steps between held-out scores, and the last step; needs --eval-corpus |
--device | yes | The device to compute on |
--ordinal | with metal or cuda | The GPU to use; required for metal and cuda |
The values a flag with a fixed set takes:
| Flag | Value | Description |
|---|---|---|
--decay-scope | all | Every weight decays |
--decay-scope | matrices | Only weights of two or more dimensions decay; biases and norm scales do not |
--decay-shape | cosine | Along half a cosine period: slowly first, fastest midway, slowly last |
--decay-shape | linear | By the same amount every update |
--precision | f32 | Every operation in 32-bit floats |
--precision | bf16 | Weights cast to bfloat16 for the pass; norms and the loss in f32 |
--device | cpu | The CPU; takes no --ordinal |
--device | metal | An Apple GPU through Metal; needs a binary built with the metal feature and --ordinal |
--device | cuda | An NVIDIA GPU through CUDA; needs a binary built with the cuda feature and --ordinal |
What it does#
Updates every weight of any model directory with AdamW on next-token cross-entropy. A --context window holds that many tokens: the model reads all but the last and is scored on all but the first. Step n reads windows n × batch onward, in corpus order or, with --shuffle, in an order drawn per pass from --seed. --micro-batch-size reads a step's batch in equal parts whose gradients add up to the batch's. The rate rises over --warmup-steps, holds for --stable-steps, then falls over --decay-steps to --min-learning-rate along a cosine or a line. --clip-norm scales down a larger global gradient norm. --decay-scope matrices leaves biases and norm scales undecayed. Progress names each step's loss, learning rate, gradient norm and tokens per second; with --eval-corpus and --eval-every it scores the held-out corpus during the run. It writes run/step-<n>/ every --checkpoint-every steps and at --steps, and with --keep-checkpoints keeps only the newest.
--precision bf16 is mixed precision on a GPU: weights and AdamW moments stay f32, norms, the attention softmax and the loss are computed in f32. On the CPU it is refused before anything is written.
Refusals#
It refuses before any weight changes, except for a non-finite loss, which is met only while training. Each refusal prints Error: and one of these, with the values it read in place of <…>:
--learning-rate is <value>; it must be a positive finite number, the same for--epsilon, and--beta1 is <value>; a moment decay lies above zero and below 1, the same for--beta2--weight-decay is <value>; it must be a finite number that is not negative--clip-norm is <value>; it must be a positive finite number--decay-steps needs --min-learning-rate, the rate the decay ends atand--min-learning-rate needs --decay-steps, the updates the rate decays over--decay-steps needs --decay-shape, cosine or linear, the way the rate fallsand--decay-shape shapes a decay, and no --decay-steps is given--stable-steps holds the rate before a decay, and no --decay-steps is given; without a decay the rate stays at --learning-rate--min-learning-rate is <value>; it must be a finite number from zero up to --learning-rate <value>--micro-batch-size <n> does not divide --batch-size <n>; every micro-batch of a step holds the same number of windows--context <n> leaves no token to read before the one predicted; it needs two or morethe corpus holds <n> tokens, fewer than one window of --context <n>--context <n> makes the model read <n> tokens, more than the <n> positions its configuration states<path> was encoded with another tokenizer than <path>: its corpus.json records tokenizer <digest>, and <path> has digest <digest>; prepare the corpus again with <path>, for the corpus and for--eval-corpusthe corpus holds token <n>, outside the model's vocab_size <n>--eval-corpus needs --eval-every, the steps between held-out scoresand--eval-every needs --eval-corpus, the held-out corpus to scoreinvalid value '<value>' for '--device <DEVICE>', from the parser, with the values it accepts;--device metal needs --ordinal, the card to train on, the same forcuda;--ordinal names a GPU; the cpu device takes none--precision bf16 needs a GPU: the CPU backend has no bfloat16 matrix product; run with --device metal or --device cuda and --ordinal, or with --precision f32<path> already exists; Zrodlo writes a new directory and overwrites nothing, for a checkpoint directory;<path> exists and is not a directory; a run writes its checkpoints into a directory; and the missing-parent refusal of model init- with
--keep-checkpoints:<path> is named like a checkpoint, but <name> is no step count,<path> records <n> steps, not the <n> its directory name states, aconfig.jsonthat does not load with its own sentence, and<path> holds <name>, which Zrodlo does not write into a checkpoint directory; move it out, then delete the checkpoint again the step after step <n> produced loss <loss>; lower --learning-rate or check the corpus