zrodlo checkpoint resume
Continue a checkpoint with its own optimizer state and data position.
Usage#
zrodlo checkpoint resume --checkpoint run/step-<N>/ --corpus corpus/ --output resumed/ --steps <TOTAL> \
[--eval-corpus held-out/ --eval-every <N>] [--checkpoint-every <N>] [--keep-checkpoints <N>] --device cpu|metal|cuda [--ordinal <N>]Flags#
| Flag | Required | Description |
|---|---|---|
--checkpoint | yes | The step-<n> checkpoint directory to continue |
--corpus | yes | The corpus the checkpoint was trained on |
--output | yes | The run directory each new step-<n> checkpoint is written into |
--steps | yes | Total steps the run reaches, counting the checkpoint's own |
--checkpoint-every | no | Steps between checkpoints written under --output; the last step is always written |
--keep-checkpoints | no | Keep only the newest this many checkpoints under --output |
--eval-corpus | with --eval-every | A prepared held-out corpus scored during the run; needs --eval-every |
--eval-every | with --eval-corpus | Steps between held-out scores, and the last step; needs --eval-corpus |
--device | yes | The device to compute on |
--ordinal | with metal or cuda | The GPU to use; required for metal and cuda |
--device takes cpu (The CPU; takes no --ordinal), metal (An Apple GPU through Metal; needs a binary built with the metal feature and --ordinal) or cuda (An NVIDIA GPU through CUDA; needs a binary built with the cuda feature and --ordinal).
What it does#
Continues a checkpoint to --steps total with the checkpoint's own optimizer state, hyperparameters, schedule, clipping, precision, window order, window, batch size and seed, on the same corpus and tokenizer; the result equals the run that was never interrupted. Every seeded draw comes from ChaCha20 through draws written in Zrodlo itself, and trainer.json names the generator.
Refusals#
It refuses before any weight changes, and with --keep-checkpoints refuses the checkpoints model train refuses, with its sentences:
<path> cannot be read; only a checkpoint can be resumed, for a directory withouttrainer.json<path> is not the tokenizer <path> was trained with<path> holds corpus <digest>, and this checkpoint was trained on corpus <digest>; resume on the same corpus--steps <n> is not past the <n> steps already donethis checkpoint's <n> steps drew their window order and dropout masks from <generator>, and this build draws from <generator>; a resume would not continue the same run, for a run that shuffled or drew dropout with another generator, or that names none