zrodlo model evaluate
Score a model directory's saved weights on a corpus; no weight changes.
Usage#
zrodlo model evaluate --model run/step-<N>/ --corpus held-out/ --context <TOKENS> --batch-size <WINDOWS> --precision f32|bf16 --device cpu|metal|cuda [--ordinal <N>]Flags#
| Flag | Required | Description |
|---|---|---|
--model | yes | The model directory or checkpoint whose weights are scored |
--corpus | yes | The prepared corpus scored, encoded with the model's tokenizer |
--context | yes | Tokens per scored window; the model reads all but the last |
--batch-size | yes | Windows read per forward pass |
--precision | yes | The number type the pass computes in: f32, or bf16 for mixed precision |
--device | yes | The device to compute on |
--ordinal | with metal or cuda | The GPU to use; required for metal and cuda |
--precision takes f32 (Every operation in 32-bit floats) or bf16 (Weights cast to bfloat16 for the pass; norms and the loss in f32). --device takes cpu (The CPU; takes no --ordinal), metal (An Apple GPU through Metal; needs a binary built with the metal feature and --ordinal) or cuda (An NVIDIA GPU through CUDA; needs a binary built with the cuda feature and --ordinal).
What it does#
Reports the token-weighted mean loss and perplexity of the saved weights over every window of a corpus, without dropout. No weight changes and nothing is written.
Refusals#
It refuses a corpus corpus inspect refuses, with that command's sentence, and the device and precision refusals of model train. Nothing is written:
<path> has no tokenizer.json; a model directory carries the tokenizer it was trained with<path> was encoded with another tokenizer than <path>: its corpus.json records tokenizer <digest>, and <path> has digest <digest>; prepare the corpus again with <path>--context <n> leaves no token to read before the one predicted; it needs two or more