Usage#

zrodlo model generate --model run/step-<N>/ --prompt <TEXT> --tokens <N> [--temperature <T> --seed <SEED>] --precision f32|bf16 --device cpu|metal|cuda [--ordinal <N>]

Flags#

Flag	Required	Description
`--model`	yes	The model directory or checkpoint that continues the prompt
`--prompt`	yes	The text the model continues, encoded with the model's tokenizer.json
`--tokens`	yes	How many tokens to generate
`--temperature`	no	Draw each token with the logits divided by this; without it the most likely token is taken
`--seed`	with `--temperature`	Keys the draw --temperature makes; required with it
`--precision`	yes	The number type the pass computes in: f32, or bf16 for mixed precision
`--device`	yes	The device to compute on
`--ordinal`	with `metal` or `cuda`	The GPU to use; required for metal and cuda

--precision takes f32 (Every operation in 32-bit floats) or bf16 (Weights cast to bfloat16 for the pass; norms and the loss in f32). --device takes cpu (The CPU; takes no --ordinal), metal (An Apple GPU through Metal; needs a binary built with the metal feature and --ordinal) or cuda (An NVIDIA GPU through CUDA; needs a binary built with the cuda feature and --ordinal).

What it does#

Continues a prompt: greedily, or with --temperature and --seed by a seeded draw that repeats exactly. Each token after the first pass reads only itself and attends to the keys and values kept from earlier passes. No weight changes and nothing is written.

Refusals#

It refuses the device and precision of model train as that command does. Nothing is written:

--temperature needs --seed, which keys the draw so it can be repeated
--seed keys a draw only --temperature makes; without it the most likely token is taken
--temperature is <value>; it must be a positive finite number
the prompt encodes to no token; give text the model's tokenizer reads
the prompt encodes to token <n>, outside the model's vocab_size <n>
<path> has no tokenizer.json; a model directory carries the tokenizer it was trained with