Usage#

zrodlo model init --config config.json --tokenizer tokenizer.json --seed <SEED> --output base/

Flags#

Flag	Required	Description
`--config`	yes	A Transformers config.json whose model_type is gpt2 or llama
`--tokenizer`	yes	The tokenizer.json the model reads; its vocabulary must fit vocab_size
`--seed`	yes	Keys the draw of every initial weight
`--output`	yes	The new model directory to write; it must not exist, its parent must

What it does#

Draws every weight from the seed with GPT-2's scheme: embeddings and projections from a normal of deviation initializer_range, the two residual output projections of each block with that deviation divided by the square root of twice the layer count, norm scales at one and biases at zero. It writes config.json, tokenizer.json and model.safetensors, a Transformers model directory of the configuration's family. The fields each family's configuration must state are in What Zrodlo reads and writes.

Refusals#

A configuration missing a field its family needs is refused by the parser, naming the field. A tokenizer too large for the configuration is found after the output directory is created, which is then removed again:

<path>: model_type is <type>; Zrodlo trains gpt2 and llama decoders, and <path>: model_type is missing; Zrodlo reads it to choose between gpt2 and llama
n_inner is null or absent: Zrodlo builds the feed-forward layer only at a width config.json states
<path>: n_embd <n> is not divisible by n_head <n>, and for Llama <path>: hidden_size <n> is not divisible by num_attention_heads <n>, and no head_dim is stated and <path>: num_attention_heads <n> is not a multiple of num_key_value_heads <n>
<path>: <name> is <p>; a dropout probability is at least zero and below 1
<path>: activation_function is <name>; Zrodlo implements gelu_new only and <path>: hidden_act is <name>; Zrodlo implements silu only
<path>: rope_scaling is <value>; Zrodlo computes unscaled rotary embeddings, <path>: attention_bias is true; Zrodlo trains Llama projections without bias and <path>: attention_dropout is <p>; Zrodlo trains Llama without dropout
tokenizer <path> has <n> tokens, more than the configuration's vocab_size <n>
<path> already exists; Zrodlo writes a new directory and overwrites nothing
<path> cannot be created: its parent <path> is not an existing directory, and Zrodlo creates only the directory it writes, never a missing parent; create <path> first