Configuration#

A Transformers config.json whose model_type is gpt2 or llama chooses the decoder family. Zrodlo reads a published configuration with every field it carries and writes all of them back.

A GPT-2 configuration states vocab_size, n_positions, n_embd, n_layer, n_head, n_inner, layer_norm_epsilon, initializer_range, embd_pdrop, attn_pdrop and resid_pdrop. Many published files leave n_inner null; Zrodlo refuses that and asks for the width. It refuses a configuration it does not compute: an activation_function other than gelu_new, tie_word_embeddings: false, scale_attn_weights: false, scale_attn_by_inverse_layer_idx: true or add_cross_attention: true.

A Llama configuration states vocab_size, hidden_size, intermediate_size, num_hidden_layers, num_attention_heads, num_key_value_heads, max_position_embeddings, rms_norm_eps, rope_theta, initializer_range, hidden_act and tie_word_embeddings; head_dim is hidden_size / num_attention_heads when absent. It is refused when hidden_act is not silu, when the attention heads are not a multiple of the key and value heads or the head width is odd, and when attention_bias or mlp_bias is true, attention_dropout is above zero or rope_scaling is set.

The Llama model computes RMS norms, rotary position embeddings, grouped-query attention and a SiLU-gated feed-forward layer, as Transformers' LlamaForCausalLM does, and trains without dropout.

Corpus#

A directory with two files:

tokens.bin: little-endian u32 tokens, with one end-of-sequence token after every document.
corpus.json: the sources, each document named by its file or, for JSON Lines, its file and line; the tokenizer digest, the token count and the digest of tokens.bin.

Training, evaluation and corpus inspect map tokens.bin from disk and read it in place, so a corpus larger than memory is read page by page.

Model directory#

A Transformers checkpoint of its family: config.json, tokenizer.json and model.safetensors. A GPT-2 directory stores a fused c_attn and a tied output head under transformer., the GPT2LMHeadModel layout; Zrodlo also reads the bare GPT2Model layout that openai-community/gpt2 publishes. A Llama directory stores every tensor under its LlamaForCausalLM name, and lm_head.weight only when the embeddings are not tied. Ster posttrains this directory as it is.

Checkpoint#

A model directory plus:

optimizer.safetensors: AdamW moments and bias-correction state for every weight.
trainer.json: window size, batch and micro-batch size, seed, the generator the run's draws came from, precision, optimizer hyperparameters, corpus and tokenizer digests, held-out scores, and the loss of every step in order.

The number of steps done is the number of recorded losses.