zrodlo tokenizer inspect
Load a tokenizer and report its vocabulary size, special tokens and digest.
Usage#
zrodlo tokenizer inspect --tokenizer tokenizer.jsonFlags#
| Flag | Required | Description |
|---|---|---|
--tokenizer | yes | The tokenizer.json file to inspect |
What it does#
Loads the file as a Hugging Face tokenizer and reports its vocabulary size, its special tokens with their ids and the file's SHA-256. It writes nothing.
Refusals#
tokenizer <path> cannot be loaded: <error>