Tokenization as a Compression Contract

A model never receives the user's string directly. The tokenizer first commits to a character-to-byte path. UTF-8 gives a deterministic encoding, but Unicode permits visually similar strings with different code-point and byte sequences. Normalization can intentionally collapse some distinctions; omitting normalization preserves them. Either is a protocol choice that must be named, versioned, and reproduced by every writer and reader.

1. 01Capture the original stringRetain a bounded, access-controlled test fixture and its exact UTF-8 bytes; display equivalence is not byte equivalence.
2. 02Apply the declared normalizationUse one named algorithm and version, or explicitly preserve code points without normalization.
3. 03Require a reversible terminal alphabetA byte fallback can cover every valid UTF-8 byte without silently mapping unseen content to one lossy unknown token.

A token piece is not necessarily a word, morpheme, or stable semantic unit. It is a byte sequence selected by the training algorithm and corpus. Frequent patterns can receive short representations, while rare scripts, names, source code, or changed traffic may remain close to byte-level. The learned inventory is an allocation of finite vocabulary capacity.

Token count flows into context occupancy, truncation, batch shapes, KV-cache growth, and usage-based pricing. Report medians and tails by language, product surface, code versus prose, and input age. A global average can hide a cohort whose prompts consume twice as many positions and are truncated much earlier.

The laboratory holds inputs and policy identities fixed while changing illustrative merge coverage or byte fallback. Predict both the token-count direction and whether the original UTF-8 bytes survive. The grade is about this fixture only; it does not rank production tokenizers.

The reference artifact freezes concrete token and merge records, validates unique IDs and contiguous ranks, binds scope and release owners, and captures source version, observation time, text, and exact input bytes. Its public boundary reconstructs every record and digest, so constructor bypass, mutable collections, stale contracts, malformed timestamps, and oversized evidence fail before encoding.

1def format_example() -> str:
2    report = audit_tokenization(ILLUSTRATIVE_CONTRACT, ILLUSTRATIVE_EVIDENCE)
3    return "\n".join(
4        (
5            "example=illustrative_only",
6            f"contract_id={report.contract_content_id}",
7            f"evidence_id={report.evidence_content_id}",
8            f"tokens={','.join(report.token_ids)}",
9            f"compression={report.input_bytes}_bytes/{len(report.token_ids)}_tokens",
10            f"fallback_tokens={report.fallback_tokens}",
11            f"round_trip={report.round_trip}",
12            f"claim={report.compression_claim}",
13            f"decision={report.decision}",
14        )
15    )

example=illustrative_only
contract_id=tokenizer-contract@sha256:bef3dd116eb77f4817d0154efeded07f31a55bd6406cf12bc8c5183a02f10542
evidence_id=tokenization-evidence@sha256:a3115aa0672c1e2262e320b68deb6a6614ac559007dd934ebf116cf45b6423f1
tokens=piece-ba,piece-nana,byte-21
compression=7_bytes/3_tokens
fallback_tokens=1
round_trip=EXACT_UTF8_BYTES
claim=OBSERVED_FOR_THIS_EVIDENCE_ONLY
decision=PASS

A safe rollout expands readers before writers: deploy code able to identify and decode both tokenizer versions, shadow-tokenize sampled traffic, compare exact bytes and length distributions, then move writers by cohort. Retain version tags and rollback until old stored IDs, caches, and queued jobs have expired or migrated.