Chunking Is a Recall Policy

A retriever cannot return an evidence unit that was never represented. Fixed token windows make cost predictable but can cut headings from their bodies, tables from labels, or an exception from its governing rule. Sentence and structure-aware segmentation preserve different boundaries but depend on parser quality. Overlap gives spans near one boundary a second chance while duplicating storage and candidate content.

Build a query-to-evidence set with exact source offsets and inspect the distribution of evidence lengths, boundary types, languages, tables, lists, code, and cross-section dependencies. Evaluate policy candidates with the same embedding and retrieval settings first, then study interactions deliberately. A chunk size that works for short factoid passages may fail contracts, narratives, or multi-hop questions.

1. 01Preserve source identityCarry document ID, revision, content digest, ordered token offsets, headings, and access metadata into every chunk.
2. 02Generate candidate policiesMake size, overlap, boundary parser, parent-context treatment, and tokenizer revision explicit.
3. 03Evaluate stagesMeasure representability, candidate retrieval, reranking, context packing, grounding, latency, storage, and cost separately.
4. 04Inspect failuresRead boundary misses and wasted contexts by document family instead of selecting from one average score.

Inspect the frozen evidence span before changing the retriever. The lab asks only whether each declared interval policy emits at least one chunk that contains the complete span. If none does, segmentation is already blocking that case. If one does, continue with separate retrieval, reranking, context, and generation evidence rather than treating geometric containment as end-to-end success.

The fixture uses half-open token spans, enforces canonical source order and document-edge coverage, bounds overlap and chunk size, binds document/chunk/query content identities, and separates representability from retrieval coverage. Unrepresentable cases receive a full-document waste penalty so they cannot improve the average by disappearing. The offsets and digests remain caller-attested; the audit does not run a tokenizer or judge semantic sufficiency.

1def main():
2    result = audit(example_contract())
3    print("example=illustrative_only")
4    print(f"span_coverage={result.span_coverage:.3f}")
5    print(f"retrieval_coverage={result.retrieval_coverage:.3f}")
6    print(f"mean_minimal_waste_tokens={result.mean_minimal_waste_tokens:.3f}")
7    print("decision=" + result.decision)
8    print("claim=" + result.claim)

example=illustrative_only
span_coverage=1.000
retrieval_coverage=1.000
mean_minimal_waste_tokens=3.000
decision=PASS_DECLARED_COVERAGE_GATES
claim=TOKEN_SPAN_FIXTURE_ONLY

A new tokenizer, parser, maximum size, overlap, or heading policy creates new chunk identities. Build them beside the old generation, embed them under an explicit contract, populate a separate index, and compare the same query cases. Dual-write source changes and deletions during backfill. At cutover, switch chunk, vector, index, reranking, and context-assembly versions coherently; do not let new queries search only half of a migrated corpus.

Monitor chunk count per source, token-length and overlap distributions, parse failures, uncovered source ranges, missing embeddings, near-duplicate top-k results, representability samples, retrieval coverage, final context utilization, and supported-answer outcomes. Expire and delete old chunks only after a separate retention-aware contraction decision.