Build a Golden Dataset That Can Disagree with You

A golden dataset is a versioned set of cases and reference judgments used for a named evaluation decision. Its value comes from provenance, coverage, labeling discipline, and separation from optimization—not from the adjective. Some tasks have exact answers; others have multiple acceptable responses, conditional policy outcomes, or genuine reviewer disagreement. Preserve that structure instead of forcing every case into an artificial universal truth.

Begin with the units the product will actually face: conversations, requests, documents, tool trajectories, sessions, or decisions. Define inclusion, exclusion, time window, channel, language, tenant, privacy handling, and deduplication before drawing examples. Frequency-weighted sampling estimates common performance; it will often under-sample rare high-cost events, so add a clearly labeled targeted stratum rather than pretending the combined set is one random sample.

1. 01Describe the frameState what records could have been selected and which deployment population they approximate.
2. 02Sample ordinary workDraw real tasks using a reproducible process and preserve bounded provenance.
3. 03Target material risksAdd rare but costly policy, safety, security, and tool-use cases with explicit selection reason.
4. 04Deduplicate by meaningDetect exact and near duplicates, template families, and repeated user sessions so support is not inflated.
5. 05Document exclusionsRecord privacy, legal, quality, or operational exclusions and the blind spots they create.

Define coverage over product behaviors and failure mechanisms, not only demographics or topic labels. Cross key axes when interactions matter, but avoid a combinatorial checklist with one token case per cell. Minimum support should reflect the decision and expected variance. Report empty or weak cells as missing evidence rather than filling them with low-quality synthetic examples merely to make a dashboard green.

Write the rubric before exposing reviewers to the favored candidate. Define required context, acceptable alternatives, severity, abstention, and escalation. Recruit reviewers with the domain and policy competence the judgment requires. Use blinded presentation where identity or order could bias the label, and give reviewers a way to flag an invalid case instead of manufacturing certainty.

Every time developers inspect a release failure and tune the prompt, retrieval, policy, or model against it, that case becomes development evidence. It may remain useful as a regression test, but repeated adaptation reduces its value as an unbiased estimate of future performance. Maintain separate development, grader-calibration, and protected release partitions with access controls and audit logs. Rotate protected samples and keep a truly sequestered mechanism for high-stakes decisions when feasible.

• Use content and near-duplicate checks across training, fine-tuning, prompt examples, retrieval corpora, development evals, and protected sets where access permits.
• Bind exclusion attestations to dataset and candidate versions; an attestation is reviewable evidence, not proof of absence.
• Do not publish protected labels, expected outputs, or evaluator rationales into systems that generate future candidates.
• Track how often a case has been inspected and which changes were made because of it.
• Preserve stable anchor cases for longitudinal comparison while refreshing a portion for drift and new failure modes.

1def audit_coverage(contract: object, snapshot: object) -> CoverageReport:
2    """Audit declared coverage without running a model or changing the dataset."""
3    checked_contract = _rebuild_contract(contract)
4    checked_snapshot = _rebuild_snapshot(snapshot)
5    if checked_snapshot.scope != checked_contract.scope:
6        raise ValueError("snapshot belongs to another scope")
7    if checked_snapshot.contract_content_id != checked_contract.content_id:
8        raise ValueError("snapshot is not bound to this contract content")

example=illustrative_only
dataset_id=golden-dataset@sha256:e77405fc5a7aed80ec68877e7342a83b10091610597dfd5dfc0686d496a041ea
cases=8;slices=4
coverage=hard_negative,long_tail,policy_boundary,real_task
decision=PASS_COVERAGE
claim=LOCAL_COVERAGE_AUDIT_NOT_DATASET_QUALITY_CERTIFICATION

The invented support-assistant snapshot contains eight unique cases across billing, policy, returns, and technical slices. The contract requires per-slice case classes, all four global case classes, two reviewers, adjudication, a training-exclusion attestation, a minimum total, and an exact integer concentration limit. The audit reports only whether those declarations satisfy the local contract.

The implementation reconstructs frozen nested records, copies caller-owned collections, binds the snapshot to contract content and scope, rejects duplicate IDs and input digests, undeclared slices, booleans in integer fields, string subclasses, malformed hashes, and constructor bypass. It does not inspect source records, reviewer competence, consent, training corpora, semantic duplication, label correctness, or deployment representativeness. `PASS_COVERAGE` is not a dataset-quality certification.

Publish a dataset change log: additions, removals, label changes, provenance changes, known leakage, slice support, and expected score discontinuities. Run old and new versions during transition to distinguish candidate movement from measurement movement. Retire cases whose consent, policy, source, or task assumptions no longer hold rather than silently preserving a clean trend line.