Preference Optimization and Reward Failure

A comparison has a prompt, two exact candidate responses, a presentation order, an annotation instruction, and an observed label. Calling one response chosen records a judgment under that process; it does not make the response objectively correct. A rater may prefer a fluent hallucination, reject an appropriately cautious answer, or abstain because both candidates are defective. Preserve that ambiguity instead of forcing every record into a binary win.

The InstructGPT paper gives a concrete demonstration/comparison/reward/policy pipeline, while DPO supplies a different route from comparisons to a policy objective. Treat these as mechanisms to reason about, not interchangeable product badges. Both depend on whose preferences were collected and where the resulting model will be used.

For illustrative values policy=(-2,-4), reference=(-3,-3), and β=0.1, the margin is 0.2 and the loss is approximately 0.598139. Compute softplus as max(0,-m) + log1p(exp(-abs(m))) so extreme valid margins do not overflow. This arithmetic can be perfectly correct while the pair label is wrong. Numerics and supervision validity are separate obligations.

The artifact binds contract, model, reference, policy, data, annotator-policy, evaluation, held-out split, and source revisions in a fixed scope. Each pair adds pair and prompt IDs, exact prompt text, distinct chosen/rejected text, presentation position, rater labels, and four supplied log probabilities. The pair digest covers every field. Changing an answer while retaining its old digest is rejected; changing the digest creates different evidence that must be evaluated again.

• Chosen/rejected labels describe content identity, while left/right records presentation position; never conflate the two.
• A tie means neither candidate was preferred under the rubric. Abstention means the rater withheld a judgment. Neither is a decisive win.
• Duplicate annotator IDs within a pair inflate support; duplicate prompts across pairs can overweight a narrow task even with different pair IDs.
• A held-out split label is a declared contract, not evidence that no training contamination exists. Join source lineage and perform contamination checks separately.

The frozen records copy nested sequences at construction. The public audit reconstructs and rechecks concrete records so constructor bypass, stale digests, malformed numbers, and noncanonical values do not receive a report. Hashes make content changes visible; they do not authenticate raters or establish that upstream log probabilities were measured correctly.

1def audit_preferences(contract: PreferenceContract, pairs: tuple) -> PreferenceReport:
2    """Revalidate identities, support, bias indicators, and stable pairwise loss."""
3    validate_record(contract, PreferenceContract)
4    if type(pairs) not in (tuple, list):
5        raise ValueError("pair sequence required")
6    count(len(pairs), 1, 1000)
7    pairs = tuple(pairs)
8    for pair in pairs:
9        validate_record(pair, PreferencePair)
10        if pair.scope != contract.scope:
11            raise ValueError("model/data/policy/evaluation scope mismatch")

example=illustrative_only
status=SUPPORTED
decisive=4;ties=2;abstentions=2
mean_dpo_loss=0.598139
claim=PREFERENCE_EVIDENCE_NOT_REWARD_OR_CAPABILITY

The fixture contains two prompts, opposite chosen positions, four decisive votes, two ties, and two abstentions. The audit first checks shape, scope, identity, and content integrity. It then blocks insufficient pair or decisive support, requires every pair to have a decisive judgment, rejects a chosen label without a chosen-vote majority, and flags excessive rejection disagreement or a one-sided presentation layout. The thresholds are deliberately small teaching values, not a sampling recommendation.

Suppose a policy's proxy reward rises while users report more persuasive incorrect answers. Start by preserving the old checkpoint, reward version, prompt cohort, and annotation rubric. Replay both policies on the same held-out prompts, blind the presentation, and collect factuality or task-outcome judgments independently of the reward score. Split by answer length, refusal behavior, difficult prompts, and rater cohort. A shift in the mix can make a global preference rate look better without improving any important slice.

1. 01Check the comparison processConfirm exact candidate content, randomized presentation, tie handling, and annotation instructions before blaming an optimizer.
2. 02Separate evaluator from targetDo not use the reward model being optimized as the only release judge. Preserve independent task, safety, and user-outcome evidence.
3. 03Bound the conclusionDescribe which paired cohort and rubric improved, which regressed, and how much support each conclusion has. Do not extrapolate a local result to all users.

The reward-overoptimization paper studies how optimizing an imperfect proxy can hurt a separate target in a controlled synthetic setting. The transferable engineering lesson is to preserve independent outcomes and watch optimization pressure; its fitted relations should not be transplanted as universal production thresholds.

Attack the audit with reversed content, duplicate prompt IDs, repeated prompt text under new IDs, duplicated raters, stale annotation policies, zero decisive support, and constructor-bypassed records. Numeric tests include booleans, NaN, infinities, subnormal values, positive sequence log probabilities, oversized magnitudes, and extreme valid margins. A non-finite loss should not be the first indication that evidence was malformed.

At operation time, associate the comparison dataset and its policy with the adapter or full checkpoint it produced. Keep the reference policy revision, objective settings, evaluation cohort, and release decision together. When a rubric changes, version it and decide whether old labels remain usable; silently mixing incompatible rubrics destroys the meaning of a preference rate. Rollback must recover both the prior serving policy and the evaluation contract used to approve it.