The Vision-Language Bridge

Start with a media object and its revision, not a tensor of anonymous numbers. Decoding determines orientation, channels, dimensions, and color interpretation. Resizing, cropping, and normalization define which information reaches the encoder. Patching assigns an ordered region of that processed image to each visual position. The encoder then transforms these positions into features; its outputs are neither raw pixels nor language token IDs.

The vision-transformer paper establishes a patch-sequence treatment of images. A language interface needs additional choices beyond that construction. Visual Instruction Tuning and BLIP-2 illustrate different ways to connect visual and language components. Do not assume every vision-language system exposes one visual token per original patch at the decoder boundary.

In the illustrative fixture, a 4×4 RGB image with 2×2 patches yields four raster-ordered visual positions. Supplied projector rows each have hidden width two. Two text tokens produce six fused positions, so the unpadded visibility mask contains six integer ones. That mask is a padding/visibility declaration, not a complete causal attention matrix; autoregressive attention semantics still belong to the chosen model implementation.

A bridge has two sides: the visual encoder's feature semantics and the language model's expected hidden-state interface. Equal dimensions do not imply compatible coordinates. Replacing an encoder while retaining an old projector can invalidate the interface even if every matrix multiplication succeeds. The contract binds encoder, projector, tokenizer, model, patching, normalization, data, policy, source, and evaluation revisions. Its teaching normalization accepts RGB values in [-1,1]; no other normalization name is silently mapped to that convention.

Raster patch order must equal the exact range from zero to the final patch index. Duplicate or permuted indices are rejected, projected row count must equal the patch count, and every projected row must match the model-side width. The content digest covers exact normalized pixels, supplied projected rows, text, tokens, media revision, and mask. This identifies the evidence but does not independently prove that the supplied rows were computed from those pixels or that token IDs encode the supplied text.

A language model may answer a familiar question from textual priors while ignoring the visual path. Preserve matched image/text evaluation and compare it with a deliberately shuffled-image control under the same model, prompts, and scoring rubric. Bind that comparison to the exact input digest and evaluation cohort. A stale evaluation from an earlier projector cannot authorize a newly substituted projector, even if the reported scores are identical.

The artifact accepts supplied matched and shuffled scores plus support, then requires a declared minimum gap and support floor. This is a small executable alignment-evidence gate, not a benchmark runner or statistical significance test. Real evaluation must span independent examples, control for label and prompt leakage, inspect uncertainty, and separate OCR, counting, spatial relations, charts, and other intended tasks. A positive gap does not show that every answer is grounded or correct.

1def audit_bridge(contract: BridgeContract, media: BridgeInput, alignment: AlignmentEvidence) -> BridgeReport:
2    """Bind structural validity to exact cross-modal evaluation evidence."""
3    validate_record(contract, BridgeContract)
4    validate_record(media, BridgeInput)
5    validate_record(alignment, AlignmentEvidence)
6    if media.scope != contract.scope or alignment.scope != contract.scope:
7        raise ValueError("encoder/projector/tokenizer/model/data scope mismatch")
8    if alignment.input_content_id != media.content_id:
9        raise ValueError("alignment is not bound to exact image/text content")
10    if media.width * media.height > contract.max_pixels:
11        raise ValueError("resolution budget")

example=illustrative_only
status=BOUND
patches=4;fused_tokens=6
claim=DECLARED_ALIGNMENT_NOT_SEMANTIC_UNDERSTANDING

The code requires concrete frozen contract, input, and alignment records. Constructors normalize nested lists to immutable tuples; public entry reconstructs every record and verifies the content identity again. It bounds pixel and token counts, prohibits boolean counts, rejects non-finite and subnormal numbers, checks normalized RGB values, and enforces shape and scope. Structural violations raise an error, while insufficient alignment evidence returns BLOCK_ALIGNMENT.

The executable surface intentionally accepts already decoded pixels and supplied projected rows. It does not include a file decoder, pretrained encoder, tokenizer, learned bridge, or language generation loop. That keeps the representation contract inspectable without implying that the illustrative vectors are meaningful visual embeddings. BOUND means the declared structural and alignment conditions passed for this exact record, not that a real image was understood.

1. 01Break media integrityDrop one RGB value, change a media revision, duplicate a modality, or provide a non-divisible grid. Require rejection before interpreting alignment scores.
2. 02Break representation routingSwap two patch indices, provide an incorrect projector width, or shorten the fused mask. Show that dimensions, order, and visibility are distinct invariants.
3. 03Break evidence lineageReuse alignment evidence after changing text, pixels, or a component version. The exact input binding must fail.
4. 04Break confidenceKeep every shape valid but remove alignment support or increase the shuffled score. Structural success must not bypass the alignment gate.

In production, partition failures by preprocessing revision, media type, resolution bucket, language, task slice, and model cohort. Keep enough non-sensitive lineage to replay a failed route without retaining raw media indefinitely. If an encoder or projector changes, rerun structural and behavioral evaluations together. When the image is unavailable, explicitly abstain or invoke a separately evaluated text-only mode; silently continuing with a missing image changes the task.