Agent Red-Team: Untrusted Instructions Meet Real Tools

A system prompt is an instruction to a model, not an operating-system protection ring. Direct injection arrives from the user; indirect injection arrives inside content the user asked the agent to process, such as a page, ticket, document, image, retrieved chunk, tool result, or peer message. Because instructions and data ultimately share model context, the threat model should ask what a compromised planner could reach rather than assuming a phrase like ignore untrusted instructions creates isolation.

1. 01Classify originPreserve whether text came from an authenticated user channel, system policy, untrusted content, a tool, or a peer; never upgrade content provenance because a model repeats it.
2. 02Parse a closed proposalReject unknown tools, fields, types, destinations, and open-ended commands before any authorization decision.
3. 03Mediate current authorityResolve authenticated principal, tenant, resource, capability, data class, and policy at the tool boundary for every call.
4. 04Control impactApply least privilege, destination allowlists, payload-bound human approval, rate and spend limits, idempotency, and sandboxing.
5. 05Validate the resultTreat tool output as typed untrusted data; prevent it from becoming a command, script, URL fetch, or trusted memory without another check.

Vary placement and encoding: visible text, metadata, quoted replies, OCR text, multilingual content, split payloads, tool descriptions, delayed memory, and peer summaries. Vary the requested effect and data class independently. A defense that blocks a familiar phrase may be a useful filter, but it is not the authority boundary and can over-block benign content while missing adaptive attacks.

1def audit_threat_model(
2    contract: ThreatContract, suite: ThreatSuite
3) -> ThreatReport:
4    """Exercise declared trust boundaries without executing tools or model prompts."""
5    contract = validate_record(contract, ThreatContract)
6    suite = validate_record(suite, ThreatSuite)
7    if suite.scope != contract.scope or suite.contract_content_id != contract.content_id:
8        raise ValueError("threat suite belongs to another contract")

example=illustrative_only
status=THREAT_MODEL_COMPLETE_FOR_FIXTURE
cases=7;blocked=6;allowed=1
covered=untrusted-instruction,tenant-authority,capability,secret-egress,human-approval,output-schema
coverage=1.000
claim=LOCAL_POLICY_AUDIT_NOT_PENETRATION_TEST_OR_SECURITY_CERTIFICATION

The invented suite contains one allowed read and six blocked proposals covering indirect instruction, cross-tenant deputy behavior, an undeclared capability, secret egress, destructive action without approval, and malformed tool output. The policy reports the first failing boundary in a fixed order. It binds exact grants, principal, tenant, tool, capability, destination, security flags, arguments digest, expected result, and declared attack-class coverage.

The implementation copies collections, freezes and hashes records, reconstructs nested evidence at the public boundary, and rejects duplicate cases, stale contracts, boolean impersonation, non-finite coverage targets, string subclasses, and constructor bypass. It invokes no model or tool, generates no adversarial payload, authenticates no grant, discovers no unknown threat, and measures no model robustness. Complete fixture coverage is set membership over the local contract, not penetration-test completion or certification.

Begin with a safe simulator or shadow tool whose state is inspectable and resettable. For each benign task, create adversarial variants that preserve the user's legitimate objective while changing injection location, attacker knowledge, target capability, tenant, data class, and destination. Record immutable versions of model, system/developer prompts, tool schemas, policy, retrieval corpus, memory seed, sampling configuration, judge, and environment.

1. 01Define the oracleState exact allowed and forbidden actions, resources, destinations, data flows, and terminal outcomes before running the case.
2. 02Capture side effectsUse tool and policy traces plus final resource state; redact secrets while preserving evidence identities and denial reasons.
3. 03Repeat stochastic casesUse enough independent runs to expose variance and report counts or uncertainty, without treating one sample as a stable rate.
4. 04Adapt attacksInclude held-out templates and human or automated adaptation within authorized safe bounds; static public strings invite overfitting.
5. 05Regression-test incidentsConvert every confirmed failure into a minimized case plus a system-level invariant, then test neighboring channels and effects.

• Expose task-specific tools instead of shells, generic HTTP fetchers, or broad database interfaces; issue short-lived, audience-bound, tenant-scoped credentials.
• Keep secret-bearing data out of model context unless the task requires it, and deny destinations that are not explicitly authorized for that data class.
• Run untrusted parsing and generated code in isolated environments with network, filesystem, process, time, and resource constraints.
• Require independent approval at the last responsible moment for destructive, financial, external-communication, or privilege-changing actions.
• Log proposals, policy decisions, tool receipts, and lineage without copying raw secrets or attacker-controlled content into unsafe observability sinks.

Prepare a response path before granting authority: disable a tool or route, revoke and rotate credentials, quarantine affected memory or retrieved content, stop new runs, identify completed effects, reconcile ambiguous work, notify owners, and preserve a bounded forensic record. Changing the prompt alone is not containment when tokens, queued actions, or poisoned durable state remain active.