How to Design a Multi-Agent System: Orchestration, Memory, Tool Calling, and Failure Recovery

Design a production multi-agent system as a durable task graph, not a group chat. Put one orchestrator in charge of decomposition, dependencies, budgets, deadlines, and recovery. Give each worker a bounded objective, a minimal context view, and an allowlist of typed tools. Persist task transitions and tool-call receipts outside the model context, then verify artifacts before any consequential action. If a process dies, resume unfinished work from the last checkpoint instead of replaying the conversation.

Start with one agent. Add another only when independent work can run in parallel, a specialist needs different context or permissions, or ownership crosses a real service boundary. Otherwise, extra agents mostly add coordination latency, token cost, and new ways to fail.

The production architecture in one table

The important separation is control plane versus work plane. The orchestrator owns truth about the run. Agents produce proposed plans, decisions, or artifacts; they do not get to redefine authorization, declare their own side effects successful, or hide retry state in a prompt.

Do you need multiple agents at all?

A multi-agent design is justified by the shape of the problem, not by the number of personas you can name. Use this decision sequence:

1. Can deterministic code perform the step? Use code, a rule engine, or a workflow node. Do not spend a model call on known control logic.
2. Does one agent have enough context and tools? Keep one agent if the task is sequential and its information fits a coherent context.
3. Are two or more subtasks truly independent? Parallel workers can reduce wall-clock latency and explore different evidence when their work does not depend on constant cross-talk.
4. Does a boundary improve safety or ownership? Separate agents when they need different credentials, data access, models, scaling, or deployment ownership.
5. Can you afford coordination and verification? If the task value does not pay for extra calls, longer tails, and broader evaluation, remain single-agent.

Production reports support this restraint. Anthropic's multi-agent research system uses an orchestrator-worker pattern because research branches can run in parallel, but it also reports materially higher token use and poor fit for tasks whose agents must continuously share the same context. Google ADK similarly distinguishes graph, dynamic, collaborative, sequential, loop, and parallel workflow structures instead of treating every multi-step application as free-form agent conversation.

Strong reasons to split an agent

• A search worker and a policy analyst can investigate independent evidence in parallel, then return structured artifacts to a verifier.
• A finance agent may read billing data while an operations agent can restart a service; separating their capability sets reduces blast radius.
• A remote specialist is owned and deployed by another team, so its lifecycle needs an explicit task contract.
• A worker needs a large domain-specific context that would distract the primary agent or overflow its context window.

Weak reasons to split an agent

• “Researcher,” “writer,” and “reviewer” all receive the same transcript and can call the same tools.
• A fixed three-step pipeline is dressed up as three autonomous agents even though ordinary functions would be easier to test.
• The system hopes that majority voting will make correlated model errors independent.
• More agents are added to compensate for vague objectives, weak tool schemas, or missing evaluation.

If you are still deciding where MCP or A2A belongs, keep protocol selection separate from runtime design. The MCP vs A2A vs A2UI system design guide maps those protocols to capability, delegation, and interface boundaries.

Choose an orchestration model

There is no universally best topology. There is, however, a strong default for a first production system: centralized durable control with decentralized bounded execution.

Avoid an unbounded “everyone talks until consensus” loop. It has no stable owner for termination, cost, or truth. If peers collaborate, define the message types, who may publish each type, the convergence condition, and the maximum rounds in code.

Keep deterministic control outside the model

Models can propose a plan or classify the next action. Deterministic code should still enforce:

• valid task state transitions;
• dependency readiness and maximum fan-out;
• token, step, tool-call, cost, and wall-clock budgets;
• per-agent tool permissions and data scopes;
• retry eligibility and backoff limits;
• approval gates, cancellation, and terminal states; and
• schema validation before an artifact enters shared state.

This does not eliminate model autonomy. It gives autonomy a boundary that can be observed, tested, and stopped.

Define the task contract before writing prompts

An agent role such as “be the database expert” is not an execution contract. A scheduled task needs at least:

type TaskStatus =
  | "pending"
  | "leased"
  | "completed"
  | "failed"
  | "waiting_for_approval"
 
type AgentTask = {
  runId: string
  taskId: string
  objective: string
  agentId: string
  dependsOn: string[]
  inputArtifactIds: string[]
  outputSchema: string
  allowedTools: string[]
  deadline: string
  maxAttempts: number
  budget: { tokens: number; toolCalls: number; costUsd: number }
  status: TaskStatus
  planVersion: number
}

Use stable runId and taskId values across retries. Put large outputs in an artifact store and pass references plus short summaries through the coordinator. That avoids repeatedly copying documents through several context windows—the agentic version of the “game of telephone.”

Every output contract should say what “done” means. For example, a risk-review worker might return { decision, risks[], evidence[], unknowns[] }, not an unstructured paragraph. A validator can reject a missing evidence reference before the planner treats the task as complete.

Design memory as four different stores

“Give the agents shared memory” is too vague for a production design. Separate memory by purpose, scope, and trust.

This split mirrors the distinction in LangGraph's current persistence model: checkpoints retain thread-scoped graph state, while stores retain application-defined data across threads. The principle is framework-independent.

Use a scoped context view, not a shared transcript

Build each prompt from:

• the task objective and output schema;
• only the dependency artifacts the task requires;
• a compact record of relevant attempts and tool outcomes;
• verified durable facts with source, timestamp, and confidence; and
• the worker's current tool and policy envelope.

A full shared transcript increases token cost and creates accidental coupling. One worker's unsupported claim can be repeated by three others until it looks like consensus. When agents share a blackboard, make entries typed and versioned:

type BlackboardEntry<T> = {
  key: string
  value: T
  authorTaskId: string
  sourceArtifactIds: string[]
  verification: "unverified" | "verified" | "rejected"
  version: number
  expiresAt?: string
}

Require compare-and-set updates when two workers can write the same key. If a fact matters to a side effect, re-read its authoritative source immediately before acting; memory freshness is not transaction isolation.

Put tool calling behind a capability broker

Do not give every agent an SDK client and a bag of credentials. A tool broker should be the only path from model-selected actions to external systems.

For every call, the broker should:

1. validate the tool name and arguments against a runtime schema;
2. derive user, tenant, and agent identity from verified execution context;
3. enforce a per-agent, per-task capability allowlist;
4. apply deadlines, rate limits, cost budgets, and circuit breakers;
5. classify the operation as read-only, idempotent write, or non-idempotent write;
6. require approval for policy-defined high-impact actions;
7. attach a stable idempotency key and trace context;
8. validate and sanitize the result before returning it to the model; and
9. write an audit receipt without logging secrets or sensitive payloads.

The current MCP tool contract also emphasizes input validation, access control, rate limits, result handling, timeouts, auditability, and confirmation for sensitive operations. MCP can standardize the capability boundary, but the orchestrator still owns workflow state and recovery. For tenant-aware identity and token exchange, use the secure multi-tenant MCP gateway design.

Tool descriptions are not permissions

Text such as “read-only” in a tool description helps a model choose, but it is not enforcement. The broker must decide from trusted configuration and verified identity. Never let a prompt add a scope, choose a tenant, skip approval, or lower the audit level.

Writes need special handling. If a chargeCard call times out, you do not know whether the charge failed or the response was lost. Repeating it blindly can double-charge the customer. Query by idempotency key or operation ID, reconcile the remote state, and only then decide whether another attempt is safe.

Build failure recovery as a state machine

“Retry three times” is not a recovery design. The right action depends on what failed and whether a side effect may have committed.

Checkpoint state transitions, not token streams

A useful checkpoint is a business-level transition: plan accepted, task leased, artifact validated, approval granted, tool operation reconciled, task completed. Persisting every generated token creates volume without a reliable recovery boundary.

On restart:

1. load the latest run and plan version;
2. expire leases whose heartbeat deadline passed;
3. inspect unfinished tool operations and reconcile ambiguous writes;
4. validate that approvals, credentials, and input versions are still current;
5. schedule only pending tasks whose dependencies are complete; and
6. emit a recovery event linked to the original trace.

Durable graph systems use the same principle: save state at meaningful steps so a failed execution can continue from the last successful boundary rather than run every prior node again.

A runnable TypeScript supervisor with crash recovery

The following dependency-free example demonstrates the control-plane mechanics. Two specialist agents run in parallel. One retries a simulated model timeout. The orchestrator checkpoints both outputs, the process crashes, and a new supervisor resumes only the unfinished verifier task.

type TaskStatus = "pending" | "completed" | "failed"
type RunStatus = "running" | "completed" | "failed"
 
type Task = {
  id: string
  agentId: string
  dependsOn: string[]
  status: TaskStatus
  attempts: number
  maxAttempts: number
  output?: string
  lastError?: string
}
 
type RunState = {
  runId: string
  goal: string
  version: number
  status: RunStatus
  tasks: Task[]
}
 
class TransientError extends Error {}
 
interface CheckpointStore {
  load(runId: string): Promise<RunState | undefined>
  save(state: RunState, expectedVersion: number): Promise<RunState>
}
 
class InMemoryCheckpointStore implements CheckpointStore {
  private readonly runs = new Map<string, RunState>()
 
  async load(runId: string): Promise<RunState | undefined> {
    const state = this.runs.get(runId)
    return state ? this.copy(state) : undefined
  }
 
  async save(state: RunState, expectedVersion: number): Promise<RunState> {
    const actualVersion = this.runs.get(state.runId)?.version ?? 0
    if (actualVersion !== expectedVersion) {
      throw new Error(
        `checkpoint conflict: expected v${expectedVersion}, found v${actualVersion}`
      )
    }
 
    const saved = this.copy({ ...state, version: expectedVersion + 1 })
    this.runs.set(state.runId, saved)
    return this.copy(saved)
  }
 
  private copy(state: RunState): RunState {
    return JSON.parse(JSON.stringify(state)) as RunState
  }
}
 
type ToolArgs = {
  "deployments.read": { service: string }
}
 
type ToolResults = {
  "deployments.read": { service: string; version: string; pods: number; region: string }
}
 
type ToolName = keyof ToolArgs
 
class ToolBroker {
  private readonly receipts = new Map<string, unknown>()
  private readonly allowed: Record<string, ToolName[]> = {
    "inventory-agent": ["deployments.read"],
    "risk-agent": [],
    "verifier-agent": [],
  }
 
  async call<K extends ToolName>(
    agentId: string,
    name: K,
    args: ToolArgs[K],
    idempotencyKey: string
  ): Promise<ToolResults[K]> {
    if (!this.allowed[agentId]?.includes(name)) {
      throw new Error(`${agentId} is not allowed to call ${name}`)
    }
 
    const prior = this.receipts.get(idempotencyKey)
    if (prior) {
      console.log(`[tool] replayed receipt ${idempotencyKey}`)
      return prior as ToolResults[K]
    }
 
    // A production broker validates args at runtime and calls the real service.
    const result = {
      service: args.service,
      version: "v43",
      pods: 12,
      region: "eu-west-1",
    } as ToolResults[K]
 
    this.receipts.set(idempotencyKey, result)
    console.log(`[tool] executed ${name}`)
    return result
  }
}
 
type AgentContext = {
  runId: string
  taskId: string
  attempt: number
  goal: string
  inputs: Record<string, string>
  tools: ToolBroker
}
 
interface Agent {
  readonly id: string
  run(context: AgentContext): Promise<string>
}
 
class InventoryAgent implements Agent {
  readonly id = "inventory-agent"
 
  async run(context: AgentContext): Promise<string> {
    const deployment = await context.tools.call(
      this.id,
      "deployments.read",
      { service: "checkout" },
      `${context.runId}:${context.taskId}:deployments.read`
    )
    return `Deployment ${deployment.service}@${deployment.version}: ${deployment.pods} pods in ${deployment.region}`
  }
}
 
class RiskAgent implements Agent {
  readonly id = "risk-agent"
  private failFirstAttempt = true
 
  async run(): Promise<string> {
    if (this.failFirstAttempt) {
      this.failFirstAttempt = false
      throw new TransientError("model timeout")
    }
    return "Risk review: backward-compatible schema; tested rollback available"
  }
}
 
class VerifierAgent implements Agent {
  readonly id = "verifier-agent"
 
  async run(context: AgentContext): Promise<string> {
    const inventory = context.inputs.inventory
    const risk = context.inputs.risk
    if (!inventory || !risk) throw new Error("missing dependency artifact")
    return `APPROVE — ${inventory}; ${risk}`
  }
}
 
type TaskOutcome =
  | { ok: true; taskId: string; attempts: number; output: string }
  | { ok: false; taskId: string; attempts: number; error: string }
 
class Supervisor {
  constructor(
    private readonly store: CheckpointStore,
    private readonly tools: ToolBroker,
    private readonly agents: Map<string, Agent>,
    private readonly crashAfterBatch?: number
  ) {}
 
  async run(runId: string, goal: string): Promise<RunState> {
    let state = await this.store.load(runId)
 
    if (state) {
      console.log(`[recovery] loaded ${runId} at v${state.version}`)
    } else {
      state = await this.store.save(this.createRun(runId, goal), 0)
    }
 
    let batchNumber = 0
    while (state.status === "running") {
      if (state.tasks.every((task) => task.status === "completed")) {
        state.status = "completed"
        state = await this.store.save(state, state.version)
        console.log(`[run] completed at v${state.version}`)
        return state
      }
 
      const failed = state.tasks.find((task) => task.status === "failed")
      if (failed) {
        state.status = "failed"
        state = await this.store.save(state, state.version)
        throw new Error(`run failed at ${failed.id}: ${failed.lastError}`)
      }
 
      const completed = new Set(
        state.tasks
          .filter((task) => task.status === "completed")
          .map((task) => task.id)
      )
      const ready = state.tasks.filter(
        (task) =>
          task.status === "pending" &&
          task.dependsOn.every((dependency) => completed.has(dependency))
      )
 
      if (ready.length === 0) throw new Error("task graph is blocked or cyclic")
      console.log(`[run] ready: ${ready.map((task) => task.id).join(", ")}`)
 
      // Workers execute concurrently; only the supervisor commits state.
      const outcomes = await Promise.all(
        ready.map((task) => this.executeTask(state!, task))
      )
 
      for (const outcome of outcomes) {
        const task = state.tasks.find((candidate) => candidate.id === outcome.taskId)!
        task.attempts = outcome.attempts
        if (outcome.ok) {
          task.status = "completed"
          task.output = outcome.output
        } else {
          task.status = "failed"
          task.lastError = outcome.error
        }
        state = await this.store.save(state, state.version)
        console.log(`[checkpoint] ${task.id} -> ${task.status} (v${state.version})`)
      }
 
      batchNumber += 1
      if (batchNumber === this.crashAfterBatch) {
        throw new Error("simulated process crash")
      }
    }
 
    return state
  }
 
  private async executeTask(state: RunState, task: Task): Promise<TaskOutcome> {
    const agent = this.agents.get(task.agentId)
    if (!agent) {
      return { ok: false, taskId: task.id, attempts: 0, error: "agent missing" }
    }
 
    const inputs = Object.fromEntries(
      task.dependsOn.map((dependency) => {
        const source = state.tasks.find((candidate) => candidate.id === dependency)
        return [dependency, source?.output ?? ""]
      })
    )
 
    for (let attempt = task.attempts + 1; attempt <= task.maxAttempts; attempt += 1) {
      try {
        const output = await agent.run({
          runId: state.runId,
          taskId: task.id,
          attempt,
          goal: state.goal,
          inputs,
          tools: this.tools,
        })
        return { ok: true, taskId: task.id, attempts: attempt, output }
      } catch (error) {
        const message = error instanceof Error ? error.message : String(error)
        const retryable = error instanceof TransientError
        console.log(`[agent] ${task.id} attempt ${attempt} failed: ${message}`)
        if (!retryable || attempt === task.maxAttempts) {
          return { ok: false, taskId: task.id, attempts: attempt, error: message }
        }
      }
    }
 
    return { ok: false, taskId: task.id, attempts: task.maxAttempts, error: "exhausted" }
  }
 
  private createRun(runId: string, goal: string): RunState {
    return {
      runId,
      goal,
      version: 0,
      status: "running",
      tasks: [
        { id: "inventory", agentId: "inventory-agent", dependsOn: [], status: "pending", attempts: 0, maxAttempts: 2 },
        { id: "risk", agentId: "risk-agent", dependsOn: [], status: "pending", attempts: 0, maxAttempts: 2 },
        { id: "verify", agentId: "verifier-agent", dependsOn: ["inventory", "risk"], status: "pending", attempts: 0, maxAttempts: 1 },
      ],
    }
  }
}
 
async function main() {
  const store = new InMemoryCheckpointStore()
  const tools = new ToolBroker()
  const agents = new Map<string, Agent>([
    ["inventory-agent", new InventoryAgent()],
    ["risk-agent", new RiskAgent()],
    ["verifier-agent", new VerifierAgent()],
  ])
 
  try {
    await new Supervisor(store, tools, agents, 1).run(
      "run-42",
      "Assess the checkout deployment"
    )
  } catch (error) {
    console.log(`[process] ${(error as Error).message}`)
  }
 
  const recovered = await new Supervisor(store, tools, agents).run(
    "run-42",
    "Assess the checkout deployment"
  )
  console.log(recovered.tasks.find((task) => task.id === "verify")?.output)
}
 
void main()

Expected behavior (the exact order of parallel worker logs may vary):

[run] ready: inventory, risk
[tool] executed deployments.read
[agent] risk attempt 1 failed: model timeout
[checkpoint] inventory -> completed (v2)
[checkpoint] risk -> completed (v3)
[process] simulated process crash
[recovery] loaded run-42 at v3
[run] ready: verify
[checkpoint] verify -> completed (v4)
[run] completed at v5
APPROVE — Deployment checkout@v43: 12 pods in eu-west-1; Risk review: backward-compatible schema; tested rollback available

This example keeps persistence in memory so it is easy to run. In production, replace it with a transactional database or workflow engine, add leases and heartbeats, use runtime schema validation, encrypt sensitive artifacts, and put jittered backoff around eligible retries. The optimistic version check matters: it prevents two supervisors from silently overwriting each other's decisions. The compact demo checkpoints terminal task outcomes; a production scheduler should reserve the attempt, lease, and budget transactionally before dispatch so a crash cannot forget in-flight work. Tool receipts must be durable too, not held in the broker process as they are here.

Observability and evaluation

A green model API status does not mean the workflow succeeded. Trace one request as a hierarchy:

run
├── plan (model, prompt version, plan version)
├── task:inventory
│   └── tool:deployments.read
├── task:risk
│   ├── attempt:1 (timeout)
│   └── attempt:2 (success)
├── recovery (checkpoint v3)
└── task:verify (artifact verdict)

Record identifiers, timestamps, model and prompt versions, task status, attempt, latency, token and cost usage, tool name, policy decision, idempotency key hash, artifact reference, and failure class. Do not record credentials, raw private context, or unrestricted tool arguments. OpenTelemetry publishes GenAI semantic attributes for workflows, agent invocation, tool execution, and usage, while warning that tool arguments and results can contain sensitive information.

Evaluate at three levels:

• Node: Did the worker satisfy its schema and evidence requirements?
• Workflow: Did the system complete the user goal within quality, latency, safety, and cost thresholds?
• Recovery: After injected timeouts, duplicate deliveries, process crashes, stale versions, and dependency outages, did the run converge to a correct terminal state without duplicate business effects?

Keep a labeled end-to-end evaluation set. Break failures down into planning, routing, context, tool selection, tool execution, memory, verification, policy, and recovery. Optimizing a single-agent answer score can hide a workflow that is slow, unsafe, or unable to resume.

Common design mistakes

Letting the LLM own the workflow state

If the only record of completed work is a conversation, a restart cannot distinguish committed effects from proposed ones. Persist explicit states and receipts; rebuild prompts from them.

Treating every failure as retryable

Retries are correct for some transient reads and idempotent operations. They are dangerous for validation failures, policy denials, bad plans, and ambiguous writes. Classify first.

Giving every worker every tool

This increases prompt complexity and blast radius. Specialization should reduce capability, not just change the system prompt.

Using one mutable shared transcript as memory

It spreads stale or hallucinated claims and creates hidden dependencies. Share typed artifacts with provenance and verification state.

Adding a verifier with no independent signal

A verifier that sees the same prompt, context, and model failure mode may only rephrase the original answer. Give it a concrete rubric, deterministic checks, independent evidence, or a different source of information.

Forgetting cancellation and late results

A timed-out worker may still finish. Mark a run and plan version on every result; reject or quarantine results from cancelled runs, expired leases, and superseded plans.

Production checklist

Architecture and control

• Each agent exists for a documented specialization, isolation, parallelism, or ownership reason.
• The orchestrator owns the task DAG, state machine, termination, budgets, deadlines, and plan versions.
• Deterministic workflow nodes handle known logic; models handle uncertain classification, planning, or generation.
• Fan-out, recursion depth, rounds, tokens, tool calls, cost, and wall time have enforced limits.

Memory and data

• Run checkpoints, task context, durable knowledge, artifacts, and systems of record are separate.
• Every shared artifact has schema, owner, provenance, verification status, version, and retention policy.
• Agents receive the minimum context needed; tenant and user boundaries are preserved.
• Concurrent writes use compare-and-set, transactions, or another explicit conflict policy.

Tools and security

• Tool inputs and outputs are validated at runtime.
• Tool permissions are enforced outside prompts using verified identity and least privilege.
• Sensitive actions require current policy checks and, where appropriate, out-of-band human approval.
• Idempotency keys, status lookup, and reconciliation exist for side effects.
• Secrets and sensitive payloads are redacted from prompts, traces, logs, and durable receipts.

Reliability and quality

• State transitions are durably checkpointed and recovery is tested across a real process restart.
• Retry rules differ for transient reads, idempotent writes, ambiguous writes, validation errors, and policy denials.
• Leases, heartbeats, cancellation, late-result handling, circuit breakers, and dead-letter or human-review paths are defined.
• End-to-end evaluations measure goal success, safety, latency, and cost, with failures decomposed by architecture stage.
• Deployments preserve in-flight run compatibility or pin each run to its prompt, model, tool, policy, and state-schema versions.

How to explain this in a system design interview

Lead with the control boundary, not a list of frameworks:

I would start with one agent and split only where independent parallel work, specialist context, security isolation, or separate ownership justifies the coordination cost. A durable orchestrator owns a versioned task DAG, budgets, deadlines, and state transitions. Workers receive bounded objectives, scoped context, and least-privilege typed tools, then return versioned artifacts. I checkpoint after meaningful transitions, verify artifacts before side effects, and recover by classifying failures: retry safe transient work, reconcile ambiguous writes by idempotency key, replan quality failures, and escalate policy or budget decisions to a human.

Then make the design concrete with one task path, one failure, and one recovery. For example: two analysis workers run in parallel; a tool write times out; the orchestrator queries its operation receipt, observes that the write committed, records completion, and resumes the verifier without repeating the action.

Realistic follow-up questions

Why not use one larger model?

A larger model may improve reasoning but does not create permission isolation, independent deployment, or lower latency through parallelism. If those boundaries do not matter, one model plus deterministic workflow code is probably better.

Does the supervisor become a bottleneck?

The supervisor is logically centralized, not necessarily one process. Partition runs by runId, keep scheduling stateless around a durable store, use optimistic concurrency or leases, and scale workers independently. Keep global coordination off the hot path.

How do you prevent two agents from corrupting shared memory?

Prefer immutable artifacts. For mutable keys, define one owner or use versions and compare-and-set. Store provenance and verification status, and never treat a worker's output as authoritative business state.

What happens if the process crashes after a tool committed?

On recovery, the operation is ambiguous. Look it up with the same idempotency key or durable operation ID, reconcile the remote state, then advance the checkpoint. Do not infer failure from a missing response.

How do you stop agent loops?

Enforce maximum steps, fan-out, repeated-call detection, budgets, deadlines, and terminal conditions in the orchestrator. Give the planner structured prior outcomes so it does not rediscover the same failed action.

How would you test the design?

Use deterministic fake agents and tools for state-machine tests, then inject timeouts, duplicate messages, stale versions, lost responses, process crashes, and verifier rejection. Finally run labeled end-to-end evaluations with real models and track failure categories, cost, latency, and unsafe action rate.

For more practice, the Agentic Systems & Tool Use assessment tests these exact production judgments. The AI Architect track places them in a broader skills path, while the Architecture Judgment Index and Production AI Systems course help you rehearse the Staff-level trade-offs behind the design.

The senior-level conclusion

The hardest part of multi-agent architecture is not getting agents to talk. It is deciding who owns state, authority, truth, and recovery when several probabilistic workers act concurrently.

A strong design keeps those responsibilities explicit: deterministic control, bounded delegation, scoped memory, brokered tools, durable transitions, evidence-based verification, and failure-specific recovery. If a second agent does not make one of those properties better—or unlock valuable parallel work—it does not belong in the system yet.