Design reviewAdvanced120 min estimateOriginal publication

Parallelism for Inference

Choose a distributed inference topology from measured model, memory, latency, throughput, and link evidence instead of treating more accelerators as automatically faster.

Authorship: InterviewsVector
Published / updated: 2026-09-26 / 2026-09-26
Review status: Artifact tests passing · primary sources recorded

Original InterviewsVector teaching. Executable artifacts are deterministic illustrative audits with focused tests and recorded primary sources; they do not claim hardware benchmarks, distributed execution, production deployment, capacity certification, or universal model quality.

The decision in one pass

Use the smallest parallel topology that satisfies model residency, latency, throughput, and failure-domain requirements on the actual interconnect. First ask whether one replica fits and meets the workload; a single device avoids distributed collectives. If it does not, partition tensors within a fast-link domain, pipeline stages across larger boundaries only when necessary, shard experts when a mixture-of-experts layer makes that useful, and add data-parallel replicas for capacity and isolation. Treat tensor, pipeline, expert, and data degrees as a placement plan tied to exact ranks, nodes, model revision, context distribution, precision, scheduler, and kernels. Benchmark complete candidate topologies because communication volume, synchronization, bubbles, routing imbalance, KV placement, and batching interact. Preserve memory and link headroom, reject incomplete telemetry, and select only among measured eligible candidates. A synthetic planner can make the contract reviewable; it cannot predict unmeasured hardware or certify the chosen topology.

Why this matters

Parallelism solves a local constraint by creating a distributed one. Tensor sharding can fit weights but add a collective on the token path. Pipeline stages can cross nodes but introduce bubbles and stage imbalance. Expert parallelism can reduce per-rank expert residency while creating all-to-all traffic and hot experts. Data parallelism increases independent serving capacity, replicates weights, and provisions separate KV capacity and state per replica. A plan that ignores physical links, request shapes, scheduler behavior, and failure domains can use more accelerators while reducing useful throughput and making recovery harder.

You will be able to

Separate model residency, single-request latency, aggregate capacity, and failure isolation as distinct reasons to distribute inference.
Explain tensor, pipeline, expert, and data parallelism by the state partitioned, communication introduced, and placement boundary required.
Map logical parallel groups onto ranks, nodes, fast-link islands, network links, and replica failure domains.
Compare candidate topologies with exact model, workload, kernel, scheduler, memory, latency, throughput, and link evidence.
Defend why an eligible measured topology is a bounded local choice rather than a universal configuration recommendation.

Prerequisite contract

Your Vector Loop for this lab

01
Model
Map model layers and experts, precision, weights, KV state, context and batch distributions, latency and throughput objectives, devices, links, nodes, scheduler, and failure domains.
02
Derive
Derive residency and communication hypotheses for tensor, pipeline, expert, and data groups, naming where synchronization and imbalance enter the request path.
03
Build
Build a deterministic planner over invented, version-bound topology trials; require exact world-size decomposition, placement, telemetry, memory, latency, throughput, and link gates.
04
Stress
Inject cross-node tensor groups, incomplete telemetry, memory pressure, tail-latency loss, low throughput, saturated links, hot experts, stage bubbles, rank loss, and stale benchmark evidence.
05
Operate
Monitor per-rank memory, collective and all-to-all time, link utilization, queueing, KV occupancy, stage balance, expert load, throughput, tail latency, errors, and replica health by topology revision.
06
Defend
Defend the smallest eligible topology, the workload and hardware boundary of the measurements, the failure behavior, and which result would force a different decomposition.

Start with the constraint, not a favorite parallel mode

One accelerator is the baseline because it has no cross-rank synchronization. Distribute only when measured residency, latency, throughput, tenancy, or availability requirements demand it. Freeze model and tokenizer revisions, precision, maximum and observed sequence shapes, KV policy, scheduler, kernel stack, device type, topology, and SLO before comparing plans. Otherwise a faster candidate may simply have received easier requests or a different serving policy.

Need	First question	Likely response
weights or KV do not fit	which state must be partitioned?	tensor, pipeline, expert partitioning, compression, or bounded offload
one replica misses capacity	can requests be isolated across replicas?	data-parallel replicas plus routing and admission control
one request misses latency	does added communication reduce enough compute?	benchmark an intra-node tensor group before wider distribution
failure blast radius is too large	which ranks fail together?	smaller replicas, explicit placement, redundancy, and load shedding

Name the state and communication of each dimension

Tensor parallelism partitions operations or tensors and usually requires collectives inside layers, so it belongs on the fastest links available. Pipeline parallelism assigns layer ranges to stages; it reduces per-rank residency but creates stage transfers and bubbles whose cost depends on request scheduling and microbatching. Expert parallelism distributes experts and introduces token routing, typically with all-to-all communication and load imbalance risk. Data parallelism creates independent replicas for different request groups, replicating model state while scaling aggregate capacity and providing cleaner failure isolation.

illustrative_world_size = tensor_degree × pipeline_degree × expert_degree × data_degree

The artifact uses this deliberately closed decomposition to make rank accounting auditable. Real frameworks may compose, share, or constrain groups differently, so the framework's actual process groups and placement remain authoritative.

Dimension	Primary partition	Dominant risks
tensor	weights and intermediate tensor work	fine-grained collectives, synchronization, topology sensitivity
pipeline	ordered layer or stage ranges	bubbles, stage imbalance, activation transfer, recovery coupling
expert	expert weights and routed tokens	all-to-all traffic, hot experts, dropped or delayed tokens
data	requests across complete replicas	weight replication, fragmented KV capacity and affinity, router skew, cold replicas

Bind logical groups to physical links and a workload

01Draw the rank mapPlace every tensor, pipeline, expert, and replica group on device and node identifiers; label bandwidth, contention domain, and failure boundary for each crossing.
02Budget persistent and transient stateInclude weights, KV cache, activations, communication buffers, graph or kernel workspaces, allocator headroom, uneven partitions, and recovery overlap per rank.
03Benchmark the complete serving pathUse representative prompt/output distributions, concurrency, batching, cache occupancy, cancellations, routing, warmup, and failure behavior; capture tails and useful throughput together.
04Keep provenance with every resultBind model, topology, hardware, driver, runtime, kernel, scheduler, workload, and telemetry versions so a pass cannot authorize a changed system silently.

Select among bounded topology trials

parallelism_plan.py

1def plan_parallelism(contract: ParallelismContract, evidence: ParallelismEvidence) -> ParallelismPlan:
2    contract = validate_record(contract, ParallelismContract)
3    evidence = validate_record(evidence, ParallelismEvidence)
4    if evidence.scope != contract.scope or evidence.contract_content_id != contract.content_id:
5        raise ValueError("evidence belongs to another parallelism contract")
6    # Reject malformed rank decompositions, filter candidates through hard
7    # placement and performance gates, then choose by deterministic ordering.

Expected output

example=illustrative_only
decision=SELECT_TOPOLOGY
topology=tp4-pp2-ep2-dp1
world_size=16;nodes=2;gpus_per_node=8
peak_memory_gib=50.000;ttft_ms=420.000
tokens_per_second=960.000;inter_node_link_utilization=0.600
claim=LOCAL_TOPOLOGY_AUDIT_NOT_PRODUCTION_BENCHMARK

Verify: python3 -m unittest discover -s courses/ai-engineering/reference-impl/inference_parallelism -p 'test_*.py' -v

The invented fixture compares three trials for a synthetic 64-billion-parameter MoE profile on two eight-device nodes. The chosen `tp4-pp2-ep2-dp1` trial is the highest-throughput candidate that also satisfies declared placement, telemetry, memory, TTFT, throughput, and inter-node-link gates. The implementation binds model and contract digests, checks exact world size, within-node tensor-group packability, and expert divisibility, copies frozen trial sequences, and rejects malformed types, booleans, nonfinite values, string subclasses, constructor bypass, duplicates, stale scope, and tampered evidence.

All profiles and measurements are invented. The program does not allocate devices, run collectives, model communication, validate numerical equivalence, estimate cost, or predict an unmeasured topology. Its deterministic ordering teaches how to review evidence, not which degrees to deploy.

Stress the topology at its synchronization points

Failure	Misleading symptom	Required defense
cross-node fine-grained collective	weights fit, but TTFT grows	keep chatty groups inside fast-link domains or measure the crossing explicitly
pipeline imbalance	some ranks appear underutilized	profile stage critical paths and scheduler bubbles by request shape
hot experts	average expert load looks healthy	measure per-expert routing, queueing, drops, and all-to-all tails
router or replica skew	cluster utilization is high	observe per-replica queue and KV pressure, then rebalance without breaking affinity
rank loss	orchestrator replaces a process	define replica failure semantics, drain or abort policy, model reload, and traffic recovery

Failure injection must include slow ranks and links, not only complete loss. A synchronized group advances at the pace of its straggler. Validate cancellation propagation, partial response behavior, timeouts, communicator teardown, model reload, KV invalidation, and whether surviving replicas can accept the shifted traffic without cascading overload.

Operate a topology as a versioned contract

Store the chosen rank map and rejected alternatives with their exact measurements. Monitor per-rank resident and reserved memory, KV occupancy, queue time, prefill and decode latency, useful token throughput, collective and all-to-all time, link utilization, stage idle time, expert balance, errors, retries, cancellations, power, and recovery. Segment by request shape and topology group; cluster averages erase the rank that controls the tail.

Re-run qualification after model, precision, context, batching, kernel, driver, runtime, hardware, link, placement, or scheduler changes.
Reserve memory, bandwidth, and replica headroom for uneven shards, bursts, maintenance, rank recovery, and failover rather than operating at the synthetic limit.
Keep admission control and load shedding independent of a distributed replica whose ranks may be slow or unavailable.
Make topology ownership explicit across serving, model, kernel, networking, capacity, and incident-response teams.

Operate at three altitudes

Production lens

— Observe per-rank memory, KV state, queueing, collectives, all-to-all traffic, link utilization, stage and expert imbalance, useful throughput, tails, and errors under the exact request distribution.
— Exercise slow and failed ranks, links, nodes, model loads, drains, cancellations, communicator teardown, traffic shift, and overload so the topology has a known recovery contract.
— Version topology evidence with model, precision, kernels, scheduler, runtime, driver, hardware, placement, workload, and telemetry; a pass expires when any material identity changes.

Staff lens

— Make parallelism a cross-layer capacity decision owned jointly by model, kernel, serving, networking, scheduling, reliability, and finance teams rather than a framework flag.
— Require every extra synchronization boundary to resolve a named constraint and carry a measured latency, throughput, failure, and operational cost.

Interview defense

A 64-billion-parameter MoE model must serve long-context traffic across two eight-GPU nodes. How do you choose the parallel topology?

I would first freeze precision, weight and KV residency, prompt/output distributions, concurrency, scheduler, SLOs, and the physical link map, then test whether a smaller single-node replica works. I would keep fine-grained tensor groups inside the fastest node links, use pipeline stages across nodes only if residency requires them, choose expert groups from expert placement and all-to-all measurements, and add data replicas for independent capacity. For each complete candidate I would measure per-rank memory, TTFT, token throughput, collectives, all-to-all tails, link utilization, stage and expert balance, quality, and failure recovery. I would reject any candidate that misses a hard gate, preserve headroom, and choose among the remaining frontier. The result is bound to that model, workload, stack, and topology—not a universal degree recipe.

Expect the interviewer to press on

— Why should tensor parallel groups usually stay within a fast-link domain?
— How does expert routing change the measurements you need?
— When would more data-parallel replicas be preferable to a larger model-parallel replica?

Misconceptions to remove

“If the parallel degrees multiply to the GPU count, the topology is valid.”

The arithmetic only accounts for ranks. Residency, physical placement, communication, scheduler behavior, numerical equivalence, tail latency, throughput, failure, and cost still require evidence.

“Tensor parallelism always reduces inference latency because each device computes less.”

It also introduces communication and synchronization inside the token path; on slow or contended links, latency can increase even when compute per rank falls.

“Data parallelism scales capacity linearly.”

Replica cold starts, routing skew, KV affinity, memory replication, shared dependencies, admission limits, and failover headroom can prevent linear useful capacity.

Check your model

1. What is the first question before adding inference parallelism?

Which measured constraint—residency, latency, aggregate capacity, tenancy, or availability—cannot be met by a smaller and simpler replica?

2. Why is expert parallelism not just tensor parallelism for MoE weights?

Requests route tokens to selected experts, creating expert-specific residency, all-to-all communication, imbalance, queueing, and drop or delay behavior that must be measured.

3. What does the artifact's selected topology prove?

Only that one supplied invented trial satisfies the declared local gates and wins the deterministic ordering among eligible trials; it proves nothing about unmeasured hardware or workloads.

Prove the mechanism

Extend the planner with explicit failure-domain and cost gates. Add a candidate that is fastest but crosses the permitted blast radius, prove it is rejected, and preserve deterministic evidence IDs and no mutation.

Add a production constraint

Design and defend a benchmark matrix for dense and MoE serving across one and two nodes. Include rank maps, weights and KV capacity, group communication, sequence distributions, batching, affinity, quality, power, cost, slow-rank injection, node loss, recovery, and topology-expiration rules.

Artifact: Inference parallelism topology planner

courses/ai-engineering/reference-impl/inference_parallelism/parallelism_plan.py

Download reference implementation

Primary references and next links

References

1. Parallelism and Scaling
vLLM. Official guidance on single-GPU and distributed tensor, pipeline, data, and expert parallel serving strategies.
2. Collective Operations
NVIDIA NCCL. Official definitions and rank-consistency requirements for collectives used by distributed execution.
3. DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale
Rajbhandari et al.. Primary research on multidimensional expert, tensor, data, and pipeline parallelism for MoE systems.

Continue through the graph

Continuous Batching and Queueing →
Benchmark topology choices under the scheduler and mixed request shapes that determine useful serving capacity.
The Model-Serving Control Plane →
Turn the chosen topology into placement, loading, health, rollout, scaling, and rollback state.

Glossary: tensor parallelism · pipeline parallelism · expert parallelism · data parallelism · process group · all-to-all · pipeline bubble · straggler · failure domain · rank map