InterviewsVector
Original Academy

Legacy mirror · noindex · upstream phase 17

Infrastructure & Production

Ship AI to the real world.

Provenance: this phase outline and its lesson readings are preserved from ai-engineering-from-scratch by Rohit Ghumare under the MIT License. InterviewsVector does not claim authorship. These archive pages remain available for old links and progress, but are excluded from indexing.

Attributed readings

  1. 01Managed LLM Platforms — Bedrock, Azure OpenAI, Vertex AIThree hyperscalers, three distinct strategies. AWS Bedrock is a model marketplace — Claude, Llama, Titan, Stability, Cohere behind one API. Azure OpenAI is an exclusive OpenAI p…
  2. 02Inference Platform Economics — Fireworks, Together, Baseten, ModalThe 2026 inference market is no longer GPU time rental. It bifurcates into custom silicon (Groq, Cerebras, SambaNova), GPU platforms (Baseten, Together, Fireworks, Modal), and A…
  3. 03GPU Autoscaling on Kubernetes — Karpenter, KAI SchedulerThree layers, not one. Karpenter provisions nodes dynamically (under one minute, 40% faster than Cluster Autoscaler). KAI Scheduler handles gang scheduling, topology awareness, …
  4. 04vLLM Serving Internals — PagedAttention, Continuous Batching, Chunked PrefillvLLM's dominance in 2026 rests on three compounding defaults, not a single trick. PagedAttention is always on. Continuous batching injects new requests into the active batch bet…
  5. 05EAGLE-3 Speculative Decoding in ProductionSpeculative decoding pairs a fast draft model with the target model. The draft proposes K tokens; the target verifies in a single forward; accepted tokens are free. In 2026, EAG…
  6. 06SGLang and RadixAttention for Prefix-Heavy WorkloadsSGLang treats the KV cache as a first-class, reusable resource stored in a radix tree. Where vLLM schedules requests FCFS (first-come, first-served), SGLang's cache-aware schedu…
  7. 07TensorRT-LLM on Blackwell with FP8 and NVFP4TensorRT-LLM is NVIDIA-only but it wins on Blackwell. On GB200 NVL72 with Dynamo orchestration, SemiAnalysis InferenceX measured $0.012 per million tokens on a 120B model in Q1-…
  8. 08Inference Metrics — TTFT, TPOT, ITL, Goodput, P99Four metrics decide whether an inference deployment is working. TTFT is prefill plus queue plus network. TPOT (equivalently ITL) is the memory-bound decode cost per token. End-t…
  9. 09Production Quantization — AWQ, GPTQ, GGUF, FP8, NVFP4Quantization format is not a universal choice — it is a function of hardware, serving engine, and workload. GGUF Q4_K_M or Q5_K_M owns CPU and edge, delivered through llama.cpp …
  10. 10Cold Start Mitigation for Serverless LLMsA 20 GB model image takes 5-10 minutes (7B) to 20+ minutes (70B) to go from cold to serving. In a true serverless world, that is not a warm-up — it is an outage. Mitigations ope…
  11. 11Multi-Region LLM Serving and KV Cache LocalityRound-robin load balancing is actively harmful for cached LLM inference. A request that does not land on the node holding its prefix pays full prefill cost — roughly 800 ms at P…
  12. 12Edge Inference — ANE, Hexagon, WebGPU, JetsonThe core edge constraint is memory bandwidth, not compute. Mobile DRAM sits at 50-90 GB/s; datacenter HBM3 clears 2-3 TB/s — a 30-50x gap. Decode is memory-bound so the gap is d…
  13. 13LLM Observability Stack SelectionThe 2026 observability market splits into two categories. Development platforms (LangSmith, Langfuse, Comet Opik) bundle monitoring with evals, prompt management, session replay…
  14. 14Prompt Caching and Semantic Caching Economics**Pricing snapshot dated 2026-04.** Numeric claims below reflect vendor rate cards captured at this lesson's publication; verify against the linked docs before quoting them down…
  15. 15Batch APIs — the 50% Discount as Industry StandardEvery major provider ships an async batch API with a 50% discount and ~24-hour turnaround. OpenAI, Anthropic, Google, and most of the inference platforms (Fireworks batch tier, …
  16. 16Model Routing as a Cost-Reduction PrimitiveA dynamic broker evaluates every request (task type, token length, embedding similarity, confidence) and sends simple queries to a cheap model, escalating complex ones to a fron…
  17. 17Disaggregated Prefill/Decode — NVIDIA Dynamo and llm-dPrefill is compute-bound; decode is memory-bound. Running both on the same GPU wastes one resource. Disaggregation splits them onto separate pools and transfers KV cache between…
  18. 18vLLM Production Stack with LMCache KV OffloadingvLLM's production-stack is the reference Kubernetes deployment — router, engines, and observability wired together. LMCache is the KV-offloading layer that extracts KV cache out…
  19. 19AI Gateways — LiteLLM, Portkey, Kong, BifrostA gateway sits between your apps and model providers. Core features are provider routing, fallback, retries, rate limiting, secret references, observability, guardrails. Market …
  20. 20Shadow, Canary, and Progressive DeploymentLLM rollouts combine the hardest parts of software deployment: no unit tests, diffuse failure modes, delayed signals. The sequence is (1) shadow mode — duplicate prod requests t…
  21. 21A/B Testing LLM Features — GrowthBook and StatsigTraditional A/B testing was not built for non-deterministic LLMs. The critical distinction: evals answer "can the model do the job?" A/B tests answer "do users care?" Both are r…
  22. 22Load Testing LLM APIs — k6, LLMPerf, GenAI-PerfTraditional load testers were not designed for streaming responses, variable output lengths, token-level metrics, or GPU saturation. Two traps bite most teams. The GIL trap: Loc…
  23. 23SRE for AI — Multi-Agent Incident ResponseAI SRE uses LLMs grounded in infrastructure data (logs, runbooks, service topology) via RAG to automate investigation, documentation, and coordination phases. The 2026 architect…
  24. 24Chaos Engineering for LLM ProductionChaos engineering for LLMs is its own discipline in 2026. Prerequisites before running experiments in production: defined SLI/SLO, trace+metric+log observability, automated roll…
  25. 25Security — Secrets, PII Scrubbing, Audit LogsEliminate secret sprawl via centralized vaults (HashiCorp Vault, AWS Secrets Manager, Azure Key Vault). Never store credentials in config files, env files in VCS, spreadsheets. …
  26. 26Compliance — SOC 2, HIPAA, GDPR, EU AI Act, ISO 42001Multi-framework coverage is table stakes for 2026 enterprise deals. **EU AI Act**: in force since August 1, 2024. Most high-risk requirements enforce August 2, 2026. Fines up to…
  27. 27FinOps for LLMs — Unit Economics and Multi-Tenant AttributionTraditional FinOps breaks on LLM spend. Costs are token-transactions, not resource-uptime. Tags don't map — an API call is a transaction, not an asset. Engineering decisions (pr…
  28. 28Self-Hosted Serving Selection — llama.cpp, Ollama, TGI, vLLM, SGLangFour engines dominate self-hosted inference in 2026. Pick based on hardware, scale, and ecosystem. **llama.cpp** is fastest on CPU — widest model support, full control over quan…