LLM Architecture Fundamentals
Attention, KV cache, decoding strategies, and the failure modes that only show up at production scale.
Architect Ā· 12 questions Ā· 18 min
Question 1 of 12Answered: 0 / 12
Your team is serving a 70B parameter model. Inference latency is dominated by the autoregressive decode step, not prefill. Which optimization will most directly reduce per-token decode latency?