What actually happens during a single AI inference: from fixed parameters to the trade-off between latency and throughput

The article explains that training is learning parameters, while inference is using parameters without changing them; a single inference run only performs forward propagation and does not require loss, gradients, or weight updates. The article breaks down the autoregressive generation of large language models, distinguishing the Prefill and Decode phases: the former is compute-intensive and determines time to first token, while the latter is limited by memory bandwidth and determines the inter-token interval. The article also provides an example of estimating KV cache usage, and discusses the benefit boundary of batching for improving throughput, the unbounded growth of waiting time as utilization approaches 100%, and how chat scenarios prioritize reducing TTFT while Agent scenarios focus on end-to-end latency.