Write down the response your users need
Imagine a support assistant answering short questions while another team submits long documents for analysis. A successful demonstration with one request does not establish capacity for both workflows during a busy period. This hypothetical example suggests a useful starting point: agree what an acceptable response means for each task. Set expectations for the first visible output, the completed answer and the handling of requests that cannot be served promptly.
Build the request mix before choosing capacity
NVIDIA’s AIPerf guidance describes testing variable input and output lengths, arrival patterns and load levels. Our proposed trial begins with a representative mix of short questions, longer context and the output lengths the application actually needs. Include expected busy periods and bursts. Use synthetic or appropriately authorised data, and record the model, hardware and serving configuration so that a later comparison measures a deliberate change. NVIDIA: Benchmarking LLM Inference at Scale with AIPerf.
Measure waiting and completion separately
AIPerf reports first-token delay, timing during generation, total request latency and output-token throughput, with percentile breakdowns. For the business review, show the slower requests alongside the average. A quick first response and a promptly completed answer are different acceptance conditions. Check the measurement definitions before comparing tools, and include the application’s own retrieval, validation and network steps when evaluating the experience people receive. NVIDIA: Benchmarking LLM Inference at Scale with AIPerf.
Count requests that meet the service target
The vLLM serving benchmark accepts service-level objectives for goodput using first-token delay, time per output token or end-to-end latency. Our recommendation is to judge capacity against agreed timing targets, rather than the largest raw throughput number alone. Review failed and rejected requests as well as completed ones. Keep answer quality as a separate acceptance check: meeting a timing target does not establish that a response is correct or useful. vLLM: Serving benchmark reference.
Make the decision reproducible
Repeat the workload at different planned load levels and retain the configuration and results. Document what happens when the target is exceeded, including queue limits, user feedback and the owner responsible for scaling decisions. Compare compute costs over the same workload and include unsuccessful attempts in the cost review. Choose capacity with an explicit allowance for variation, then revisit the evidence when request patterns, models or application requirements change.