GPU Inference Capacity Planning: Tokens per Second, Batching, Memory & Cost

What you'll build

What you will be able to do

Each lab ends with a concrete result that you build yourself and the checker confirms, cell by cell.

• Work out how much GPU memory six model variants of Kestrel Labs' assistant need and whether they fit on one GPU-A or GPU-B.
• Calculate the KV-cache memory of Kestrel's M-34 model per token and per sequence, and how many conversations fit on one GPU-A at different context lengths.
• Turn a measured throughput curve into tokens per second per user, find the largest batch that still feels fast, and chart the curve.
• Estimate how long five kinds of Kestrel requests take: waiting, reading the prompt (prefill) and writing the answer (decode).
• Size the GPU-A fleet for five Kestrel workloads at peak: by concurrent sequences (memory) and by tokens per second (throughput), whichever binds.
• Turn GPU-hour prices, throughput and utilisation into tokens per month, a monthly bill and the cost per 1 million tokens for five Kestrel setups.
• Compare an API priced per token with a self-hosted GPU-A fleet with fixed costs, find the break-even volume and chart both cost lines.
• Build Kestrel's six-month GPU plan: peak traffic growing 15% a month, 30% headroom, a spare GPU and the budget.