GPUs for the Peak: Little's Law, Throughput & ROUNDUP
This lesson is part of the course
In this lesson
You size the GPU-A fleet for five Kestrel workloads at peak: by concurrent sequences (memory) and by tokens per second (throughput), whichever binds.
Buy the course once for $9.99 to unlock every lesson and lab, the final exam and the certificate – lifetime access, no subscription.
Lessons in this course
- Model Memory: Parameters × Bytes per Parameter (FP16, INT8, INT4) Locked
- KV-Cache Memory per Token & Max Concurrent Sequences Locked
- Throughput vs Batch Size: Tokens per Second per GPU and per User (Chart) Locked
- Latency: Time to First Token, Time per Token & Total Response Time Locked
- GPUs for the Peak: Little's Law, Throughput & ROUNDUP Locked
- Utilisation & Cost per 1M Tokens from the GPU Hourly Price Locked
- Self-Hosting vs API: Monthly Cost & the Break-Even Volume (Chart) Locked
- Capacity Plan: Six Months of Growth, Headroom, GPUs & Budget Locked