Latency: Time to First Token, Time per Token & Total Response Time
This lesson is part of the course
In this lesson
You estimate how long five kinds of Kestrel requests take: waiting, reading the prompt (prefill) and writing the answer (decode).
Buy the course once for $9.99 to unlock every lesson and lab, the final exam and the certificate – lifetime access, no subscription.
Lessons in this course
- Model Memory: Parameters × Bytes per Parameter (FP16, INT8, INT4) Locked
- KV-Cache Memory per Token & Max Concurrent Sequences Locked
- Throughput vs Batch Size: Tokens per Second per GPU and per User (Chart) Locked
- Latency: Time to First Token, Time per Token & Total Response Time Locked
- GPUs for the Peak: Little's Law, Throughput & ROUNDUP Locked
- Utilisation & Cost per 1M Tokens from the GPU Hourly Price Locked
- Self-Hosting vs API: Monthly Cost & the Break-Even Volume (Chart) Locked
- Capacity Plan: Six Months of Growth, Headroom, GPUs & Budget Locked