Model Memory: Parameters × Bytes per Parameter (FP16, INT8, INT4)
This lesson is part of the course
In this lesson
You work out how much GPU memory six model variants of Kestrel Labs' assistant need and whether they fit on one GPU-A or GPU-B.
Buy the course once for $9.99 to unlock every lesson and lab, the final exam and the certificate – lifetime access, no subscription.
Lessons in this course
- Model Memory: Parameters × Bytes per Parameter (FP16, INT8, INT4) Locked
- KV-Cache Memory per Token & Max Concurrent Sequences Locked
- Throughput vs Batch Size: Tokens per Second per GPU and per User (Chart) Locked
- Latency: Time to First Token, Time per Token & Total Response Time Locked
- GPUs for the Peak: Little's Law, Throughput & ROUNDUP Locked
- Utilisation & Cost per 1M Tokens from the GPU Hourly Price Locked
- Self-Hosting vs API: Monthly Cost & the Break-Even Volume (Chart) Locked
- Capacity Plan: Six Months of Growth, Headroom, GPUs & Budget Locked