How much GPU memory does Llama 70B actually need?
A message lands in your team chat: “We need to deploy the Llama 70B model. Can you figure out the GPU infrastructure we'll need?” If you're in DevOps, SRE, or platform engineering, this is exactly the kind of question you'll be expected to answer. Let's actually answer it.
Here's what's really filling up that memory ↓
Prefer to watch this one instead of reading it? This lesson also exists as a short video, covering the same ground: what actually fills up GPU memory, and how to size it for a real deployment.
It's not just the model
The honest answer to “how many GPUs do we need” starts with “it depends.” It depends on everything that's actually sitting in GPU memory while the model runs, and there isn't just one thing in there. There are five:
Model Weights
The parameters loaded onto the GPU.
KV Cache
Stored keys and values, so the model doesn't redo work for every new word.
Activations
Intermediate results while the model is actively computing.
Runtime Buffers
Temporary workspace the software needs mid-calculation.
CUDA / Framework Overhead
Driver, allocator, and framework memory, the cost of just running the show.
Model weights and the KV Cache dominate that list by far. Let's size each one up, one at a time.
What the model learned, stored as numbers
Parameters are what a model actually learned during training, the billions of numbers you met on Day 2. Every one of them has to sit in GPU memory for the model to use it.
Here's the part that trips people up: the same 70 billion parameters can take up wildly different amounts of memory, depending on how precisely each number is stored.
| Precision | Bits/Parameter | Bytes/Parameter | Approx. size, Llama 70B |
|---|---|---|---|
| FP32 | 32 | 4 | ~280 GB |
| FP16 / BF16 | 16 | 2 | ~140 GB |
| INT8 | 8 | 1 | ~70 GB |
| INT4 | 4 | 0.5 | ~35 GB |
Can you just... use less precision? Yes, and it's one of the most useful tricks in this field. It's called quantization: storing each number with fewer decimal places. Take pi, 3.14159265. Round it to 3.14, and you've lost a little precision, but you can still cook with it.
Quantization does the same thing to a model's parameters. A handful of popular methods, GPTQ, AWQ, GGUF, and formats like FP8, do exactly this: take a model trained in FP16, and compress it down to INT8 or INT4, often with surprisingly little loss in quality.
The part that quietly outgrows the model itself
Model weights are fixed, they don't change size no matter what you ask the model. The KV Cache is the opposite: it grows, live, with every word the model generates.
Here's why it exists. Remember from Day 1 that a model generates one token at a time, and each new token depends on everything written before it? Without a cache, the model would have to re-read the entire conversation from scratch for every single new token, wildly wasteful. Instead, it saves a small summary, called the key and value, for every token it's already processed, the KV Cache, and simply reuses that summary instead of recomputing it.
That's efficient for speed, but it comes at a real cost: memory. Here's the formula:
The first “2 bytes” is because we're using FP16. The second “×2” is because we're storing both a key and a value for every token. For Llama 70B: 80 layers × 8,192 hidden size × 2 bytes × 2 = 2.5 MB, per token, per user.
That sounds tiny. It isn't. Watch what happens as you scale up context length and users:
At 10 concurrent users, the KV Cache alone hits 800 GB, more than half the size of the model itself.
The overhead of just running the show
The last piece is everything that isn't the model or the cache, but still needs a slice of memory to operate: intermediate activations, temporary tensors, the CUDA runtime itself, framework buffers, memory fragmentation from the allocator, and, if you're spreading the model across multiple GPUs, communication buffers between them.
There's no single clean formula for this one, but a reasonable rule of thumb is to reserve about 10% of everything you've already counted.
So, how many GPUs does Llama 70B actually need?
| Component | Memory |
|---|---|
| Model Weights | 140 GB |
| KV Cache | 800 GB |
| Runtime Overhead | 94 GB |
| Total | ≈ 1,034 GB |
An NVIDIA H100 carries 80 GB of VRAM. Do the division, and this single deployment needs about 13 of them, before you've served a single real customer conversation outside of testing.
Does every deployment really need 1 TB?
Not necessarily, and this is worth sitting with. That terabyte number is a worst-case scenario, built on some fairly aggressive assumptions, stacked together:
- a 32k context window, a lot of conversation history kept around
- every user active at the same time
- FP16 precision, the least memory-efficient common choice
- no inference optimization applied at all
Change any one of those, and the number moves a lot. Quantize the model to INT4 instead of FP16, and model weights alone drop from 140 GB to 35 GB. Shrink the context window, and the KV Cache shrinks with it, proportionally. This is exactly why the “how many GPUs do we need” conversation always turns into a negotiation between cost, context length, and how many people you expect to use it at once.
Four things worth carrying forward
- Model weights are only the starting point.
- The KV Cache grows with every token, and every user.
- Concurrent users multiply the KV Cache, fast.
- Always reserve additional memory beyond the “obvious” numbers.
Say it back to me
Tap each card. Can you remember what it means, before you flip it?
Model weights are the starting point, not the whole answer. The cache and the crowd decide the rest.