LLM GPU Memory Calculator
Work out the VRAM to run or fine-tune Llama, Mistral, Qwen or Gemma at any precision, with KV cache, LoRA and QLoRA, and see which GPUs it fits on.
The interactive tool loads here when JavaScript runs.
About this toolWhat is the LLM GPU Memory Calculator?
Choose a model, pick whether you want to serve it or train it, and this calculator adds up the GPU memory: weights at the precision you choose, the KV cache for your context length and batch, optimizer state and gradients for full fine-tuning, the adapters for LoRA, the 4-bit base for QLoRA, and activations. A bar shows where the memory goes, and a table shows which GPUs hold it and how many you need.
The presets carry the real shapes from each model's config.json on Hugging Face: layers, hidden size, attention and key-value heads, head dimension, MLP size and vocabulary. That matters because the KV cache depends on the number of key-value heads, not the parameter count, and grouped-query attention makes Llama 3.1 8B's cache four times smaller than a model of the same size without it. A custom option takes the numbers straight from any config.json.
- Ten model presets with config.json shapes, plus custom models
- Inference at FP32, BF16, FP8, INT8 or INT4 with a separate KV cache precision
- Full fine-tuning, LoRA and QLoRA with gradient checkpointing
- LoRA trainable parameter count by rank and target module
- Breakdown bar and GPU table from RTX 4090 to B200, MI300X and Apple Silicon
- Formulas and sources shown on the page; shareable link
Updated
How to use the LLM GPU Memory Calculator
- Pick a model preset, or choose Custom and copy the values from the model's config.json.
- Choose Inference, Full fine-tune, LoRA or QLoRA.
- For inference, set the weight and KV cache precision, the context length and the number of sequences served at once.
- For training, set sequence length, micro-batch size and gradient checkpointing; for LoRA and QLoRA also set the rank and target modules.
- Read the total and the breakdown, then the GPU table for how many cards of each type it would take, leaving some headroom.
Worked examples
Serving Llama 3.1 70B
Input
BF16 weights, 8,192-token context, 1 sequence, 10% overhead
Output
Weights 131.5 GB + KV cache 2.5 GB + overhead = about 147 GB: two H100 80 GB or one H200
Switching to INT4 cuts the weights to about 36 GB, so the same model fits a 48 GB L40S with room for a longer cache.
QLoRA on an 8B model
Input
Llama 3.1 8B, QLoRA r=16 on all linear layers, 2,048 tokens, batch 1, checkpointing
Output
About 8.5 GB in total with 41.9 million trainable parameters
This fits a 24 GB RTX 4090 with room for a larger batch or longer sequences.
Full fine-tune of the same model
Input
Llama 3.1 8B, full fine-tune, 2,048 tokens, batch 1, checkpointing
Output
About 134 GB, mostly optimizer state
At 16 bytes per parameter the states alone are 120 GB, so full fine-tuning needs two 80 GB GPUs with ZeRO-3 or FSDP sharding.
Background and insight
Weights take parameters times bytes per weight: 4 in FP32, 2 in BF16 or FP16, 1 in FP8 or INT8, and about half a byte in 4-bit formats such as GPTQ, AWQ or GGUF Q4. Quantised checkpoints usually keep the embedding and output head in 16-bit, which the calculator accounts for; on models with large vocabularies such as Gemma 2 (256,000 tokens) that is close to a billion parameters left unquantised.
During generation every token in flight keeps a key and a value vector in every layer. The cache size is 2 x layers x KV heads x head dimension x bytes per value x tokens x batch. For Llama 3.1 8B in BF16 that is 2 x 32 x 8 x 128 x 2 = 131,072 bytes per token, so 8,192 tokens take exactly 1 GiB, and a full 128K context takes 16 GiB, more than the model's 15 GiB of weights.
Training needs much more. Mixed-precision Adam keeps 16 bytes per parameter (2 for 16-bit weights, 2 for gradients, 12 for the FP32 master copy and two Adam moments), the figure the ZeRO paper uses, plus activations that grow with sequence length and batch. LoRA (Hu et al. 2021) freezes the base model and trains small low-rank matrices, r x (in + out) parameters per adapted layer, so optimizer state almost disappears. QLoRA (Dettmers et al. 2023) also stores the frozen base in 4-bit NormalFloat with double-quantised scaling constants, about 4.127 bits per weight, which is how a 65B model was fine-tuned on a single 48 GB GPU in the paper.
Common use cases
Choosing a GPU to rent
See whether a job needs an A100 40 GB, an 80 GB card, an H200 or several GPUs before you book them.
Running models locally
Check whether a quantised model and the context you want fit a 24 GB gaming card or a Mac's unified memory.
Planning a fine-tune
Compare full fine-tuning, LoRA and QLoRA for the same model and sequence length.
Sizing a serving deployment
Work out how many concurrent long-context sequences the KV cache can hold on a given card.
Tips and limitations
- For mixture-of-experts models such as Mixtral 8x7B, every expert must be in memory even though only two run per token: 46.7B parameters to store, 12.9B to compute.
- Serving engines such as vLLM reserve most of the free memory for a paged KV cache up front, so nvidia-smi shows near-full usage regardless of load.
- An FP8 KV cache halves cache memory with a small quality cost on most models; it is often the cheapest way to fit a longer context.
- Activation estimates assume FlashAttention-style kernels that do not store the full attention matrix. Without them, long sequences need far more.
- macOS lets the GPU use only part of a Mac's unified memory by default, so leave a wider margin there than on a discrete card.
- Treat results as planning numbers, accurate to roughly 10 to 20 percent. Frameworks, kernels and fragmentation vary.
Frequently asked questions
How much GPU memory does a 7B or 8B model need?
About 15 GB for BF16 weights plus the KV cache, so around 17 GB at 8K context. At 4-bit the weights drop to about 5 GB, so it runs on an 8 GB card with a short context.
Why does long context use so much memory?
Each token stores keys and values in every layer. For Llama 3.1 8B that is 128 KB per token, so 128K tokens need 16 GB of cache on top of the weights, for each sequence served.
What does gradient checkpointing save?
It stores only each layer's input during the forward pass and recomputes the rest during backpropagation. Activation memory falls by more than ten times for about a third more compute.
Why is full fine-tuning so much more expensive than LoRA?
Full fine-tuning keeps gradients and Adam state for every weight, 16 bytes per parameter. LoRA trains well under one percent of the parameters, so that state shrinks to almost nothing and only the frozen weights and activations remain.
Does INT4 quantisation hurt quality?
Good 4-bit methods such as AWQ and GPTQ lose little on most benchmarks for models of 7B and up, but small models and maths or code tasks can suffer more. Test on your own evals.
Can I add up memory across several GPUs?
Yes, if the software splits the model: tensor or pipeline parallelism for serving, FSDP or DeepSpeed ZeRO-3 for training. Each split adds some communication buffers, so leave headroom.
Is there an in-depth guide to the LLM GPU Memory Calculator?
Yes. “LLM GPU Memory and Training Compute, Worked Out” covers the background, how it works, common pitfalls and worked examples. Read the full guide.
Related Tools You May Like
Free, in-browser tools that pair well with this one.