LLM GPU Memory and Training Compute, Worked Out

How much VRAM a model needs to serve, fine-tune or train with LoRA and QLoRA, how the KV cache grows with context, and how 6ND and Chinchilla turn into GPU-hours and cost.

November 1, 2026 · 13 min read

Two questions come up before almost any language-model project that runs on your own hardware or rented GPUs. Will the model fit in GPU memory, and how long will the training take? Both have good back-of-the-envelope answers that need only the model's shape from its config.json and a handful of well-established rules. This article works through them with real numbers: memory for serving, for full fine-tuning and for LoRA and QLoRA, the KV cache, and then pre-training compute with 6ND and the Chinchilla rule. The LLM GPU Memory Calculator and the LLM Training Compute Calculator do the same sums interactively.

All memory figures here use GB to mean 2³⁰ bytes, the unit GPU memory is actually built in.

Memory for inference: weights first

The weights take parameters times bytes per weight. A Llama 3.1 8B model has 8.03 billion parameters, so:

The 8-bit and 4-bit rows are slightly more than half and a quarter of BF16 because quantised checkpoints usually keep the token embedding and the output head in 16-bit. For Llama 3.1 that is 2 x 128,256 x 4,096, about 1.05 billion parameters, that stay at 2 bytes. Models with very large vocabularies, such as Gemma 2 at 256,000 tokens, carry an even bigger unquantised share.

Mixture-of-experts models need memory for every expert even though only some run per token. Mixtral 8x7B stores 46.7 billion parameters, 87 GB in BF16, while computing with only 12.9 billion per token.

The KV cache

While generating, the model keeps a key vector and a value vector for every previous token in every layer, so it does not recompute them. The size is:

KV cache = 2 x layers x KV heads x head dimension x bytes per value x tokens x batch

The number that matters is the KV heads, from num_key_value_heads in config.json, not the attention heads. Llama 3.1 uses grouped-query attention with 8 KV heads shared by 32 query heads, which cuts the cache by four. For Llama 3.1 8B in BF16: 2 x 32 x 8 x 128 x 2 = 131,072 bytes per token, exactly 128 KiB. An 8,192-token context takes 1 GB, and the full 131,072-token context takes 16 GB, more than the weights.

The cache scales with every sequence served at once, so a server handling 32 conversations at 8K each needs 32 GB of cache for an 8B model. Llama 3.1 70B has 80 layers but still 8 KV heads, so it needs 320 KiB per token, 2.5 GB at 8K. Phi-3 mini, without grouped-query attention, has 32 KV heads of 96 dimensions over 32 layers: 384 KiB per token, three times Llama 3.1 8B's despite being half the size. An FP8 cache halves all of these.

Serving engines also need room for activations, CUDA context and allocator fragmentation. Ten to twenty percent on top is a reasonable planning allowance. Putting it together, Llama 3.1 70B in BF16 with an 8K context needs about 131.5 + 2.5 + 13 = 147 GB: two 80 GB H100s or one 141 GB H200. In INT4 it is about 42 GB, which fits a single 48 GB L40S.

Memory for training: why full fine-tuning is expensive

Training keeps far more than the weights. With mixed-precision Adam, the standard recipe, each parameter needs:

• 2 bytes for the 16-bit weights used in the forward and backward pass,
• 2 bytes for the 16-bit gradients,
• 12 bytes for the optimizer: an FP32 master copy of the weights (4) and Adam's two moment estimates (4 + 4).

That is 16 bytes per parameter, the figure Rajbhandari et al. use in the ZeRO paper (2019). For an 8B model it is 120 GB before any activations, which is why full fine-tuning of even a small model needs several GPUs with the optimizer state sharded across them (ZeRO-3 or PyTorch FSDP).

Activations are the intermediate values saved for backpropagation. Korthikanti et al. (2022) estimate about 34 bytes x sequence x batch x hidden size per transformer layer in 16-bit when the attention matrix is not stored, which is what FlashAttention-style kernels achieve. For Llama 3.1 8B at 2,048 tokens and batch 1 that is about 0.27 GB per layer and 8.5 GB over 32 layers. Gradient checkpointing keeps only each layer's input (2 x sequence x batch x hidden bytes) and recomputes the rest during the backward pass, which cuts this more than tenfold for roughly a third more compute. The output logits, sequence x batch x vocabulary in FP32, add another 1 GB here because of the 128K vocabulary.

LoRA and QLoRA

LoRA (Hu et al. 2021) freezes the base model and trains a pair of small matrices beside each chosen weight matrix: one of size in x r and one of size r x out, so r x (in + out) new parameters. With rank 16 on the four attention projections of Llama 3.1 8B, that is 425,984 parameters per layer and 13.6 million in total; adding the three MLP projections brings it to 41.9 million, about half a percent of the model. Those adapters need gradients and Adam state (about 16 bytes each in FP32), but at 42 million parameters that is only 0.6 GB. The frozen base still sits in BF16, 15 GB, and the activations still exist because backpropagation passes through the frozen layers. With checkpointing at 2,048 tokens the total is about 19 GB: a 24 GB card is enough.

QLoRA (Dettmers et al. 2023) also stores the frozen base in 4-bit NormalFloat (NF4). Each block of 64 weights has a scaling constant, and those constants are themselves quantised to 8 bits ("double quantisation"), bringing the cost to about 4.127 bits per weight. The paper fine-tuned a 65B model on a single 48 GB GPU this way. For Llama 3.1 8B the base drops to about 5.3 GB and the whole job to about 8.5 GB. For Llama 3.1 70B with rank 16 on the attention projections and 1,024-token sequences, the total is about 44 GB, which fits a 48 GB card; longer sequences or adapters on every layer push it past that.

Pre-training compute: 6ND

For training from scratch the limit is usually compute, not memory. A dense transformer with N parameters does about 2N floating-point operations per token in the forward pass, one multiply and one add per weight, and about 4N in the backward pass. So training on D tokens costs:

C ≈ 6 x N x D FLOPs

Kaplan et al. (2020) used this in their scaling-laws paper and it is still the standard. It leaves out the attention score computation, which adds a few percent at ordinary sequence lengths.

Chinchilla is the classic example: 70 billion parameters on 1.4 trillion tokens is 6 x 70 x 10⁹ x 1.4 x 10¹² = 5.88 x 10²³ FLOPs. Llama 3.1 405B on about 15.6 trillion tokens is 3.8 x 10²⁵ FLOPs.

Chinchilla: how to split a budget

Hoffmann et al. (2022) trained more than 400 models and found that for a fixed compute budget, the loss-minimising model grows in step with its data, at roughly 20 tokens per parameter. Substituting D = 20N into C = 6ND gives C = 120N², so N = √(C / 120). A budget of 10²² FLOPs points to about 9.1 billion parameters on 183 billion tokens.

Chinchilla itself made the point: at the same compute as the 280B Gopher, the 70B model trained on more data performed better. Many current models deliberately train well past 20 tokens per parameter. Llama 3.1 8B saw about 15 trillion tokens, nearly 1,900 per parameter, because training cost is paid once while a smaller model is cheaper every time it answers a request.

From FLOPs to GPU-hours, days, dollars and energy

To turn FLOPs into time, divide by what the GPUs actually deliver. Use the dense BF16 tensor peak, not the headline sparse figure: the H100 SXM is 989 TFLOPS dense (1,979 with 2:4 sparsity), the A100 312, the H200 the same as the H100, the B200 about 2,250 and AMD's MI300X 1,307. Then multiply by model FLOPs utilisation (MFU), the share of that peak a real run achieves after communication and memory stalls, typically 30 to 50 percent.

A worked example: a 7B model on 2 trillion tokens is 6 x 7 x 10⁹ x 2 x 10¹² = 8.4 x 10²² FLOPs. An H100 at 40 percent MFU does 396 TFLOPS of useful work, so the run needs 8.4 x 10²² / 3.96 x 10¹⁴ = 2.1 x 10⁸ GPU-seconds, about 59,000 GPU-hours. On 64 GPUs that is 38 days. At $2.50 per GPU-hour it is about $147,000, and at 700 W per GPU with a PUE of 1.2, about 50 MWh of electricity.

Check this against a published run. Meta reports 30.84 million H100 GPU-hours for Llama 3.1 405B. Dividing 3.8 x 10²⁵ FLOPs by that time gives about 340 TFLOPS per GPU, roughly 35 percent of dense peak by the 6ND count, squarely in the usual range.

Real projects cost more than the headline run: failed and restarted jobs, evaluation, ablations and data processing often add as much again. And inference has its own simple rule, about 2N FLOPs per generated token, 16 GFLOPs for an 8B model, although generating one sequence at a time is limited by memory bandwidth long before FLOPs.

A checklist before you book GPUs

• Read num_hidden_layers, hidden_size, num_key_value_heads, head_dim and vocab_size from the model's config.json rather than guessing from the name.
• For serving, add the KV cache for your longest context times the number of concurrent sequences, then 10 to 20 percent.
• For fine-tuning, start with QLoRA or LoRA and gradient checkpointing; move to full fine-tuning only when adapters measurably fall short.
• For pre-training, compute 6ND, compare the tokens-per-parameter ratio with 20, and budget at 30 to 40 percent MFU unless you have measured better.

The Fine-Tuning Dataset Validator gives the token count of a training file, which is the D to use for a fine-tuning run, and the guide to fine-tuning datasets and evals covers what to put in it.

Open the LLM GPU Memory Calculator