LLM Training Compute Calculator

Estimate training FLOPs with C = 6ND, find the Chinchilla compute-optimal model size, and turn it into GPU-hours, days, cost and energy on A100, H100 or B200.

The interactive tool loads here when JavaScript runs.

About this tool

What is the LLM Training Compute Calculator?

Enter a model size and a number of training tokens and this calculator gives the training compute with the standard rule C ≈ 6ND. Switch the unknown to solve for tokens or parameters from a FLOP budget, or ask for the Chinchilla compute-optimal split, about 20 tokens per parameter. Sizes accept shorthand such as 70B, 1.4T or 5.9e23.

Then choose a GPU, a model FLOPs utilisation (MFU), the number of GPUs, a price per GPU-hour and a data-centre PUE, and it converts the FLOPs into GPU-hours, wall-clock days, dollars and megawatt-hours. It also shows inference compute, about 2N FLOPs per token. Every formula is printed under the results.

  • Solve for compute, tokens or parameters with C = 6ND
  • Chinchilla compute-optimal split at 20 tokens per parameter
  • Dense BF16 presets for A100, H100, H200, B200 and MI300X, or custom
  • GPU-hours, wall-clock days, cost and energy with MFU and PUE
  • Inference FLOPs per token and a compute-bound throughput ceiling
  • Shorthand input (70B, 1.4T, 5.9e23), formulas and sources on the page
Read the guide: LLM GPU Memory and Training Compute, Worked Out

Updated

How to use the LLM Training Compute Calculator

  1. Choose what to solve for: compute from parameters and tokens, tokens or parameters from a budget, or the Chinchilla optimum from a budget.
  2. Type sizes as 8B, 70B, 1.4T or scientific notation such as 3.8e25.
  3. Pick a GPU preset or enter a custom dense BF16 peak and board power.
  4. Set MFU, the number of GPUs, the price per GPU-hour and the PUE of the data centre.
  5. Read GPU-hours, days, cost and energy, plus the tokens-per-parameter ratio and inference FLOPs per token.

Worked examples

Chinchilla

Input

N = 70B, D = 1.4T

Output

C = 6 x 70 x 10⁹ x 1.4 x 10¹² = 5.88 x 10²³ FLOPs, exactly 20 tokens per parameter

On 1,024 H100s at 40% MFU this is about 413,000 GPU-hours, roughly 17 days.

Llama 3.1 405B

Input

N = 405B, D = 15.6T

Output

C ≈ 3.8 x 10²⁵ FLOPs

Meta reports 30.84 million H100 GPU-hours for this model, which implies about 35% of dense peak by the 6ND count.

Budget first

Input

C = 1 x 10²² FLOPs, Chinchilla mode

Output

N ≈ 9.1B parameters, D ≈ 183B tokens

N = √(C / 120) because C = 6 x N x 20N.

Background and insight

A dense transformer with N parameters does about 2N floating-point operations per token in the forward pass (one multiply and one add per weight) and about twice that in the backward pass, so training costs about 6N per token. Kaplan et al. (2020) used C ≈ 6ND in their scaling-laws paper, and it is the usual way to quote training compute. It ignores the attention score computation, which adds a few percent at typical sequence lengths and more for very long ones.

Hoffmann et al. (2022) trained over 400 models and found that for a fixed budget, parameters and tokens should grow in equal proportion, at roughly 20 tokens per parameter. Their 70B Chinchilla model, trained on 1.4 trillion tokens with about 5.9 x 10²³ FLOPs, beat the 280B Gopher trained with the same compute. Many recent models train far past 20 tokens per parameter on purpose: Llama 3.1 8B saw about 15 trillion tokens, nearly 1,900 per parameter, because a smaller model trained longer is cheaper to run once deployed.

Converting FLOPs to time needs the GPU's real throughput. Datasheets quote peak tensor-core numbers, and NVIDIA often headlines the figure with 2:4 structured sparsity, twice the dense rate that training actually uses: the H100 SXM is 989 dense BF16 TFLOPS, listed as 1,979 with sparsity. Large runs typically reach 30 to 50 percent of the dense peak as model FLOPs utilisation, after communication, memory stalls and pipeline bubbles.

Common use cases

  • Scoping a pre-training run

    Turn a model size and token count into GPU-hours and a budget before asking for a cluster.

  • Choosing model size for a budget

    Use the Chinchilla mode to see the compute-optimal size, then decide whether to train smaller for longer.

  • Comparing GPU generations

    See how A100, H100, H200, B200 and MI300X change days and cost for the same job.

  • Reporting energy and emissions

    Estimate megawatt-hours from board power and PUE as a starting point for a carbon estimate.

Tips and limitations

  • For mixture-of-experts models, use the active parameters per token, not the total, in the FLOP count.
  • Repeated epochs count: D is the number of tokens processed, not the size of the unique dataset.
  • The energy figure uses GPU board power at full load, times PUE. CPUs, networking and storage add more, while GPUs rarely draw full TDP the whole time.
  • Budget for failures and experiments. Restarts, evaluation and ablation runs often add as much again to the headline run.
  • Fine-tuning compute follows the same 6ND rule with the fine-tuning tokens; LoRA saves memory more than FLOPs, because the backward pass still runs through the frozen layers.
  • MFU above 50% on large clusters is rare. If a plan needs 60% or more, it is probably optimistic.

Frequently asked questions

Where does the 6 in 6ND come from?

Each parameter takes part in one multiply and one add per token in the forward pass, 2N FLOPs, and the backward pass costs about twice that, 4N, for gradients with respect to both activations and weights.

What is MFU?

Model FLOPs utilisation: the useful model FLOPs per second divided by the hardware's dense peak. It measures how well a training run keeps the GPUs busy; 30 to 50% is typical for large runs.

Is 20 tokens per parameter still the right target?

It is the compute-optimal point for training loss at a fixed budget. If the model will serve a lot of traffic, training a smaller model on many more tokens costs more up front but less to run.

Why do H100 datasheets show 1,979 TFLOPS?

That figure assumes 2:4 structured sparsity. Dense BF16 matrix maths, which training uses, peaks at 989 TFLOPS on the H100 SXM, which is what the presets use.

How many FLOPs does it take to generate a token?

About 2N for the forward pass, so 16 GFLOPs for an 8B model. In practice single-stream generation is limited by memory bandwidth rather than FLOPs.

How accurate are these estimates?

The FLOP count is within a few percent for standard dense transformers. Time and cost depend mostly on the MFU you assume, so try a range such as 30 to 50%.

Is there an in-depth guide to the LLM Training Compute Calculator?

Yes. “LLM GPU Memory and Training Compute, Worked Out” covers the background, how it works, common pitfalls and worked examples. Read the full guide.

Related Tools You May Like

Free, in-browser tools that pair well with this one.