Fine-Tuning Datasets and LLM Evals: A Practical Guide

SFT versus preference tuning, the JSONL formats trainers expect, what makes training data good, and how to hold out an eval set and read metrics like exact match, F1 and ROUGE-L.

November 1, 2026 · 12 min read

Fine-tuning a language model is mostly a data problem. The training code is a few lines with any hosted service or with open-source trainers such as TRL, Axolotl or LLaMA-Factory, and the defaults are sensible. What decides whether the result is better than the base model is what goes into the training file and how you measure the result afterwards. This article covers the two main kinds of post-training, the file formats each one uses, what makes a dataset good, and how to build a held-out eval and read its numbers without fooling yourself. The Fine-Tuning Dataset Validator checks and converts training files, and the LLM Eval Scorer grades model outputs, both in the browser.

SFT and preference tuning

Supervised fine-tuning (SFT) shows the model conversations and the assistant replies you want, and trains it to predict those replies token by token. The loss is the usual next-token cross-entropy, computed only on the assistant turns. SFT is the right tool when you can write down good answers: a support bot's tone and policies, a fixed output format, a classification label, extraction into JSON, or a house style. It teaches format and behaviour well. It is a poor way to add new facts, which are better supplied at run time through retrieval, because a model fine-tuned on a few hundred examples of a fact rarely recalls it reliably.

Preference tuning shows the model two answers to the same prompt and tells it which one is better. The classic method, reinforcement learning from human feedback (RLHF), trains a separate reward model on those comparisons and then optimises the language model against it with PPO. Direct Preference Optimization (DPO), from Rafailov et al. (2023), skips the reward model: it adjusts the model so the probability of the preferred answer rises relative to the rejected one, compared with a frozen reference copy. A parameter called beta controls how far the model may move from that reference. DPO is simpler to run and has become the default way to apply preference data outside the largest labs.

In practice the two are steps, not alternatives. OpenAI's own documentation recommends running SFT on the preferred responses first and then DPO on the pairs. SFT gets the model into the right format and register; preference tuning then sharpens choices that are hard to express as one correct answer, such as which of two helpful replies is clearer, shorter or safer.

The file formats trainers expect

Almost every trainer reads JSON Lines: one JSON object per line, no commas between lines, no surrounding array. The object shape differs.

One change to know about: OpenAI is winding down its own fine-tuning platform. Its documentation (checked November 2026) says the platform no longer accepts new users and that eligible existing customers can create training jobs until January 6, 2027, unless the base model shuts down sooner. The chat format itself is not going away, because it is what most chat-template trainers, such as TRL, Axolotl and LLaMA-Factory, read when you fine-tune an open model.

A few rules in the OpenAI chat format cause most upload failures. The conversation must end with an assistant message, because that is what is trained. Roles are limited to system, user, assistant and tool. An assistant message may carry "weight": 0 to keep it as context without training on it, or "weight": 1; no other role may have a weight. When an assistant calls a tool, its content can be null and it holds a tool_calls array, and each following tool message needs the matching tool_call_id. Any key the format does not define, such as a stray "tone" field left over from labelling, makes the row invalid.

The preference format adds its own constraints. OpenAI supports only single-turn pairs: the prompt goes in input.messages and ends on the user, and each of preferred_output and non_preferred_output holds exactly one assistant message. A pair where both answers are identical teaches nothing.

Alpaca can only hold one instruction and one answer, so converting a multi-turn chat to Alpaca loses rows; ShareGPT keeps every turn. Converting from CSV is common when examples were written in a spreadsheet: a prompt column becomes the user message and a completion column the assistant reply.

What makes training data good

OpenAI asks for at least 10 examples and reports clear improvements from 50 to 100 well-chosen ones; that matches the general experience that a few hundred consistent, high-quality rows beat thousands of noisy ones for behaviour and format. The checks that matter most:

• Consistency. If two rows answer the same kind of question in different styles, the model learns the average. Decide the format, length and tone first, and make every row follow it.
• One system prompt. Use the same system prompt in every row and send it again at inference. A model trained with a prompt behaves differently without it.
• No duplicates. An exact duplicate is counted twice in the loss, which overweights it. Near-duplicates (the same question reworded) are harder to spot and worth a separate pass.
• Balanced lengths. A handful of rows that are ten times longer than the median contribute most of the tokens, and so most of the gradient. Split or trim them unless they are the point.
• Coverage of the hard cases. The examples that matter are the ones the base model gets wrong: refusals you want, edge cases, the formats it breaks.
• Cost awareness. Hosted fine-tuning is billed per trained token, which is tokens in the file times epochs. A 1,200-row file averaging 310 tokens is about 372,000 tokens per epoch, or 1.1 million over three epochs.

Run a validator before every upload. It is faster to fix a line-numbered list of problems locally than to wait for a remote file check, and the token counts let you budget the job.

Hold out an eval set first

Before training, set aside a test set the model never sees: typically 50 to 300 prompts that represent the real task, each with an expected answer or a clear grading rule. Keep it fixed across runs so numbers are comparable, and keep it out of the training file, or the score measures memorisation. A validation split taken from the training data (10 percent with a fixed shuffle seed is common) is useful for watching the loss during training, but it is drawn from the same distribution you wrote, so it flatters the model. The real eval should be written separately, ideally from real user requests.

Score three things on that set: the base model with your best prompt, the fine-tuned model, and, when you change anything later, the previous version. A fine-tune that does not beat a well-prompted base model is not worth deploying.

Reading eval metrics and their limits

Reference-based metrics compare each output with an expected answer.

• Exact match is 1 when the normalised strings are equal. SQuAD normalisation (Rajpurkar et al. 2016) lower-cases, removes punctuation and the articles a, an and the, and collapses whitespace, so "The Eiffel Tower." equals "eiffel tower".
• Token F1 gives partial credit. "Canberra is the capital" against "Canberra" normalises to three words with one match: precision 1/3, recall 1, F1 0.5.
• ROUGE-L (Lin 2004) scores the longest common subsequence of words, rewarding the same words in the same order. It was built for summaries.
• Contains, regex and numeric checks suit tasks with one right answer. For maths, take the last number in the output and allow a tolerance: 3.1 against 3.11 fails at zero tolerance and passes at 1 percent.
• JSON checks confirm the output parses and that each expected field has exactly the expected value, which catches "34" written where 34 was required.

Each has blind spots. Overlap metrics punish a correct answer phrased differently and can reward a wrong answer that repeats the right words. Contains-expected passes "No, Canberra is not the capital". For open-ended tasks, add a rubric graded by people or by a stronger model, and always read a sample of failures before trusting a number.

Small sets are noisy. With 100 questions, one model scoring 72 percent and another 68 percent is a four-question difference, within what a different random sample could produce. Looking at wins, losses and ties per question is more informative than the two means: if model B wins 15 rows and loses 11, the change is real but modest, and the 11 losses show you what regressed.

A workflow that holds up

1. Write the eval set and the grading rule before writing training data.
2. Score the base model with a strong prompt to get the baseline.
3. Build the training file, validate it, fix every error and look at every warning.
4. Shuffle with a fixed seed, split off a validation set, and train.
5. Score the fine-tuned model on the same eval, compare row by row with the baseline, and read the regressions.
6. If the gaps are about which of two good answers is better, collect preference pairs and add a DPO step.

Keep the files, the seed and the scores together, so the next round can be compared with this one. Before you train, the LLM GPU Memory Calculator shows whether a LoRA or full fine-tune fits your GPU, and the LLM Token Counter gives exact counts for rows near a length limit.

Open the Fine-Tuning Dataset Validator