Fine-Tuning Dataset Validator
Check an OpenAI chat, DPO, Alpaca or ShareGPT JSONL file line by line, count tokens, then convert formats and split train and validation sets in your browser.
The interactive tool loads here when JavaScript runs.
About this toolWhat is the Fine-Tuning Dataset Validator?
Paste or drop a fine-tuning dataset and this validator checks every line before you spend money on a training job. It recognises the OpenAI chat format with a messages array, OpenAI preference (DPO) pairs, Alpaca instruction, input and output rows, ShareGPT conversations, and prompt and completion CSV files, and it reports each problem with the line number it is on.
The checks cover what makes uploads fail or training go wrong: lines that are not valid JSON, unknown keys, roles that do not exist, empty messages, a conversation that ends on a user turn, weights on the wrong role, tool messages without a tool call id, duplicates, and examples far longer than the rest. Then you can convert the clean rows to another format, add a system prompt to every example, shuffle with a fixed seed and download train.jsonl and validation.jsonl. The file is read by your browser and never uploaded.
- Auto-detects OpenAI chat, OpenAI DPO, Alpaca, ShareGPT and prompt/completion
- Line-numbered errors for JSON, keys, roles, empty content, weights and tool messages
- Warnings for duplicates, very long rows, uneven lengths and missing system prompts
- Token estimates, length histogram, role counts and trained tokens by epoch
- Convert formats, add a system prompt, seeded shuffle and train/validation split
- Checks large files in chunks so the tab stays responsive; nothing is uploaded
Updated
How to use the Fine-Tuning Dataset Validator
- Paste JSONL, a JSON array or a CSV with prompt and completion columns, open a file, or load the sample that contains typical mistakes.
- Leave the format on auto-detect or force one, and set the token length that should raise a warning, normally your model's training context.
- Read the summary: valid examples, rows with errors, estimated tokens per example and in total, and trained tokens for your number of epochs.
- Work through the problems list, which gives the line number and the exact field at fault, fix the source file and load it again.
- Pick an output format, optionally add or replace the system prompt, set the validation share and seed, and download train.jsonl and validation.jsonl.
Worked examples
Conversation that ends on the user
Input
{"messages":[{"role":"system",...},{"role":"user","content":"Opening hours?"},{"role":"assistant","content":"9 to 6."},{"role":"user","content":"And Sundays?"}]}Output
line 6: Last message is "user"; it should be the assistant reply.
The trainer has nothing to learn from the final turn. Add the assistant answer or delete the trailing question.
CSV to chat JSONL
Input
prompt,completion "How do I reset my password?","Open Settings, then Security, then Reset password."
Output
{"messages":[{"role":"user","content":"How do I reset my password?"},{"role":"assistant","content":"Open Settings, then Security, then Reset password."}]}Columns named input/output, question/answer or instruction/response work too, and a system column becomes a system message.
Budgeting a job
Input
1,200 valid examples, about 310 tokens each, 3 epochs
Output
About 372,000 tokens per epoch and 1.1 million trained tokens
Hosted fine-tuning is usually billed per trained token, so multiply by the provider's training price to estimate the cost.
Background and insight
Supervised fine-tuning (SFT) teaches a model by example: each row shows a conversation and the assistant reply you want. OpenAI's chat format stores that as a JSON object per line with a messages array of system, user, assistant and tool messages; assistant messages may carry a weight of 0 or 1 to exclude or include them in the loss, and assistant turns can hold tool_calls with content set to null. The model learns from the assistant turns, which is why a row that ends on a user message is useless and is rejected by most trainers.
Preference tuning with Direct Preference Optimization (DPO, Rafailov et al. 2023) uses pairs instead of single answers. OpenAI's format puts the prompt in input.messages and one assistant message each in preferred_output and non_preferred_output; its documentation limits DPO to single-turn conversations and recommends running SFT on the preferred answers first. Open-source trainers such as Axolotl, LLaMA-Factory and TRL accept Alpaca rows (instruction, input, output, from the Stanford Alpaca project) and ShareGPT rows (conversations with from set to system, human or gpt), so converting between these shapes is a routine chore.
Token counts here are estimates at about four characters per token plus three tokens of framing per message, which is close to GPT-style tokenizers on English prose. Code, non-English text and other model families tokenise differently, so treat the totals as a budget check and use the exact tokenizer of your target model when a limit is tight.
Common use cases
Before a hosted fine-tuning upload
Catch format errors locally instead of waiting for the provider's file check to reject the upload.
Moving between training stacks
Convert an Alpaca or ShareGPT dataset to OpenAI chat JSONL, or the other way round, for Axolotl, LLaMA-Factory or TRL.
Cleaning scraped or labelled data
Find empty answers, duplicated rows and outliers in length before they skew training.
Holding out an eval set
Make a reproducible train and validation split with a seed so later runs compare like with like.
Tips and limitations
- OpenAI is winding down its fine-tuning platform: its docs (checked November 2026) say new users cannot sign up and existing customers can start jobs until January 6, 2027. The chat format stays useful for open-model trainers such as TRL and Axolotl.
- OpenAI requires at least 10 examples and its documentation suggests starting with 50 to 100 good ones. Quality and consistency matter more than volume.
- Use one system prompt for every row, and send the same one at inference time; mixing prompts teaches the model to depend on wording you will not reproduce.
- Duplicates are counted twice in the loss. Exact copies are flagged here; near-duplicates with small wording changes are not, so dedupe paraphrases separately.
- A handful of very long rows can dominate training. Split them, trim them, or raise the warning limit only if your trainer packs sequences sensibly.
- Preference data cannot be made from plain chat rows: it needs a rejected answer for each prompt, so the DPO output is only offered for DPO input.
- Token counts are approximate. Use the LLM Token Counter or your model's tokenizer to check rows that sit near the limit.
Frequently asked questions
What format does OpenAI fine-tuning expect?
JSONL with one object per line, each holding a messages array of role and content pairs, ending on an assistant message. Tool definitions can go in a tools field and assistant turns can contain tool_calls.
What does the weight field on a message do?
On assistant messages, weight 0 excludes that turn from training and weight 1 includes it. It lets a multi-turn example keep context without training on weak earlier answers. It is not allowed on other roles.
How is the DPO preference format different from chat?
The prompt goes in input.messages, and the good and bad replies go in preferred_output and non_preferred_output, each one assistant message. OpenAI supports single-turn pairs only.
Can I convert a multi-turn chat to Alpaca?
No. Alpaca holds one instruction and one output, so only single-turn rows convert; the tool tells you how many rows it had to leave out. ShareGPT keeps every turn.
How big a file can I check?
Up to 150 MB. Files are processed in chunks of 2,000 lines with a progress bar, and the first 1,000 problems are listed. Split larger datasets and check each part.
Does my training data leave my computer?
No. The file is read with the browser's File API and checked by script in this page. There is no upload and no server call.
How many examples should be in the validation split?
Ten percent is a common default, with at least a few dozen rows so the validation loss is not noise. Keep the split fixed with the same seed between runs.
Is there an in-depth guide to the Fine-Tuning Dataset Validator?
Yes. “Fine-Tuning Datasets and LLM Evals: A Practical Guide” covers the background, how it works, common pitfalls and worked examples. Read the full guide.
Related Tools You May Like
Free, in-browser tools that pair well with this one.