LLM Eval Scorer
Score model outputs against expected answers with exact match, token F1, ROUGE-L, regex, numeric and JSON checks, compare two models and export the results.
The interactive tool loads here when JavaScript runs.
About this toolWhat is the LLM Eval Scorer?
Paste rows of input, expected answer and model output, as JSONL, a JSON array, CSV or two tab-separated columns, and this scorer grades every row with eight metrics: exact match, token F1, ROUGE-L, contains-expected, regex match, numeric within tolerance, valid JSON, and JSON fields equal. Pick one metric and a threshold to decide pass or fail, filter to the failures, and download the per-row results as CSV.
Add a second output column and it compares model A with model B on every metric: mean scores side by side and wins, losses and ties per row. That makes it a quick regression check after a prompt change or a fine-tune. The rows are scored in your browser; nothing is sent anywhere.
- Exact match and token F1 with SQuAD normalisation
- ROUGE-L, contains, regex, numeric tolerance, valid JSON and JSON field checks
- JSONL, JSON array, CSV with headers or two tab-separated columns
- Model A versus B: mean scores and wins, losses and ties per metric
- Failure and disagreement filters and CSV export of every score
- Runs in the browser with a sample dataset to start from
Updated
How to use the LLM Eval Scorer
- Paste rows or open a file with expected and output fields; add output_b to compare two models. Load the sample to see the format.
- Choose the metric that decides pass or fail, and for F1, ROUGE-L or JSON fields set the minimum score.
- For numeric answers, set an absolute or percentage tolerance.
- Read the pass rate and the table of mean scores, and with two models the wins, losses and ties.
- Filter the rows to failures or disagreements, then download the results CSV for your tracker.
Worked examples
Partial credit with F1
Input
expected: "Canberra", output: "Canberra is the capital"
Output
Exact match 0, F1 0.5, contains 1
After normalisation the output has three words, one of which matches: precision 1/3, recall 1, F1 0.5.
Numeric tolerance
Input
expected: "3.11", output: "About 3.1 miles"
Output
Fails at 0 tolerance, passes at 1% relative tolerance
The difference is 0.01, which is within 1% of 3.11 (0.0311).
Typed JSON fields
Input
expected: {"name": "Maria", "age": 34}, output: {"name": "Maria", "age": "34"}Output
Valid JSON 1, JSON fields equal 0.5
The age is a string instead of a number, so one of two fields matches.
Background and insight
Exact match and token F1 come from the SQuAD reading-comprehension benchmark (Rajpurkar et al. 2016). Before comparing, both strings are lower-cased, punctuation and the articles a, an and the are removed, and whitespace is collapsed, so "The Eiffel Tower." matches "eiffel tower". F1 treats the answer as a bag of words: precision is the share of output words found in the reference, recall the share of reference words found in the output, and F1 their harmonic mean.
ROUGE-L (Lin 2004) scores the longest common subsequence of words, so it rewards the same words in the same order without requiring them to be adjacent; it was designed for summaries. This tool reports the F-measure with equal weight on precision and recall. All of these metrics measure overlap with one reference. A correct answer phrased differently scores low, and a wrong answer that repeats the reference words can score high, which is why open-ended tasks are usually also judged by people or by a model acting as a grader.
The structured checks suit tasks with one right answer. Numeric match takes the last number in each string, the usual convention for grade-school maths sets such as GSM8K, and passes it within an absolute or relative tolerance. Regex match treats the expected column as a pattern. JSON checks parse the output, tolerating a ```json fence, and compare the expected object's top-level fields exactly, so "34" and 34 differ.
Common use cases
Prompt regression tests
Re-run a fixed question set after changing a prompt and see which rows started failing.
Fine-tune versus base model
Put the base model in output_b and the fine-tune in output to measure the change per metric.
Extraction and tool output
Check that structured outputs parse and that every required field has the expected value.
Short-answer and maths sets
Grade QA and arithmetic answers with SQuAD normalisation and numeric tolerance.
Tips and limitations
- Pick the metric that matches the task: exact match or numeric for single answers, F1 for short spans, ROUGE-L for summaries, JSON fields for extraction.
- Overlap metrics punish correct paraphrases. Read a sample of failures before trusting a low score.
- With fewer than about 100 rows, a few points of difference between two models can be noise; look at wins and losses, not only the mean.
- Contains-expected is lenient. "No, Canberra is not the capital" passes a contains check for Canberra.
- Keep the eval set out of your fine-tuning data, or the scores measure memorisation.
- Regex patterns are matched case-insensitively anywhere in the output; add ^ and $ to require a whole-string match.
Frequently asked questions
What is the difference between exact match and F1?
Exact match gives 1 only when the normalised output equals the normalised reference. F1 gives partial credit for overlapping words, so a longer correct answer still scores above zero.
What does SQuAD normalisation remove?
It lower-cases the text, strips ASCII punctuation, removes the words a, an and the, and collapses whitespace. "The answer is 4." becomes "answer is 4".
Is ROUGE-L good for judging chat answers?
Only roughly. It measures word overlap in order with one reference, so it suits summaries and short rewrites. Open-ended answers need human review or a rubric-based model grader.
How are A and B compared?
For each metric, each row where both have a score counts as a win if A scores higher, a loss if B does, and a tie if equal. Means are shown alongside.
Which column names does it understand?
Expected can be expected, reference, answer, target, ideal or label; output can be output, prediction, response or completion; the second model is output_b, model_b or baseline; input can be input, prompt or question.
Why does a row show n/a for some metrics?
A metric is skipped when it cannot apply: regex when the expected text is not a valid pattern, numeric when expected has no number, and JSON fields when expected is not a JSON object.
Is there an in-depth guide to the LLM Eval Scorer?
Yes. “Fine-Tuning Datasets and LLM Evals: A Practical Guide” covers the background, how it works, common pitfalls and worked examples. Read the full guide.
Related Tools You May Like
Free, in-browser tools that pair well with this one.