> ## Documentation Index
> Fetch the complete documentation index at: https://docs.baseten.co/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluate a model

> Define what a good answer is, freeze a held-out set, score with code before an LLM judge, and compare models or checkpoints on the same task.

An evaluation measures how well a model does your task, whether it's an open model you could use as is or a checkpoint you trained. It decides whether a checkpoint is better than the model it would replace. Build it before you train: the same definition of a good answer selects training data, tells you whether a run is learning, and decides whether a checkpoint ships.

This guide uses one running example: a tool-calling assistant that should call a tool when one can serve the request, with the right arguments, and answer directly when none can. The [SFT example](https://docs.baseten.co/examples/sft-tool-calling.md) trains that assistant and reports these scores.

## Write down what a good answer is

Turn quality into rules a reviewer can apply to one output. Write each rule as a trigger and a required behavior, not an adjective:

| Avoid | Prefer |
| - | - |
| Responses should be accurate. | If the reply states a dollar amount, it matches the source record exactly. |
| Use tools well. | If no offered tool can serve the request, answer directly without calling one. |
| Don't over-flag. | Patient-reported home readings in the history are acceptable; don't flag them as lab results. |

Include output-format requirements in these rules. For JSON, check that the reply parses, required fields have the right types, and enum values are allowed. For tool calls, check the function name and argument schema. For prose, check requirements such as sections or length. Score format and task correctness separately: a reply can follow the schema and still give the wrong answer.

Use [structured outputs](https://docs.baseten.co/inference/structured-outputs.md) when you need to constrain JSON generation, then evaluate the content against your task's rules.

Collect rules from material that already encodes them:

* **System prompts and docs:** every must or must-not instruction is a rule someone wrote down.
* **Templates and schemas:** required sections, field formats, and enum values.
* **Production outputs:** choices the outputs make consistently. These describe what production does, which isn't necessarily correct.
* **Escalations and corrections:** complaints and "don't do this again" notes. A rule broken once in an escalation can outrank a pattern seen in hundreds of rows.

Where production outputs disagree on the same trigger, keep both versions of the rule and decide between them with the people who own the task. Then record which failure is unacceptable and which error is worse than another. Those asymmetries decide how you weigh the scores later.

Find out where your reference outputs came from. If they're production outputs that nobody verified, an evaluation that compares against them rewards matching production, not being correct. Score those tasks against the rules instead of the reference, or have a person verify a slice of references to use as ground truth.

## Freeze a held-out set

Hold back examples that never enter training data or prompt examples, and score every model on that same set. Split with a fixed seed so the set doesn't change between runs, and split by group so related rows stay together. A group is whatever makes rows near-duplicates: the same customer, document, or tool set. If one group lands on both sides, the evaluation measures recall of training data instead of the task.

Store the evaluation records in [JSON Lines (JSONL)](https://jsonlines.org/), a UTF-8 text format with one complete JSON value per line. This script expects one JSON object per line. Keep the object on one line, without a surrounding array or commas between records. Don't insert blank lines. Represent line breaks inside strings as `\n`.

**group\_key** `string` *required*

Identifier for the customer, document, or tool set that related records share. Records with the same value stay in the same split. The script preserves every other field in the record.

For example, save the following as `data.jsonl`. The two records from `customer-001` must stay together. `prompt` and `reference` are example fields for an evaluation record.

```jsonl data.jsonl wrap theme={"system"}
{"group_key": "customer-001", "prompt": "What is the invoice total?", "reference": "$120.00"}
{"group_key": "customer-001", "prompt": "What amount is due?", "reference": "$120.00"}
{"group_key": "customer-002", "prompt": "When does invoice 42 fall due?", "reference": "June 15"}
{"group_key": "customer-003", "prompt": "Which currency is the invoice in?", "reference": "USD"}
```

JSONL describes how records are stored. It doesn't define the training schema. For SFT chat data, see [Format conversations as chat JSONL](https://docs.baseten.co/loops/sft-data.md#format-conversations-as-chat-jsonl).

Save the following as `split.py`. It reads `data.jsonl`, assigns about 10% of the groups to the held-out set with a fixed seed, and writes `train.jsonl` and `holdout.jsonl`. Groups can have different sizes, so the fraction of held-out rows can differ from 10%.

```python split.py theme={"system"}
import json
import random

SEED = 20261101
HOLDOUT_FRACTION = 0.1

with open("data.jsonl", encoding="utf-8") as source:
    rows = [json.loads(line) for line in source]
groups = sorted({row["group_key"] for row in rows})
if len(groups) < 2:
    raise ValueError("Use at least two groups so both splits contain records")
random.Random(SEED).shuffle(groups)
held_out = set(groups[: max(1, int(len(groups) * HOLDOUT_FRACTION))])

with (
    open("train.jsonl", "w", encoding="utf-8") as train,
    open("holdout.jsonl", "w", encoding="utf-8") as holdout,
):
    for row in rows:
        (holdout if row["group_key"] in held_out else train).write(
            json.dumps(row) + "\n"
        )

held_out_rows = sum(row["group_key"] in held_out for row in rows)
print(f"train.jsonl: {len(rows) - held_out_rows} rows")
print(f"holdout.jsonl: {held_out_rows} rows")
```

Run the script from the directory containing `data.jsonl`:

```bash theme={"system"}
python3 split.py
```

For the sample records above, the script prints:

```text Output theme={"system"}
train.jsonl: 3 rows
holdout.jsonl: 1 rows
```

Record the seed and a hash of `holdout.jsonl` with your results. A score is comparable to another only when both came from the same file. While you design the evaluation, iterate on rows from the training side. Score the held-out set when you compare finished candidates, so the design isn't tuned to the rows that later decide the release.

## Score with code first

Use code for any rule a person wouldn't need to reason about: valid JSON, a required field, a length limit, an exact label, the tool a model called. Code is free to run, gives the same answer every time, and can't be argued with.

For the tool-calling assistant, four checks compare the model's first reply with the reference reply:

* `decision`: the model calls a tool exactly when the reference does.
* `tools`: it calls the same tools.
* `arguments`: it calls the same tools with the same arguments.
* `well_formed`: the reply finished and every tool call parsed.

```python checks.py theme={"system"}
import json


def tool_calls(message):
    """Return sorted (name, arguments) pairs, or None if any call's arguments don't parse."""
    calls = []
    for call in message.get("tool_calls") or []:
        arguments = call["function"]["arguments"]
        # API replies send JSON text; datasets often store objects.
        if isinstance(arguments, str):
            try:
                arguments = json.loads(arguments)
            except json.JSONDecodeError:
                return None
        calls.append((call["function"]["name"], json.dumps(arguments, sort_keys=True)))
    return sorted(calls)


def score(reply, finish_reason, reference):
    got, want = tool_calls(reply), tool_calls(reference)
    if want is None:
        raise ValueError(
            "the reference's tool call arguments don't parse; fix the held-out row"
        )
    return {
        "decision": got is not None and bool(got) == bool(want),
        "tools": got is not None and [n for n, _ in got] == [n for n, _ in want],
        "arguments": got == want,
        "well_formed": got is not None and finish_reason in ("stop", "tool_calls"),
    }
```

Report each check separately. A single combined score hides which direction a change moved: a model that stops calling tools for every request raises `decision` on requests no tool can serve and lowers it everywhere else.

## Judge what code can't decide

Use an LLM judge only for rules that need reading and reasoning, such as whether a direct answer addresses the request. Design each judged rule as its own axis:

* **One prompt per axis.** A prompt that scores several rules at once lets one obvious failure bleed into unrelated scores.
* **Pass or fail by default.** A binary verdict varies less between repeated judgments than a 1 to 10 scale. If you use a scale, set the passing threshold yourself rather than asking the judge whether an output passes.
* **Reasoning before the verdict.** Ask for the judge's reasoning and quoted evidence first, then the verdict, so the verdict follows from the evidence.
* **As few axes as the rules need.** Two axes that never disagree on the same rows measure one thing. Each extra axis is another judge call per row.

Choose the judge model deliberately:

* **Data boundary first:** if the task's data can't leave your infrastructure, the judge must be a model you host, such as one deployed on Baseten.
* **A different model family from the one under test:** models tend to prefer outputs that resemble their own.
* **Cost at your data size:** a judge that scores every training row as well as the held-out set is called many times. A cheaper judge that passes the checks below is the better choice.

The following judge scores one axis, whether a direct answer addresses the request, with Kimi K3 on [Model APIs](https://docs.baseten.co/inference/model-apis/overview.md). Kimi K3 is a different model family from the Qwen model this guide trains. The [structured output](https://docs.baseten.co/inference/structured-outputs.md) schema puts `reasoning` and `evidence` before `verdict`, so the model writes its evidence before it decides:

```python judge.py theme={"system"}
import os
from typing import Literal

from openai import OpenAI
from pydantic import BaseModel

client = OpenAI(
    base_url="https://inference.baseten.co/v1", api_key=os.environ["BASETEN_API_KEY"]
)

RULE = (
    "The reply answers the user's request directly and completely, "
    "without calling or describing a tool."
)


class Verdict(BaseModel):
    reasoning: str
    evidence: str
    verdict: Literal["pass", "fail"]


def judge(request: str, reply: str) -> bool:
    response = client.beta.chat.completions.parse(
        model="moonshotai/Kimi-K3",
        messages=[
            {
                "role": "system",
                "content": f"Grade one rule. Rule: {RULE}\n"
                "Explain your reasoning, quote the part of the reply that decides it, "
                "then give the verdict.",
            },
            {"role": "user", "content": f"Request:\n{request}\n\nReply:\n{reply}"},
        ],
        response_format=Verdict,
        temperature=0,
    )
    verdict = response.choices[0].message.parsed
    if verdict is None:
        raise RuntimeError("The judge returned no verdict.")
    return verdict.verdict == "pass"
```

Call `judge()` only on rows where the code checks found a direct answer, and score each other judged axis with its own rule and call. Kimi K3 reasons before it answers by default; to lower the cost of each call, set a lower [`reasoning_effort`](https://docs.baseten.co/inference/model-apis/reasoning).

## Check the judge before you trust it

A judge is a measuring instrument. Measure it before it decides anything:

1. **Three known cases.** On real rows, confirm a known-good output passes, a known-bad output fails, and an empty or errored output neither crashes your parser nor scores as a pass.
2. **Stability.** Judge the same outputs several times and compute agreement per axis. Report a chance-corrected statistic such as Cohen's kappa. On an axis where 95% of outputs pass, a judge that always says pass reaches 95% raw agreement and a kappa near 0.
3. **Calibration.** Score outputs a person verified as good, and outputs with violations you planted on purpose. A judge that fails verified outputs is miscalibrated; a judge that misses planted violations is too lenient to gate a release or filter training data.

Expect a small fraction of judge responses to fail to parse. Re-judge those rows instead of dropping them, because dropped rows skew the score toward easy cases.

## Score a baseline

Before you train, score models you already have access to: the model that serves the task today, the base model you plan to train, and the strongest model you can call. Score them with the same prompts, tools, and sampling settings you use for every later run.

To score an open model without training it, give it an inference endpoint. [Model APIs](https://docs.baseten.co/inference/model-apis/overview.md) serve a curated set of models behind an OpenAI-compatible API, with nothing to deploy. [Dedicated Inference](https://docs.baseten.co/overview.md#dedicated-inference) deploys any open model on hardware you choose, for example from a [model recipe](https://docs.baseten.co/examples/models/overview.md). The [comparison script](#compare-a-checkpoint-with-its-baseline) takes any OpenAI-compatible base URL, so for a Model API, use `https://inference.baseten.co/v1` and the model's slug.

If an open model already meets your bar, you don't need to post-train. See [Decide whether to post-train](https://docs.baseten.co/post-training/decide.md).

Read the spread before you read the ranking. If clearly different models all land within the judge's run-to-run variation, the evaluation is saturated: it can't show a training gain, so make the task harder or the rules sharper first. If a smaller model beats a larger one, read the rows where it won before you decide whether the evaluation or your expectation is wrong.

## Compare a checkpoint with its baseline

[Deploy the checkpoint](https://docs.baseten.co/loops/deploy-checkpoints.md), then send the held-out prompts to both deployments with the same request settings. Scoring through the deployment measures what you would serve, including its chat template and tool handling.

To score two OpenAI-compatible deployments on the held-out set, configure the endpoints and run the script. The model and deployment IDs come from each deployment, and `model` is the served name: for a deployed Loops checkpoint, the checkpoint name such as `step-124`.

```python compare.py theme={"system"}
import json
import os

from openai import OpenAI

from checks import score

ENDPOINTS = {
    "baseline": (
        "https://model-<baseline_model_id>.api.baseten.co/deployment/<baseline_deployment_id>/sync/v1",
        "<served_model_name>",
    ),
    "candidate": (
        "https://model-<candidate_model_id>.api.baseten.co/deployment/<candidate_deployment_id>/sync/v1",
        "step-124",
    ),
}

rows = [json.loads(line) for line in open("holdout.jsonl")]
totals = {}
for label, (base_url, model) in ENDPOINTS.items():
    client = OpenAI(base_url=base_url, api_key=os.environ["BASETEN_API_KEY"])
    scores = []
    for row in rows:
        reference = next(m for m in row["messages"] if m["role"] == "assistant")
        prompt = row["messages"][: row["messages"].index(reference)]
        tools = {"tools": row["tools"]} if row.get("tools") else {}
        response = client.chat.completions.create(
            model=model, messages=prompt, temperature=0, max_tokens=1024, **tools
        )
        choice = response.choices[0]
        scores.append(
            score(choice.message.model_dump(), choice.finish_reason, reference)
        )
    totals[label] = {
        check: sum(s[check] for s in scores) / len(scores) for check in scores[0]
    }

for check in totals["baseline"]:
    print(
        f"{check:12} {totals['baseline'][check]:7.1%} {totals['candidate'][check]:7.1%}"
    )
```

The script prints one row per check with the baseline and candidate pass rates. The [SFT example](https://docs.baseten.co/examples/sft-tool-calling.md) scores the same four checks on the first assistant turn of 200 held-out conversations, sampled greedily. Its recorded run compared the base model with the trained adapter:

| Check | Base model | Adapter |
| - | - | - |
| `decision` | 88.0% | 99.0% |
| `tools` | 81.8% | 88.8% |
| `arguments` | 58.8% | 63.5% |
| `well_formed` | 91.0% | 98.5% |

Break each check down by the slices that matter to your task before you decide. In that run, the largest change was on requests no offered tool could serve: `decision` rose from 73.3% to 100% on those 30 conversations, because the base model called an unrelated tool and the adapter answered directly. Read the worst regressions row by row as well as the averages.

Release thresholds are a product decision. Set them from what the task needs, such as "no check drops by more than one point and `decision` on unservable requests is at least 98%", not from the baseline's numbers. Keep the held-out file, the checks, the judge prompts, and the judge model fixed across candidates; changing any of them makes earlier scores incomparable.

## Next steps

* [Prepare SFT data](https://docs.baseten.co/loops/sft-data.md): Use the same rules to choose training examples.
* [Check whether an RL run is learning](https://docs.baseten.co/loops/rl-run-health.md): Read reward and rollout metrics during training.
* [Deploy a checkpoint](https://docs.baseten.co/loops/deploy-checkpoints.md): Serve the checkpoint you evaluate.