> ## Documentation Index
> Fetch the complete documentation index at: https://docs.baseten.co/llms.txt
> Use this file to discover all available pages before exploring further.

# Design RL environments and rewards

> Define tasks, an agent, and a grader for reinforcement learning on Loops, and size tasks so every group has something to learn from.

Reinforcement learning (RL) trains on what your agent does, scored by a grader you write. The environment is your training data: a task the model can't learn from, or a grader that rewards the wrong thing, wastes every GPU hour spent on it. [RL on Loops](https://docs.baseten.co/examples/rl-sdk-loop.md) covers the training loop underneath them.

## The pieces you write

An RL run repeats one cycle. It gives the current policy a task several times, scores each attempt, and trains on which attempts did better. You provide four things:

* **Tasks:** the inputs, one per line of a JSONL file, each with whatever the grader needs to score it, such as an expected answer or a check command the agent never sees.
* **An agent:** the code that turns a task into model calls and actions. It owns its prompts, its tools, its conversation history, and when it stops.
* **An environment:** where the agent's actions take effect. A task that only needs an answer needs none. A task where the agent edits files or runs commands needs a sandbox.
* **A grader:** code that turns what happened into a reward.

The Loops cookbook schedules attempts and records the exact tokens the sampler produces for every model call. It also provisions and releases sandboxes, computes advantages, and sends training batches to the Loops trainer. Training on the sampler's own tokens matters: text re-tokenized after the fact can differ from what the model generated, and the update then trains on tokens the policy never sampled.

Install the [Loops cookbook](https://pypi.org/project/loops-cookbook/) from PyPI into a [uv](https://docs.astral.sh/uv/) project. The cookbook needs Python 3.12 or later, installs the [`baseten-loops`](https://pypi.org/project/baseten-loops/) SDK, and provides the `loops` command. `loops` runs both recipes; each training config names its recipe with `recipe = "rl"` or `recipe = "sft"`. Create a project first if you don't have one:

```sh theme={"system"}
uv init --python 3.12 loops-project
cd loops-project
uv add loops-cookbook
```

The package includes the cookbook's examples. Run one using its bundled config path, such as `examples/counting/train.toml`, from your project's directory. To change an example, copy its directory into your project at the same path. A config in your project wins over the installed one, and `loops check` prints the directory of the config it used.

Create a `.env` file in your project's directory and set `BASETEN_API_KEY` to a [workspace API key](https://docs.baseten.co/organization/api-keys.md). The commands load `.env` from the training config's directory or the directory you run them from. Never commit `.env`.

A run uses the serverless [availability mode](https://docs.baseten.co/loops/availability-modes/overview.md) when your workspace has serverless access to the model, and the dedicated mode otherwise. To choose a mode, add `BASETEN_LOOPS_SCOPE=serverless` or `BASETEN_LOOPS_SCOPE=deployment` to `.env`.

In the cookbook, an environment that isn't a [Harbor](https://github.com/laude-institute/harbor) task is an `Episode` class: `run()` is the agent and `grade()` is the grader. The smallest one asks the model one question and checks the reply:

```python direct_episode.py theme={"system"}
from pydantic import BaseModel, ConfigDict

from loops_cookbook.episodes import Episode, Grading, Model, Sampling, Task


class AnswerTask(BaseModel):
    """One line of the dataset."""

    model_config = ConfigDict(extra="forbid")

    name: str
    instruction: str
    answer: str


async def ask(model: Model, instruction: str) -> str:
    conversation = model.conversation()
    reply = await conversation.generate(
        messages=[{"role": "user", "content": instruction}]
    )
    return (reply.content or "").strip()


class AnswerEpisode(Episode[AnswerTask, None]):
    grading_retries = 2  # grade() has no side effects, so it's safe to run again

    def __init__(self, task: Task[AnswerTask], options: None) -> None:
        super().__init__(task, options)
        self.answer: str | None = None  # scored as wrong if the model never replies

    async def run(self, sample: Sampling) -> None:
        self.answer = await ask(sample.model, self.task.value.instruction)

    async def grade(self, grading: Grading) -> float:
        return float(self.answer == self.task.value.answer)
```

The task model validates every line of the tasks file before anything launches, so a malformed task fails at `check` instead of an hour into the run. If your tasks already exist as Harbor tasks, write a Harbor agent instead and keep Harbor's verifier as the grader. The [tool-using agent example](https://docs.baseten.co/examples/rl-tool-agent.md) shows an `Episode` with a sandbox, and the [coding agent example](https://docs.baseten.co/examples/rl-coding-agent.md) shows a Harbor agent.

## Size tasks so attempts disagree

The cookbook trains with group relative policy optimization (GRPO). Each task runs `group_size` times, and each attempt's advantage is its reward minus the group's mean reward. The model learns which of its own attempts at the same task did better, never the absolute reward.

That has a direct consequence for task design: a group where every attempt scores the same has zero advantage everywhere and teaches nothing. A task the model always solves and a task it never solves are equally useless. The cookbook rejects those groups as `no_variance` and doesn't train on them.

Choose tasks the starting model solves some of the time. A useful set is a ladder:

* **Easy tasks** so early groups contain some successes.
* **Harder tasks**, each with a trap a small model often falls into, such as counting `paid` orders with `grep paid`, which also matches `unpaid`.

Measure before you scale up. Run a short job and read the per-group rewards. Make any task whose groups always score alike harder or easier, and use [`max_consecutive_unusable_groups`](https://docs.baseten.co/loops/rl-run-health#rejected-groups) to stop a run that's spending compute without learning.

## Grade what happened, not what the agent said

Score the result the task asks for. For an agent that edits files, run a check on the files it left. Don't score its claim that it finished. Keep the check out of the agent's view: a grader the agent can read is a grader it can learn to satisfy without doing the task.

Prefer graders code can decide:

* **Exact checks:** an expected answer, a test suite, a shell command that exits `0` when the task is done.
* **Partial credit:** a fraction of tests passed. Partial credit spreads rewards within a group, which gives GRPO more to learn from on hard tasks.
* **An LLM judge:** when no check can decide the result. Score each attempt on its own with `grade()`, or, when an attempt can't be scored alone but can be compared with the rest of its group, rank the group with `grade_group()`. The judge needs its own client and model, separate from the policy. Validate it the way you [validate an evaluation judge](https://docs.baseten.co/post-training/evaluate.md#check-the-judge-before-you-trust-it).

## Close the gaps a policy can exploit

RL optimizes the reward you wrote, not the behavior you meant. Before a long run, look for ways to score well without doing the task:

* **A check that trusts the environment.** A grader that runs inside the agent's sandbox can be influenced by processes the agent left running. Stop the agent's processes before grading, or grade in a separate sandbox.
* **Rewards for format alone.** A check that only parses the output rewards any well-formed answer. Check content.
* **Easy outs.** If giving up, refusing, or an empty answer scores better than a wrong attempt, the policy learns to give up.

Read sample transcripts while the run trains, not only the reward curve. A reward that climbs faster than the task should allow is the usual sign of an exploit. [Check whether an RL run is learning](https://docs.baseten.co/loops/rl-run-health.md) covers what to read.

## Decide what failures score

Agents time out, hit turn limits, and produce replies that won't parse. What each outcome is worth depends on the task, so the cookbook leaves it to your grader:

* **The agent ran out of time or turns:** `grade()` still runs, on whatever the agent left, and `grading.timed_out` tells it the deadline cut the agent. Score it as the task deserves, often `0`. If the deadline cuts the agent before its first model reply, there's nothing to train and the attempt is discarded.
* **A reply was cut by the token limit:** your agent decides whether to end the episode and grade it, or retry the turn.
* **The infrastructure failed, such as a sandbox that couldn't start:** raise an exception from `run()`. The cookbook retries the attempt up to `max_rollout_retries` times and otherwise discards it instead of training on it.

Return a reward for anything the agent did, however badly. Raise only when the episode didn't happen. Record the reason an episode ended as metadata returned from `run()`; the cookbook reports each key as a `rollout/*` metric beside the reward.

## Choose where the environment runs

| Backend | Use it when | Cleanup after a crash |
| - | - | - |
| None | The task needs only the model's answer. | Nothing to clean up. |
| Docker | You run Harbor tasks locally. Install with `uv add "loops-cookbook[harbor]"`. | Harbor stops containers on shutdown; after a hard kill, check `docker ps`. |
| [Blaxel](https://blaxel.ai) | Each attempt needs its own cloud sandbox. Install with `uv add "loops-cookbook[blaxel]"`, or `uv add "loops-cookbook[harbor-blaxel]"` for Harbor tasks. | `loops cleanup` releases every sandbox the run recorded. |

Sandboxes bill until they're deleted. The cookbook records every Blaxel sandbox it creates, so a crashed run can release them.

## Check the configuration

A training config names its recipe (`recipe = "rl"`), the episode, the tasks, the model, and the training settings. Run `check` before every run: it validates the tasks and options, confirms Loops serves the model at the sequence length you set, and launches nothing.

To check the smallest pattern in the cookbook, run the following command from your project's directory:

```sh theme={"system"}
uv run loops check patterns/custom_runtime/in_process_direct/train.toml
```

The report starts with the files the check used. The config comes from the installed package unless your project has a file at the same path:

```text wrap theme={"system"}
Config: <project>/.venv/lib/python3.12/site-packages/loops_cookbook/bundled/patterns/custom_runtime/in_process_direct/train.toml
  paths, modules: beside <project>/.venv/lib/python3.12/site-packages/loops_cookbook/bundled/patterns/custom_runtime/in_process_direct
  .env: <project>/.env
  runs: <project>/runs/custom-direct-none
```

With `BASETEN_API_KEY` set, the next line confirms the model, such as `Loops model: Qwen/Qwen3.5-4B; max_seq_len 1024 (enabled up to <limit>)`. The rest describes the run:

```text wrap theme={"system"}
Runtime: loops_cookbook.episodes:create_runtime; tasks: 1
Capabilities: model access direct (AnswerEpisode.model_access); runtime-acquired phases sampling; grading retries 2 (AnswerEpisode.grading_retries); group grading false
Renderer: Qwen35Renderer (auto-detected by Prime)
Images: unavailable (ImportError: Qwen3VLVideoProcessor requires the Torchvision library but it was not found in your environment); text-only training works, run `uv sync --extra vision` for image inputs
Mode: sync; batch: 1 × 16
Loss: cispo
Loss aggregation: sum
Rollout clock: 110s (derived by the runtime)
Concurrency: 16 sampling (+2 started ahead); 16 grading
Producers: main (trainable); a call that names none uses main
Configuration and runtime checked; no sandbox or Loops trainer launched.
```

`batch: 1 × 16` is one group of 16 attempts per update. To start the run, run the following command:

```sh theme={"system"}
uv run loops run patterns/custom_runtime/in_process_direct/train.toml
```

`Qwen/Qwen3.5-4B` in this pattern trains only in the dedicated [availability mode](https://docs.baseten.co/loops/availability-modes/overview.md). The [model catalog](https://docs.baseten.co/loops/supported-models.md) shows which models also train in the serverless mode.

## Next steps

* [Train a tool-using agent with RL](https://docs.baseten.co/examples/rl-tool-agent.md): A multi-turn agent in a sandbox, graded on the files it leaves.
* [Check whether an RL run is learning](https://docs.baseten.co/loops/rl-run-health.md): Read rewards, discarded groups, and transcripts.
* [RL on Loops](https://docs.baseten.co/examples/rl-sdk-loop.md): Write the rollout, advantage, and training loop yourself with the SDK.