Design RL environments and rewards

Reinforcement learning (RL) trains on what your agent does, scored by a grader you write. The environment is your training data: a task the model can’t learn from, or a grader that rewards the wrong thing, wastes every GPU hour spent on it. RL on Loops covers the training loop underneath them.

The pieces you write

An RL run repeats one cycle. It gives the current policy a task several times, scores each attempt, and trains on which attempts did better. You provide four things:

Tasks: the inputs, one per line of a JSONL file, each with whatever the grader needs to score it, such as an expected answer or a check command the agent never sees.
An agent: the code that turns a task into model calls and actions. It owns its prompts, its tools, its conversation history, and when it stops.
An environment: where the agent’s actions take effect. A task that only needs an answer needs none. A task where the agent edits files or runs commands needs a sandbox.
A grader: code that turns what happened into a reward.

The Loops cookbook schedules attempts and records the exact tokens the sampler produces for every model call. It also provisions and releases sandboxes, computes advantages, and sends training batches to the Loops trainer. Training on the sampler’s own tokens matters: text re-tokenized after the fact can differ from what the model generated, and the update then trains on tokens the policy never sampled. Install the Loops cookbook from PyPI into a uv project. The cookbook needs Python 3.12 or later, installs the baseten-loops SDK, and provides the loops command. loops runs both recipes; each training config names its recipe with recipe = "rl" or recipe = "sft". Create a project first if you don’t have one:

uv init --python 3.12 loops-project
cd loops-project
uv add loops-cookbook

The package includes the cookbook’s examples. Run one using its bundled config path, such as examples/counting/train.toml, from your project’s directory. To change an example, copy its directory into your project at the same path. A config in your project wins over the installed one, and loops check prints the directory of the config it used. Create a .env file in your project’s directory and set BASETEN_API_KEY to a workspace API key. The commands load .env from the training config’s directory or the directory you run them from. Never commit .env. A run uses the serverless availability mode when your workspace has serverless access to the model, and the dedicated mode otherwise. To choose a mode, add BASETEN_LOOPS_SCOPE=serverless or BASETEN_LOOPS_SCOPE=deployment to .env. In the cookbook, an environment that isn’t a Harbor task is an Episode class: run() is the agent and grade() is the grader. The smallest one asks the model one question and checks the reply:

direct_episode.py

from pydantic import BaseModel, ConfigDict

from loops_cookbook.episodes import Episode, Grading, Model, Sampling, Task


class AnswerTask(BaseModel):
    """One line of the dataset."""

    model_config = ConfigDict(extra="forbid")

    name: str
    instruction: str
    answer: str


async def ask(model: Model, instruction: str) -> str:
    conversation = model.conversation()
    reply = await conversation.generate(
        messages=[{"role": "user", "content": instruction}]
    )
    return (reply.content or "").strip()


class AnswerEpisode(Episode[AnswerTask, None]):
    grading_retries = 2  # grade() has no side effects, so it's safe to run again

    def __init__(self, task: Task[AnswerTask], options: None) -> None:
        super().__init__(task, options)
        self.answer: str | None = None  # scored as wrong if the model never replies

    async def run(self, sample: Sampling) -> None:
        self.answer = await ask(sample.model, self.task.value.instruction)

    async def grade(self, grading: Grading) -> float:
        return float(self.answer == self.task.value.answer)

The task model validates every line of the tasks file before anything launches, so a malformed task fails at check instead of an hour into the run. If your tasks already exist as Harbor tasks, write a Harbor agent instead and keep Harbor’s verifier as the grader. The tool-using agent example shows an Episode with a sandbox, and the coding agent example shows a Harbor agent.

Size tasks so attempts disagree

The cookbook trains with group relative policy optimization (GRPO). Each task runs group_size times, and each attempt’s advantage is its reward minus the group’s mean reward. The model learns which of its own attempts at the same task did better, never the absolute reward. That has a direct consequence for task design: a group where every attempt scores the same has zero advantage everywhere and teaches nothing. A task the model always solves and a task it never solves are equally useless. The cookbook rejects those groups as no_variance and doesn’t train on them. Choose tasks the starting model solves some of the time. A useful set is a ladder:

Easy tasks so early groups contain some successes.
Harder tasks, each with a trap a small model often falls into, such as counting paid orders with grep paid, which also matches unpaid.

Measure before you scale up. Run a short job and read the per-group rewards. Make any task whose groups always score alike harder or easier, and use max_consecutive_unusable_groups to stop a run that’s spending compute without learning.

Grade what happened, not what the agent said

Score the result the task asks for. For an agent that edits files, run a check on the files it left. Don’t score its claim that it finished. Keep the check out of the agent’s view: a grader the agent can read is a grader it can learn to satisfy without doing the task. Prefer graders code can decide:

Exact checks: an expected answer, a test suite, a shell command that exits 0 when the task is done.
Partial credit: a fraction of tests passed. Partial credit spreads rewards within a group, which gives GRPO more to learn from on hard tasks.
An LLM judge: when no check can decide the result. Score each attempt on its own with grade(), or, when an attempt can’t be scored alone but can be compared with the rest of its group, rank the group with grade_group(). The judge needs its own client and model, separate from the policy. Validate it the way you validate an evaluation judge.

Close the gaps a policy can exploit

RL optimizes the reward you wrote, not the behavior you meant. Before a long run, look for ways to score well without doing the task:

A check that trusts the environment. A grader that runs inside the agent’s sandbox can be influenced by processes the agent left running. Stop the agent’s processes before grading, or grade in a separate sandbox.
Rewards for format alone. A check that only parses the output rewards any well-formed answer. Check content.
Easy outs. If giving up, refusing, or an empty answer scores better than a wrong attempt, the policy learns to give up.

Read sample transcripts while the run trains, not only the reward curve. A reward that climbs faster than the task should allow is the usual sign of an exploit. Check whether an RL run is learning covers what to read.

Decide what failures score

Agents time out, hit turn limits, and produce replies that won’t parse. What each outcome is worth depends on the task, so the cookbook leaves it to your grader:

The agent ran out of time or turns: grade() still runs, on whatever the agent left, and grading.timed_out tells it the deadline cut the agent. Score it as the task deserves, often 0. If the deadline cuts the agent before its first model reply, there’s nothing to train and the attempt is discarded.
A reply was cut by the token limit: your agent decides whether to end the episode and grade it, or retry the turn.
The infrastructure failed, such as a sandbox that couldn’t start: raise an exception from run(). The cookbook retries the attempt up to max_rollout_retries times and otherwise discards it instead of training on it.

Return a reward for anything the agent did, however badly. Raise only when the episode didn’t happen. Record the reason an episode ended as metadata returned from run(); the cookbook reports each key as a rollout/* metric beside the reward.

Choose where the environment runs

Backend	Use it when	Cleanup after a crash
None	The task needs only the model’s answer.	Nothing to clean up.
Docker	You run Harbor tasks locally. Install with `uv add "loops-cookbook[harbor]"`.	Harbor stops containers on shutdown; after a hard kill, check `docker ps`.
Blaxel	Each attempt needs its own cloud sandbox. Install with `uv add "loops-cookbook[blaxel]"`, or `uv add "loops-cookbook[harbor-blaxel]"` for Harbor tasks.	`loops cleanup` releases every sandbox the run recorded.

Sandboxes bill until they’re deleted. The cookbook records every Blaxel sandbox it creates, so a crashed run can release them.

Check the configuration

A training config names its recipe (recipe = "rl"), the episode, the tasks, the model, and the training settings. Run check before every run: it validates the tasks and options, confirms Loops serves the model at the sequence length you set, and launches nothing. To check the smallest pattern in the cookbook, run the following command from your project’s directory:

uv run loops check patterns/custom_runtime/in_process_direct/train.toml

The report starts with the files the check used. The config comes from the installed package unless your project has a file at the same path:

Config: <project>/.venv/lib/python3.12/site-packages/loops_cookbook/bundled/patterns/custom_runtime/in_process_direct/train.toml
  paths, modules: beside <project>/.venv/lib/python3.12/site-packages/loops_cookbook/bundled/patterns/custom_runtime/in_process_direct
  .env: <project>/.env
  runs: <project>/runs/custom-direct-none

With BASETEN_API_KEY set, the next line confirms the model, such as Loops model: Qwen/Qwen3.5-4B; max_seq_len 1024 (enabled up to <limit>). The rest describes the run:

Runtime: loops_cookbook.episodes:create_runtime; tasks: 1
Capabilities: model access direct (AnswerEpisode.model_access); runtime-acquired phases sampling; grading retries 2 (AnswerEpisode.grading_retries); group grading false
Renderer: Qwen35Renderer (auto-detected by Prime)
Images: unavailable (ImportError: Qwen3VLVideoProcessor requires the Torchvision library but it was not found in your environment); text-only training works, run `uv sync --extra vision` for image inputs
Mode: sync; batch: 1 × 16
Loss: cispo
Loss aggregation: sum
Rollout clock: 110s (derived by the runtime)
Concurrency: 16 sampling (+2 started ahead); 16 grading
Producers: main (trainable); a call that names none uses main
Configuration and runtime checked; no sandbox or Loops trainer launched.

batch: 1 × 16 is one group of 16 attempts per update. To start the run, run the following command:

uv run loops run patterns/custom_runtime/in_process_direct/train.toml

Qwen/Qwen3.5-4B in this pattern trains only in the dedicated availability mode. The model catalog shows which models also train in the serverless mode.

Next steps

Train a tool-using agent with RL: A multi-turn agent in a sandbox, graded on the files it leaves.
Check whether an RL run is learning: Read rewards, discarded groups, and transcripts.
RL on Loops: Write the rollout, advantage, and training loop yourself with the SDK.

Get started

Model APIs

Baseten Grid

Inference

Frontier Gateway

Training

Sandboxes

Development

Deployment

Engines

Model Labs

Organization

Hosting options

Observability

Troubleshooting

Design RL environments and rewards

The pieces you write

Size tasks so attempts disagree

Grade what happened, not what the agent said

Close the gaps a policy can exploit

Decide what failures score

Choose where the environment runs

Check the configuration

Next steps

​The pieces you write

​Size tasks so attempts disagree

​Grade what happened, not what the agent said

​Close the gaps a policy can exploit

​Decide what failures score

​Choose where the environment runs

​Check the configuration

​Next steps

The pieces you write

Size tasks so attempts disagree

Grade what happened, not what the agent said

Close the gaps a policy can exploit

Decide what failures score

Choose where the environment runs

Check the configuration

Next steps