Train a coding agent on Terminal-Bench

This example trains GLM-5.3-Flash with reinforcement learning (RL) as a coding agent. The agent works through Terminal-Bench 2.1 tasks with one bash tool, each task in its own Blaxel sandbox. Each task’s own tests grade the result. The tasks are Harbor tasks, so Harbor builds each environment and runs each task’s verifier. You write only the agent. Start with the tool-using agent example if you haven’t run RL with the Loops cookbook before. This run is much larger, and some tasks take hours.

Prerequisites

A Baseten workspace with Loops access to zai-org/GLM-5.3-Flash, and its API key. GLM-5.3-Flash trains in either availability mode.
A Blaxel workspace and API key, with capacity for about 50 concurrent sandboxes at 8 GB of memory each.
uv and Python 3.12 or later.

Install the cookbook

Install the Loops cookbook from PyPI into a uv project. The cookbook needs Python 3.12 or later, installs the baseten-loops SDK, and provides the loops command. loops runs both recipes; each training config names its recipe with recipe = "rl" or recipe = "sft". Create a project first if you don’t have one:

uv init --python 3.12 loops-project
cd loops-project
uv add loops-cookbook

The package includes the cookbook’s examples. Run one using its bundled config path, such as examples/counting/train.toml, from your project’s directory. To change an example, copy its directory into your project at the same path. A config in your project wins over the installed one, and loops check prints the directory of the config it used. Create a .env file in your project’s directory and set BASETEN_API_KEY to a workspace API key. The commands load .env from the training config’s directory or the directory you run them from. Never commit .env. A run uses the serverless availability mode when your workspace has serverless access to the model, and the dedicated mode otherwise. To choose a mode, add BASETEN_LOOPS_SCOPE=serverless or BASETEN_LOOPS_SCOPE=deployment to .env. Install Harbor with its Blaxel environment, and add BL_WORKSPACE and BL_API_KEY to .env:

uv add "loops-cookbook[harbor-blaxel]"

The config mirrors metrics to Weights & Biases. To use it, install the extra with the following command, then add WANDB_API_KEY to .env:

uv add "loops-cookbook[harbor-blaxel,wandb]"

To turn it off, start the run with the following command:

uv run loops run examples/terminal_bench/train.toml --output.wandb_project ""

Every setting in the config has a command-line flag named by its section and key, so you can change the example without copying it. uv run loops run --help lists the flags.

Read the agent

examples/terminal_bench/terminal_agent.py defines TerminalAgent, a Harbor agent that calls the policy directly. Its system prompt describes a software engineer in a sandbox. Its one tool runs a bash command in the task’s environment and returns the exit code, stdout, and stderr, each truncated to max_output_bytes. The agent loops until the model replies without a tool call or reaches max_turns. The agent decides what happens when a generation runs out of tokens. TerminalAgent discards a reply the token limit cut and samples the turn again, up to three times, so a cut generation never trains. After three cut attempts, the episode ends as truncated and is still graded. The tool-using agent makes the opposite choice: it ends and grades the episode, and the cut generation trains with the rest. Neither is a default; choose what fits your task. examples/terminal_bench/harbor.yaml pins the Terminal-Bench 2.1 dataset to one commit, selects the cookbook’s Blaxel environment, and passes the agent’s limits:

examples/terminal_bench/harbor.yaml

datasets:
  - repo: https://github.com/harbor-framework/terminal-bench-2-1@7131e4375048a0e408a8fb404b5f499d726b695b
    path: tasks
agents:
  - import_path: terminal_agent:TerminalAgent
    kwargs:
      max_turns: 64
      command_timeout: 120
      max_output_bytes: 16384

The job must use loops_cookbook.harbor.blaxel:BlaxelEnvironment rather than Harbor’s own Blaxel environment. The cookbook’s class records every sandbox in the run’s resource journal, so loops cleanup can release them after a crash.

Read the training config

examples/terminal_bench/train.toml trains asynchronously: the sampler collects rollouts for the next update while the trainer applies the current one. The settings that shape the run:

examples/terminal_bench/train.toml

[model]
base_model = "zai-org/GLM-5.3-Flash"
# Prime would auto-detect GLM-5.1.
renderer = "loops_cookbook.llm.renderer_extensions.glm53:renderer"
max_seq_len = 262144
lora_rank = 32
renderer_chat_template_kwargs = { reasoning_effort = "low", clear_thinking = false }

[training]
mode = "async"
max_steps = 50
learning_rate = 0.00001
adam_epsilon = 0.00000001
router_replay = false
group_size = 8
max_group_losses = 2
groups_per_step = 4
max_consecutive_unusable_groups = 100
max_steps_off_policy = 3
max_policy_delta_in_trajectory = 3

[compute]
# Both of these are account-specific opt-ins. Spot capacity can be reclaimed mid-run
# (the run stops; nothing restarts it). Sticky sessions need sampler affinity routing
# enabled for your Loops workspace; without it, requests fail. Set both to false if
# you are not sure your account has them.
spot_job = true
sampler_replica_affinity = true

Each update trains 4 groups of 8 attempts, for 50 updates.
max_steps_off_policy = 3 lets a trained rollout come from a policy up to three updates behind the trainer. Async runs trade that staleness for throughput. Read freshness/lag_mean to see the lag the run actually trained on.
max_group_losses = 2 lets a group train when up to 2 of its 8 sandboxes fail.

Turn both [compute] options off unless your workspace has spot capacity and sampler affinity routing enabled. With affinity on and routing off, sampling requests fail. To turn them off, start the run with the following command:

uv run loops run examples/terminal_bench/train.toml \
  --compute.spot_job false \
  --compute.sampler_replica_affinity false

spot_job requests spot capacity for deployments; it doesn’t choose between the serverless and dedicated modes.

Scale out the run

GLM-5.3-Flash uses the serverless mode by default when your workspace has access, which needs no capacity request and suits a first run. A long asynchronous run that needs more training and rollout throughput belongs in the dedicated mode, where you set the trainer and sampler replica counts. To use the dedicated mode, add the following line to .env:

.env

BASETEN_LOOPS_SCOPE=deployment

Then raise the replica counts when you start the run, for example as follows. trainer_replicas adds data-parallel trainer replicas for update throughput, and sampler_replicas adds sampler replicas for rollout throughput:

uv run loops run examples/terminal_bench/train.toml \
  --compute.trainer_replicas 2 \
  --compute.sampler_replicas 4

Each trainer replica is a full copy of the model’s GPU allocation, and deployments bill for GPU time, including idle time. To plan capacity for a large run in the dedicated mode, request more access. Scale the trainer explains how batches split across replicas.

Run the example

To check the configuration, run the following command. It downloads the dataset, validates the job and agent options, and launches nothing. It warns about any task that declares more disk than the sandbox provides:

uv run loops check examples/terminal_bench/train.toml

To train, run the following command. The run uses the serverless mode, billed for tokens, when your workspace has serverless access to the model, and the dedicated mode, billed for GPU time, otherwise. The run also starts one Blaxel sandbox per running attempt: 48 at a time, plus up to 5 started ahead so a freed slot is filled at once. Sandboxes bill while they run:

uv run loops run examples/terminal_bench/train.toml

Tasks vary from minutes to hours. The config allows each rollout up to 6 hours (rollout_timeout_seconds = 21600) and reports pipeline progress every 60 seconds, so a slow phase is distinguishable from a stall.

Read the results

Monitor reward/mean, discard/groups/no_variance, and rollout/* in W&B or in the run directory’s steps.jsonl. Terminal-Bench tasks vary widely in difficulty, so expect many no_variance groups from tasks the starting model always fails. See Check whether an RL run is learning. To tell whether the trained agent is a better coding agent, score a saved checkpoint on tasks it didn’t train on. Training reward on Terminal-Bench measures the tasks in the training set. See Evaluate a model.

Add a loss on command output

examples/terminal_bench/train_echo.toml runs the same job with ECHO, a custom loss defined in echo_loss.py. It adds a next-token loss on the output the agent’s commands print, weighted by observation_weight, to the importance-sampling loss on the agent’s own tokens. The module’s docstring describes where it differs from the paper. To install the loss’s extra, run the following command:

uv add "loops-cookbook[harbor-blaxel,custom-loss]"

To check the configuration, run the following command:

uv run loops check examples/terminal_bench/train_echo.toml

To start the run, run the following command:

uv run loops run examples/terminal_bench/train_echo.toml

Shut down the run

loops run leaves the Loops run active. On serverless, deactivation is optional because idle runs incur no token charges. On dedicated capacity, deactivate the run when you finish to stop GPU billing without waiting for automatic shutdown. To end the run and its paired sampler in either mode, run the following command with your run directory:

uv run loops deactivate "runs/terminal-bench/<run directory>"

The command prints the deactivated run ID. Saved checkpoints remain available.

Release sandboxes after a crash

Sandboxes bill until they’re deleted. If the run crashed, release its sandboxes with the following command:

uv run loops cleanup examples/terminal_bench/train.toml \
  --state-dir "runs/terminal-bench/<run directory>/state"

Next steps

Design RL environments and rewards: Graders, task sizing, and exploits.
Check whether an RL run is learning: Metrics for long, asynchronous runs.
Deploy a checkpoint: Serve the trained adapter.

Examples

Post-training

Models

Engines

Custom Docker servers

Custom Python models

Chains

Baseten for Model Labs

Train a coding agent on Terminal-Bench

Prerequisites

Install the cookbook

Read the agent

Read the training config

Scale out the run

Run the example

Read the results

Add a loss on command output

Shut down the run

Release sandboxes after a crash

Next steps

​Prerequisites

​Install the cookbook

​Read the agent

​Read the training config

​Scale out the run

​Run the example

​Read the results

​Add a loss on command output

​Shut down the run

​Release sandboxes after a crash

​Next steps

Prerequisites

Install the cookbook

Read the agent

Read the training config

Scale out the run

Run the example

Read the results

Add a loss on command output

Shut down the run

Release sandboxes after a crash

Next steps