Skip to main content
An RL run can fail without an error. Rewards can stay flat because no group has anything to learn from, climb because the policy found a shortcut in the grader, or drift because the sampler and trainer disagree. For runs started with the Loops cookbook, read these signals in order; the earlier ones catch problems sooner.

​
Where the metrics are

Each run writes a directory under [output] output_dir. Its run.json holds the Loops run ID. The files you read most:
  • steps.jsonl: one record per update with every metric on this page. Set [output] wandb_project and install the wandb extra to mirror them to Weights & Biases.
  • rollouts.jsonl: one record per attempt, trained or discarded, with the reason it was discarded.
  • transcripts/*.json: with [output] trace = "full", every generation each attempt sent and received, marked with whether it trains. trace = "samples" keeps only each generation’s identity, policy version, and length, at about 1% of the size, without the text.
For a long run, set [output] progress_every_seconds so the log reports progress between updates. Updates can be minutes apart, and a slow phase looks the same as a stall without it.

​
Reward the run can learn from

reward/mean is the average reward of the attempts an update trained on. Read it with two neighbors:
  • reward/group_std_mean is the average spread of rewards within each trained group. Group relative policy optimization (GRPO), the cookbook’s RL algorithm, learns only from this spread. If it falls toward zero while reward/mean is still low, the policy has stopped exploring: every attempt at a task does the same thing.
  • reward/mean_with_unusable adds the attempts from groups the run rejected. When the policy starts solving or failing whole groups, those groups are rejected and reward/mean moves less than the real reward. This metric shows the full picture.
reward/std is the spread across the whole batch, so it can read high even when every group’s attempts scored alike. Use reward/group_std_mean to judge learning signal.

​
Rejected groups

A group is rejected as no_variance when every attempt scored the same, and as group_too_small when more than max_group_losses of its attempts failed. Both appear as discard/groups/<reason>. A high no_variance count at the start means your tasks are too easy or too hard for the starting model. See size tasks so attempts disagree. A rising count later can mean the policy has mastered the tasks. Compare it with reward/mean_with_unusable to tell which. [training] max_consecutive_unusable_groups stops the run after that many rejected groups in a row. It exists so a run doesn’t spend hours on sandboxes and sampling without training.

​
Failed attempts

discard/rollouts/<reason> counts attempts that never reached training, by reason, such as rollout_failed/<exception type> for an agent that raised, grading_failed/<exception type> for a grader that raised, or unversioned for an attempt cut before its first model reply. rollouts.jsonl has the same reason for every attempt. A steady trickle of infrastructure failures is normal for cloud sandboxes, and max_rollout_retries absorbs it. A burst means the environment or grader is broken. [rollout] max_consecutive_rollout_failures stops the run when failures don’t let up.

​
Your own episode metrics

Every key your agent returns as episode metadata, and every key your grader adds, becomes a rollout/* metric: numbers are averaged, labels are counted, and booleans become rates. An agent that returns {"termination_reason": "max_turns", "turns": 12} gets rollout/turns and rollout/termination_reason/max_turns beside the reward. Record why each episode ended. A rising max_turns or truncated rate with a flat reward means the agent is running out of room, not learning the task.

​
Sampler and trainer KL divergence

The sampler generates rollouts and the trainer computes their probabilities again during the update. train/mismatch_kl measures the KL divergence between the two over the sampled tokens, and train/mismatch_ess reports the effective sample size of the importance weights. A healthy run keeps the KL near zero and the effective sample size near 1. A rising KL means the trainer is correcting for rollouts it would no longer produce, from stale weights or a numerics mismatch. See Monitor KL divergence between the sampler and trainer. In [training] mode = "async", rollouts can come from older policy versions. freshness/lag_mean and freshness/lag_p90 report how many updates behind the trainer each trained rollout was. max_steps_off_policy caps that lag; attempts older than it are dropped.

​
Read transcripts, not only curves

Reward measures what your grader checks, which isn’t always what you meant. Set trace = "full" for at least your first runs on a new environment, and read the transcripts of the highest-reward attempts every few updates:
  • A reward that climbs faster than the task allows. The usual sign that the policy found a way to satisfy the grader without doing the task, such as editing the file a check reads or emitting a format the parser accepts.
  • Behavior you’d reject in review. Correct answers reached by an approach that won’t hold up, such as hard-coding a test’s expected output.
  • Shrinking or collapsing replies. sampling/entropy_estimate falling sharply alongside repetitive transcripts means the policy is collapsing onto one response.
When you find an exploit, fix the grader or environment and start a new run. Training longer on a broken reward reinforces the exploit.

​
Confirm on held-out tasks

Reward on training tasks shows the policy is learning those tasks. Whether it’s better at the job is a separate question. Save sampler checkpoints with [output] sampler_checkpoint_every, then score them on tasks the run never trained on with your evaluation before you deploy one.

​
Next steps