> ## Documentation Index > Fetch the complete documentation index at: https://docs.baseten.co/llms.txt > Use this file to discover all available pages before exploring further. # Check whether an RL run is learning > Read reward, discarded groups, KL divergence between the sampler and trainer, and transcripts to tell a learning RL run from a stalled or exploited one. An RL run can fail without an error. Rewards can stay flat because no group has anything to learn from, climb because the policy found a shortcut in the grader, or drift because the sampler and trainer disagree. For runs started with the Loops cookbook, read these signals in order; the earlier ones catch problems sooner. ## Where the metrics are Each run writes a directory under `[output] output_dir`. Its `run.json` holds the Loops run ID. The files you read most: * `steps.jsonl`: one record per update with every metric on this page. Set `[output] wandb_project` and install the `wandb` extra to mirror them to Weights & Biases. * `rollouts.jsonl`: one record per attempt, trained or discarded, with the reason it was discarded. * `transcripts/*.json`: with `[output] trace = "full"`, every generation each attempt sent and received, marked with whether it trains. `trace = "samples"` keeps only each generation's identity, policy version, and length, at about 1% of the size, without the text. For a long run, set `[output] progress_every_seconds` so the log reports progress between updates. Updates can be minutes apart, and a slow phase looks the same as a stall without it. ## Reward the run can learn from `reward/mean` is the average reward of the attempts an update trained on. Read it with two neighbors: * `reward/group_std_mean` is the average spread of rewards within each trained group. Group relative policy optimization (GRPO), the cookbook's RL algorithm, learns only from this spread. If it falls toward zero while `reward/mean` is still low, the policy has stopped exploring: every attempt at a task does the same thing. * `reward/mean_with_unusable` adds the attempts from groups the run rejected. When the policy starts solving or failing whole groups, those groups are rejected and `reward/mean` moves less than the real reward. This metric shows the full picture. `reward/std` is the spread across the whole batch, so it can read high even when every group's attempts scored alike. Use `reward/group_std_mean` to judge learning signal. ## Rejected groups A group is rejected as `no_variance` when every attempt scored the same, and as `group_too_small` when more than `max_group_losses` of its attempts failed. Both appear as `discard/groups/`. A high `no_variance` count at the start means your tasks are too easy or too hard for the starting model. See [size tasks so attempts disagree](https://docs.baseten.co/loops/rl-environments.md#size-tasks-so-attempts-disagree). A rising count later can mean the policy has mastered the tasks. Compare it with `reward/mean_with_unusable` to tell which. `[training] max_consecutive_unusable_groups` stops the run after that many rejected groups in a row. It exists so a run doesn't spend hours on sandboxes and sampling without training. ## Failed attempts `discard/rollouts/` counts attempts that never reached training, by reason, such as `rollout_failed/` for an agent that raised, `grading_failed/` for a grader that raised, or `unversioned` for an attempt cut before its first model reply. `rollouts.jsonl` has the same reason for every attempt. A steady trickle of infrastructure failures is normal for cloud sandboxes, and `max_rollout_retries` absorbs it. A burst means the environment or grader is broken. `[rollout] max_consecutive_rollout_failures` stops the run when failures don't let up. ## Your own episode metrics Every key your agent returns as episode metadata, and every key your grader adds, becomes a `rollout/*` metric: numbers are averaged, labels are counted, and booleans become rates. An agent that returns `{"termination_reason": "max_turns", "turns": 12}` gets `rollout/turns` and `rollout/termination_reason/max_turns` beside the reward. Record why each episode ended. A rising `max_turns` or `truncated` rate with a flat reward means the agent is running out of room, not learning the task. ## Sampler and trainer KL divergence The sampler generates rollouts and the trainer computes their probabilities again during the update. `train/mismatch_kl` measures the KL divergence between the two over the sampled tokens, and `train/mismatch_ess` reports the effective sample size of the importance weights. A healthy run keeps the KL near zero and the effective sample size near 1. A rising KL means the trainer is correcting for rollouts it would no longer produce, from stale weights or a numerics mismatch. See [Monitor KL divergence between the sampler and trainer](https://docs.baseten.co/examples/rl-sdk-loop.md#monitor-kl-divergence-between-the-sampler-and-trainer). In `[training] mode = "async"`, rollouts can come from older policy versions. `freshness/lag_mean` and `freshness/lag_p90` report how many updates behind the trainer each trained rollout was. `max_steps_off_policy` caps that lag; attempts older than it are dropped. ## Read transcripts, not only curves Reward measures what your grader checks, which isn't always what you meant. Set `trace = "full"` for at least your first runs on a new environment, and read the transcripts of the highest-reward attempts every few updates: * **A reward that climbs faster than the task allows.** The usual sign that the policy found a way to satisfy the grader without doing the task, such as editing the file a check reads or emitting a format the parser accepts. * **Behavior you'd reject in review.** Correct answers reached by an approach that won't hold up, such as hard-coding a test's expected output. * **Shrinking or collapsing replies.** `sampling/entropy_estimate` falling sharply alongside repetitive transcripts means the policy is collapsing onto one response. When you find an exploit, fix the grader or environment and start a new run. Training longer on a broken reward reinforces the exploit. ## Confirm on held-out tasks Reward on training tasks shows the policy is learning those tasks. Whether it's better at the job is a separate question. Save sampler checkpoints with `[output] sampler_checkpoint_every`, then score them on tasks the run never trained on with your [evaluation](https://docs.baseten.co/post-training/evaluate.md) before you deploy one. ## Next steps * [Train a tool-using agent with RL](https://docs.baseten.co/examples/rl-tool-agent.md): Apply these checks to a first run. * [Design RL environments and rewards](https://docs.baseten.co/loops/rl-environments.md): Fix tasks and graders the metrics point to. * [Evaluate a model](https://docs.baseten.co/post-training/evaluate.md): Compare a checkpoint with its baseline on held-out tasks. * [Deploy a checkpoint](https://docs.baseten.co/loops/deploy-checkpoints.md): Serve the checkpoint that passes.