Skip to main content
With RL Rollouts, you run your own reinforcement learning (RL) trainer and use Baseten to serve rollouts on dedicated inference deployments. Your trainer publishes checkpoints to your Amazon S3 or Google Cloud Storage (GCS) bucket, and the deployment loads new weights without redeploying. Start with Deploy RL Rollouts to connect a model to your checkpoints. To use Baseten-managed trainers, see Loops.

​
Capabilities

  • Deltas contain compressed changes between checkpoints, reducing the data you upload after each training step. In Baseten’s GLM-5.3 measurements, weight updates reached independent clusters in under 40 seconds, including about 6 seconds of request pause.
  • The Baseten Delivery Network (BDN) distributes checkpoints to rollout clusters and reduces repeated downloads from your bucket.
  • Multi-cloud Capacity Management (MCM) scales your rollout fleet across Baseten’s global compute.
  • The Baseten Inference Stack (BIS) serves models with KV-aware routing to help multi-turn episodes reuse their cache.
  • Responses can return token IDs, top-k log probabilities, mixture-of-experts (MoE) routes for router replay, and the top-p sampling mask. They can also report the policy version that produced them, so asynchronous RL can sample across policy versions.
  • Baseten can work with you to align the model’s numerical behavior with your trainer.

​
How it works

  1. Publish an anchor. Upload a full checkpoint to your bucket. This checkpoint is the starting point for later updates.
  2. Publish deltas. After each training step, upload the changes from the previous snapshot.
  3. Load the snapshot. Send a hot-load request naming the snapshot. Each replica prepares the update while serving, then pauses briefly to swap weights.
  4. Track the policy. Request engine_data in each rollout to record the policy version and replica. Replicas update independently, so your trainer can briefly receive samples from different policies.
Each deployment serves an OpenAI-compatible chat completions endpoint.

​
Supported models

Each folder contains a configuration that Baseten has validated for the listed model and accelerator. Set its BUCKET value to your bucket and prefix. Check the repository’s checkpoint requirements before publishing weights for your model.

​
The RL Rollouts repository

Use the RL Rollouts repository for model configurations, checkpoint requirements, and API details:

Deploy RL Rollouts

Clone the repository, set your bucket, and push your first deployment.