Loops quickstart

Train a LoRA adapter for Qwen3.6-35B-A3B with reinforcement learning (RL) on serverless Loops, sample from its published weights, save resumable training state, and deploy the tuned model for inference. Baseten manages the infrastructure. Training bills for token usage, and deployment bills for GPU usage while the deployment is live. Use the Python SDK to write the training loop.

Try the cookbook workflow

Run the same task with a config file instead of a Python training loop.

Define the task and reward

When you fine-tune a model with RL, you provide a task and a reward function that scores the model’s replies. The rewards tell the training loop which behaviors to reinforce. For example, if you want a model to classify support tickets, reward it for assigning the correct category. This quickstart uses a task with one correct answer so you can compare rewards across training steps. The script sends this prompt:

I am thinking of either Doom or Quake. Name it.
Reply with the title only.

A reply of Quake scores 1.0. Any other reply scores 0.0. Thinking is off, and a reply can have at most 16 tokens. Each training step samples 16 replies from the current policy. A reply’s advantage is its reward minus the mean reward of the 16, so the step makes correct replies more likely and wrong replies less likely. When all 16 replies score the same, every advantage is zero and the step samples again. The script runs five training steps with LoRA rank 8 and a learning rate of 2e-5, and prints the mean reward for each step.

To have your coding agent run this quickstart, see Run a quickstart with your agent.

Prerequisites

A Baseten workspace with Loops and a workspace API key.
uv and Python 3.12 or later.

Optionally, install the Baseten CLI to list and deploy checkpoints and deactivate the run from your terminal.

Install the SDK

Create a uv project and install baseten-loops:

uv init loops-quickstart
cd loops-quickstart
uv add baseten-loops

Export your workspace API key as BASETEN_API_KEY. Replace YOUR_API_KEY with your key:

export BASETEN_API_KEY="YOUR_API_KEY"

$env:BASETEN_API_KEY = "YOUR_API_KEY"

Save the following code as train_loops.py, then verify the installation by running uv run python train_loops.py:

train_loops.py

import baseten.loops

print(f"Loops SDK {baseten.loops.__version__}")

Output

Loops SDK <version>

Create a training run

Create a ServiceClient, then call create_lora_training_client() with the ID of the model you want to train. This example uses Qwen/Qwen3.6-35B-A3B. Set scope="serverless" so Baseten runs the training on hardware it selects and configures for you. The method waits until the trainer accepts operations and returns a TrainingClient. Replace the contents of train_loops.py with:

train_loops.py

from baseten.loops import (
    AdamParams,
    Datum,
    ModelInput,
    SamplingParams,
    ServiceClient,
    TensorData,
)

BASE_MODEL = "Qwen/Qwen3.6-35B-A3B"
MAX_SEQ_LEN = 1024
INSTRUCTION = "I am thinking of either Doom or Quake. Name it.\nReply with the title only."
ANSWER = "Quake"
GROUP_SIZE = 16
STEPS = 5
MAX_UNUSABLE_GROUPS = 20

service_client = ServiceClient()
training_client = service_client.create_lora_training_client(
    base_model=BASE_MODEL,
    rank=8,
    max_seq_len=MAX_SEQ_LEN,
    scope="serverless",
)

print(f"session_id={service_client.session_id}")
print(f"run_id={training_client.run_id}")

Score a reply

Write the question in Qwen’s chat format. The empty thinking block turns thinking off. The reward decodes a reply and compares it with the answer. Append to train_loops.py:

train_loops.py

tokenizer = training_client.get_tokenizer()
prompt_text = (
    f"<|im_start|>user\n{INSTRUCTION}<|im_end|>\n"
    "<|im_start|>assistant\n<think>\n\n</think>\n\n"
)
prompt_ids = tokenizer.encode(prompt_text, add_special_tokens=False)
prompt = ModelInput.from_ints(prompt_ids)


def reward(sequence):
    reply = tokenizer.decode(sequence.tokens, skip_special_tokens=True).strip()
    return float(reply == ANSWER)

Train on sampled replies

Each completed training step samples a usable group of 16 replies, updates the model from that group, and publishes new sampling weights for the next step. For RL, each sampled reply becomes one datum. It carries the reply’s tokens as targets, the sampler’s logprobs for those tokens, and the reply’s advantage. Prompt positions carry -100 as the target, so they don’t train. Append to train_loops.py:

train_loops.py

def build_datum(sequence, advantage):
    completion = sequence.tokens
    start = len(prompt_ids) - 1
    length = start + len(completion)
    return Datum(
        model_input=ModelInput.from_ints(prompt_ids + completion[:-1]),
        loss_fn_inputs={
            "target_tokens": TensorData(
                data=[-100] * start + completion, dtype="int64", shape=[length]
            ),
            "logprobs": TensorData(
                data=[0.0] * start + sequence.logprobs, dtype="float32", shape=[length]
            ),
            "advantages": TensorData(
                data=[0.0] * start + [advantage] * len(completion),
                dtype="float32",
                shape=[length],
            ),
        },
    )


sampling_client = training_client.save_weights_and_get_sampling_client()
step = unusable = 0
while step < STEPS:
    response = sampling_client.sample(
        prompt,
        num_samples=GROUP_SIZE,
        sampling_params=SamplingParams(max_tokens=16),
    )
    rewards = [reward(sequence) for sequence in response.sequences]
    mean = sum(rewards) / len(rewards)
    if all(r == mean for r in rewards):
        unusable += 1
        if unusable == MAX_UNUSABLE_GROUPS:
            raise SystemExit(f"{unusable} groups in a row scored {mean}")
        continue
    unusable = 0

    datums = [
        build_datum(sequence, r - mean)
        for sequence, r in zip(response.sequences, rewards)
    ]
    training_client.forward_backward(datums, loss_fn="cispo").result(timeout=600.0)
    training_client.optim_step(AdamParams(learning_rate=2e-5)).result(timeout=600.0)
    step += 1
    print(f"step {step}: reward {mean:.3f}")
    sampling_client = training_client.save_weights_and_get_sampling_client(
        f"step-{step}"
    )

The forward_backward() call uses the cispo loss and the sampler’s logprobs to correct for any difference between the policy that sampled a reply and the policy that trains on it. For more information, see Loss functions. save_weights_and_get_sampling_client() publishes a sampler checkpoint under sampler_weights/ that you can deploy to inference, and returns a SamplingClient for it. This checkpoint omits optimizer state, so you can’t resume training from it; use save_state() when you need a resumable checkpoint.

Sample from the tuned weights

The last save_weights_and_get_sampling_client() call returned a client for the weights from step 5. Generate one reply from it. Append to train_loops.py:

train_loops.py

sample = sampling_client.sample(
    prompt,
    num_samples=1,
    sampling_params=SamplingParams(max_tokens=16),
)
print(f"completion={tokenizer.decode(sample.sequences[0].tokens, skip_special_tokens=True)!r}")

You’ve now trained a LoRA adapter and generated a reply from its published weights. This confirms that sampling works; it doesn’t evaluate fine-tuning quality. To sample from newer weights, create a new sampling client. Baseten pins each serverless sampling client to the checkpoint used to create it.

Save resumable training state

Call save_state() to save the model and optimizer state. Append to train_loops.py:

train_loops.py

state = training_client.save_state(name="step-5-state").result(timeout=600.0)
print(f"resumable state at {state.path}")

Use the returned weights/ path to resume training.

Inspect the results

Call list_checkpoints() to list the run’s checkpoints, then close the local clients. Append to train_loops.py:

train_loops.py

for ckpt in training_client.list_checkpoints():
    print(ckpt.id, ckpt.checkpoint_id, ckpt.created_at)

sampling_client.close()
training_client.close()

Run the script from your project directory:

uv run python train_loops.py

The output has the following shape. Values and checkpoint rows vary:

Output

session_id=<session_id>
run_id=<run_id>
step 1: reward <mean_reward>
...
step 5: reward <mean_reward>
completion='<generated_reply>'
resumable state at <checkpoint_path>
<id> <checkpoint_name> <created_at>
...

The script prints the session and run IDs before the first sampling request. That request can take several minutes while serverless capacity starts.

Check that the script finishes without an error. Compare the mean rewards across steps. Sampled rewards can vary between runs. Use the returned checkpoint paths, names, and IDs directly. Serverless checkpoint names can include a unique suffix. To list the run’s checkpoints after the script exits: Replace <run_id> with the printed run ID:

REST API
Baseten CLI

curl --url "https://api.baseten.co/v1/loops/checkpoints?run_id=<run_id>" \
  --header "Authorization: Bearer $BASETEN_API_KEY"

baseten loops checkpoint list --run-id "<run_id>"

Deactivate the run

Closing the local clients doesn’t deactivate the run or its paired sampler. The run incurs no token charges while idle, so deactivation is optional. To deactivate the run and its paired sampler: Replace <run_id> with the ID printed by your script:

Python
REST API
Baseten CLI

from baseten.loops import ServiceClient

ServiceClient().deactivate_run("<run_id>")

Deactivate the run. The request returns HTTP 200:

curl --request POST \
  --url "https://api.baseten.co/v1/loops/runs/<run_id>/deactivate" \
  --header "Authorization: Bearer $BASETEN_API_KEY"

Check that run.status.name is INACTIVE:

curl --request GET \
  --url "https://api.baseten.co/v1/loops/runs/<run_id>" \
  --header "Authorization: Bearer $BASETEN_API_KEY"

The command asks for confirmation; pass --yes to skip it when scripting.

baseten loops run deactivate --run-id "<run_id>"

Saved checkpoints remain available after deactivation.

Deploy the checkpoint

Now that you’ve fine-tuned your model, deploy its latest sampler checkpoint to run evaluations or inference. Baseten serves the checkpoint and your trained adapter as a dedicated inference deployment. The deployment needs an hf_access_token in workspace secrets and bills for its GPU while it’s live.

Baseten CLI
REST API
Dashboard

To deploy a checkpoint:Replace <id> with a sampler checkpoint’s id, the first column of the listing above, then run:

baseten loops checkpoint deploy --checkpoint-id "<id>"

Follow the prompts for the model name, GPU type and count, and Hugging Face secret name. The output includes the IDs needed to call the model:

Output

Model ID: <model_id>
Deployment ID: <deployment_id>

To deploy a checkpoint:Replace <id> with a sampler checkpoint’s id, choose an instance type id from the instance-type catalog, and set the model name and Hugging Face secret:

Request

curl --request POST \
  --url "https://api.baseten.co/v1/loops/checkpoints/deploy" \
  --header "Authorization: Bearer $BASETEN_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{
    "checkpoint_ids": ["<id>"],
    "model_name": "loops-quickstart",
    "instance_type_id": "H100",
    "hf_secret_name": "hf_access_token"
  }'

The response returns the IDs needed to call the model:

Response

{
  "model_id": "<model_id>",
  "deployment_id": "<deployment_id>"
}

Use the returned model and deployment IDs to run inference on your endpoint. Baseten handles model serving and infrastructure management. When you’re done, delete the deployment to stop GPU billing. Deactivating the Loops run doesn’t delete the deployment.

Next steps

Design RL environments: Create tasks and reward functions for your own RL training.
Train a vision model to count shapes: Apply RL to image tasks without a sandbox.
Dedicated Loops: Configure dedicated hardware, replicas, and spot capacity for training.

Get started

Model APIs

Baseten Grid

Inference

Frontier Gateway

Training

Sandboxes

Development

Deployment

Engines

Model Labs

Organization

Hosting options

Observability

Troubleshooting

Loops quickstart

Try the cookbook workflow

Define the task and reward

Prerequisites

Install the SDK

Create a training run

Score a reply

Train on sampled replies

Sample from the tuned weights

Save resumable training state

Inspect the results

Deactivate the run

Deploy the checkpoint

Next steps

Try the cookbook workflow

​Define the task and reward

​Prerequisites

​Install the SDK

​Create a training run

​Score a reply

​Train on sampled replies

​Sample from the tuned weights

​Save resumable training state

​Inspect the results

​Deactivate the run

​Deploy the checkpoint

​Next steps

Define the task and reward

Prerequisites

Install the SDK

Create a training run

Score a reply

Train on sampled replies

Sample from the tuned weights

Save resumable training state

Inspect the results

Deactivate the run

Deploy the checkpoint

Next steps