Skip to main content
In the serverless mode, you create a run and call the same training and sampling operations as in any Loops run. Baseten manages the infrastructure, and you pay for the tokens you train and sample. It’s self-serve: start runs, and start more of them, without contacting Baseten. The serverless mode supports a subset of models. Filter the model catalog by Serverless to see them and the sequence length each supports.

​
Start a serverless run

When your workspace has serverless access to the model, runs use the serverless mode by default; you don’t need to change your code. To require the serverless mode, set scope to serverless:
With scope="serverless", creating the run fails instead of falling back to the dedicated mode when serverless can’t take the run. See choose a mode for the environment variable and REST equivalents. To check serverless access for your workspace, call Get server capabilities. Each model in the response has a serverless field.

​
When a run can’t use serverless

With the default scope, a run uses the dedicated mode instead when:
  • Your workspace doesn’t have serverless access to the model.
  • The run asks for a LoRA rank or sequence length that serverless doesn’t support for the model.
  • The run reuses another run’s deployments.
Replica counts and the availability_model capacity type apply only in the dedicated mode. To scale out a run with them, set scope to deployment.

​
Billing

Serverless runs bill for training and sampling token usage. Time between training operations, such as time spent reading results or changing a reward function, doesn’t bill.

​
Usage and logs

baseten loops usage reports dedicated GPU allocations. It doesn’t report serverless token usage or spend. To inspect dedicated GPU allocations: Run the following command with an authenticated Baseten CLI:
When allocations hold GPUs, the output also includes a table of runs, GPU types, and statuses. For more information, see baseten loops usage. Deployment logs cover dedicated runs. For more information about errors from serverless training and sampling calls, see Loops troubleshooting.

​
Finish a run

An idle serverless run incurs no token charges, so deactivation isn’t required to avoid idle billing. Saved checkpoints remain available to download, deploy, or start a new run from. To end a run and its paired sampler: Save the checkpoints you need, then run the following command. Replace <run_id> with TrainingClient.run_id, and confirm when prompted:
The CLI confirms success with Deactivated Loops run <run_id>. For more information, see Deactivate a run.