Skip to main content
Training Jobs runs your training code (an Axolotl config, TRL script, or custom loop) unchanged in a container on managed GPUs. Baseten provisions the hardware you declare, syncs checkpoints as your job saves them, and deploys any checkpoint as a production endpoint. Deciding between Jobs and Loops? See Training on Baseten.

​
How Training Jobs works

Baseten stores your checkpoints while the job runs and deploys any of them as a production endpoint. You don’t download weights, re-upload them, or manage separate serving infrastructure. The core workflow is two commands:
From job submission to a served model:
  1. Define your job: Declare compute, container image, runtime, and checkpointing in a Python config file.
  2. Submit it: truss train push packages your code and starts the job on H100 or H200 GPUs, single-node or multi-node.
  3. Watch checkpoints sync: Baseten stores each checkpoint your job saves.
  4. Deploy a checkpoint: truss train deploy_checkpoints turns any synced checkpoint into a production endpoint.

​
Supported frameworks

Training Jobs is framework-agnostic: if your code runs in a container, it runs here. Browse the ML Cookbook for more examples including multi-node training with FSDP and DeepSpeed.

​
Key features

​
Checkpoint management

Checkpoints sync automatically to Baseten storage during training. You can:
  • Deploy any checkpoint as a production endpoint with truss train deploy_checkpoints.
  • Download checkpoints for local evaluation and analysis.
  • Resume from any checkpoint if a job fails or you want to train further.
Learn more about checkpointing.

​
BDN weight and data loading

Load model weights and training data through Baseten Delivery Network (BDN). Mount weights from Hugging Face, S3, GCS, R2, or any HTTPS URL directly into your training container with no download code needed. BDN mirrors weights before compute is provisioned, then caches them for faster mounting on subsequent jobs. See storage and data ingestion for setup details.

​
Persistent caching

Speed up training iterations by caching models, datasets, and preprocessed data between jobs. The cache persists across training runs, so you don’t re-download 70B models every time. See the training cache guide for configuration options.

​
Multi-node training

Scale training across multiple GPU nodes with InfiniBand networking. Baseten handles node orchestration, communication setup, and environment variables. You set node_count in your configuration. Learn more about multi-node training.

​
Remote access

Connect to running training containers to debug, inspect state, and iterate without resubmitting. Baseten offers two options:
  • SSH: Connect from any OpenSSH client for terminal sessions and file transfer with scp or sftp.
  • VS Code & Cursor: Connect from VS Code or Cursor Remote Tunnels for a full IDE experience.
See the Remote access overview to choose between them.

​
Next steps

Jobs quickstart

Run your first training job and deploy the result.

Compare Jobs and Loops

Choose between container-driven Jobs and the Loops training SDK.

ML Cookbook

Production-ready examples for frameworks and models.

​
Reference