baseten train push --config config.py, checkpoint sync, deployment) stays the same.
Swap the dataset
The tutorial’strain.py loads a public Hugging Face dataset:
load_dataset() at your own Hugging Face repo, or at files bundled with your project (everything in your project directory ships with baseten train push --config config.py):
hf_access_token secret to the job and read it from the environment; the tutorial’s container makes unauthenticated Hugging Face requests, so a gated model fails at training time without this. See secrets in training.
TRL’s SFTTrainer consumes chat-format datasets (a messages column) directly. For other shapes, apply a formatting function; the TRL SFT docs cover the options.
Load weights and data through BDN
The tutorial’strain.py downloads the base model inside the container, on billed GPU time, on every job. The better pattern is to mount weights and large datasets through BDN: declare a WeightsSource on the job, and the files are on local disk before your start commands run. BDN mirrors each source once and caches it, so re-runs skip the download entirely.
Add WeightsSource to the truss_train imports in config.py and declare the mounts:
config.py
train.py
auth block. For S3 and GCS, OIDC uses
short-lived credentials, and S3 also supports AWS AssumeRole. See weights and
data authentication for the full
configuration.
Adjust the training config
The tutorial caps the run at 50 steps so it finishes fast. For a real run, train on the full dataset and checkpoint less often. Keep the tutorial’s other settings (learning_rate, bf16, max_length); these are the fields that change:
output_dir on $BT_CHECKPOINT_DIR: that’s the directory Baseten syncs and deploys from. And every checkpoint you save is uploaded, so pick a save_steps cadence you’d actually resume or deploy from; frequent saves on a large model cost sync time and storage.
If the job hits GPU out-of-memory, lower per_device_train_batch_size and raise gradient_accumulation_steps to hold the effective batch size, or move up a GPU tier.
Scale the hardware
Hardware lives inconfig.py, not in your training code. A bigger base model needs more GPUs:
node_count and Baseten handles the InfiniBand networking and orchestration; see multi-node training.
Iterate faster on re-runs
Your second submission shouldn’t re-download the base model or dataset. The tutorial’s config already enables the training cache (CacheConfig(enabled=True)); keep it on, and cache model downloads and preprocessed data under the cache directory so subsequent jobs skip them.
Seed large datasets into the cache
For a dataset that’s expensive to download, run a seed job once, then read the dataset from the cache in later training jobs. A GPU training job can seed the cache, but a CPU-only job keeps the download off billed GPU time. For a 400 GB Hugging Face dataset, seeding can save nearly an hour of compute on each subsequent job. Add a seed job toconfig.py. Enable the cache. The example uses a CPU-only job and sets project_accelerator to the GPU type that consumes the data, so Baseten places the CPU job on a cluster with that GPU type. Run the seed job in the same TrainingProject as the training jobs so they share the project cache.
config.py
seed.py.
Create seed.py next to config.py. Write the dataset to $BT_PROJECT_CACHE_DIR so the project’s jobs can read it.
seed.py
snapshot_download skips files that are already cached, so re-running the seed job resumes an incomplete download.
For S3 or GCS data, use the download pattern in Load data in your training script, install the source’s client library, and write the files under $BT_PROJECT_CACHE_DIR.
Push the job from the directory containing config.py:
train.py
$BT_TEAM_CACHE_DIR instead; the team cache is shared across a team’s jobs on the same cluster.
Debug a live job
To debug a live job instead of resubmitting, SSH into the running container or attach VS Code or Cursor.Next steps
- Browse the ML Cookbook for complete recipes: Axolotl, DPO, RL with VeRL, and multi-node FSDP.
- Deploy your checkpoints when training finishes.