Skip to main content
The Jobs quickstart fine-tunes Qwen3-4B on a demo dataset. Moving to your own workload changes the dataset, how weights and data load, the training config, and the hardware. Everything else (the project layout, baseten train push --config config.py, checkpoint sync, deployment) stays the same.

Swap the dataset

The tutorial’s train.py loads a public Hugging Face dataset:
Point load_dataset() at your own Hugging Face repo, or at files bundled with your project (everything in your project directory ships with baseten train push --config config.py):
For gated or private Hugging Face models or datasets, add your hf_access_token secret to the job and read it from the environment; the tutorial’s container makes unauthenticated Hugging Face requests, so a gated model fails at training time without this. See secrets in training. TRL’s SFTTrainer consumes chat-format datasets (a messages column) directly. For other shapes, apply a formatting function; the TRL SFT docs cover the options.

Load weights and data through BDN

The tutorial’s train.py downloads the base model inside the container, on billed GPU time, on every job. The better pattern is to mount weights and large datasets through BDN: declare a WeightsSource on the job, and the files are on local disk before your start commands run. BDN mirrors each source once and caches it, so re-runs skip the download entirely. Add WeightsSource to the truss_train imports in config.py and declare the mounts:
config.py
Then load from the mount paths instead of remote IDs:
train.py
BDN supports Hugging Face, S3, GCS, R2, and HTTPS sources; private sources authenticate through a per-source auth block. For S3 and GCS, OIDC uses short-lived credentials, and S3 also supports AWS AssumeRole. See weights and data authentication for the full configuration.

Adjust the training config

The tutorial caps the run at 50 steps so it finishes fast. For a real run, train on the full dataset and checkpoint less often. Keep the tutorial’s other settings (learning_rate, bf16, max_length); these are the fields that change:
Keep output_dir on $BT_CHECKPOINT_DIR: that’s the directory Baseten syncs and deploys from. And every checkpoint you save is uploaded, so pick a save_steps cadence you’d actually resume or deploy from; frequent saves on a large model cost sync time and storage. If the job hits GPU out-of-memory, lower per_device_train_batch_size and raise gradient_accumulation_steps to hold the effective batch size, or move up a GPU tier.

Scale the hardware

Hardware lives in config.py, not in your training code. A bigger base model needs more GPUs:
For workloads beyond one machine, set node_count and Baseten handles the InfiniBand networking and orchestration; see multi-node training.

Iterate faster on re-runs

Your second submission shouldn’t re-download the base model or dataset. The tutorial’s config already enables the training cache (CacheConfig(enabled=True)); keep it on, and cache model downloads and preprocessed data under the cache directory so subsequent jobs skip them.

Seed large datasets into the cache

For a dataset that’s expensive to download, run a seed job once, then read the dataset from the cache in later training jobs. A GPU training job can seed the cache, but a CPU-only job keeps the download off billed GPU time. For a 400 GB Hugging Face dataset, seeding can save nearly an hour of compute on each subsequent job. Add a seed job to config.py. Enable the cache. The example uses a CPU-only job and sets project_accelerator to the GPU type that consumes the data, so Baseten places the CPU job on a cluster with that GPU type. Run the seed job in the same TrainingProject as the training jobs so they share the project cache.
config.py
The seed job installs the Hugging Face client libraries and runs seed.py. Create seed.py next to config.py. Write the dataset to $BT_PROJECT_CACHE_DIR so the project’s jobs can read it.
seed.py
snapshot_download skips files that are already cached, so re-running the seed job resumes an incomplete download. For S3 or GCS data, use the download pattern in Load data in your training script, install the source’s client library, and write the files under $BT_PROJECT_CACHE_DIR. Push the job from the directory containing config.py:
The command prints the job ID. Follow the run until it succeeds:
Read the dataset from the same directory in the training job. A successful read confirms the cache is populated:
train.py
For jobs in other projects, use $BT_TEAM_CACHE_DIR instead; the team cache is shared across a team’s jobs on the same cluster.

Debug a live job

To debug a live job instead of resubmitting, SSH into the running container or attach VS Code or Cursor.

Next steps