> ## Documentation Index
> Fetch the complete documentation index at: https://docs.baseten.co/llms.txt
> Use this file to discover all available pages before exploring further.

# Training Jobs quickstart

> Run your first training job and deploy it to production.

Training Jobs runs your training code on managed cloud GPUs. You bring your
own framework, point it at a GPU type, and submit. Baseten handles provisioning,
syncs checkpoints as they're saved, and deploys any checkpoint as a production
endpoint in one command.

This tutorial fine-tunes Qwen3-4B with LoRA on a single H100, from job
submission to calling the deployed model.

You'll set up a project directory, define your infrastructure in a configuration file, and write the training scripts that run on an H100.

You're billed per minute of GPU time while the job runs and while the deployed model serves traffic; see [Baseten pricing](https://www.baseten.co/pricing/) for H100 rates. The training job is capped at 50 steps and ends on its own within minutes.

## Prerequisites

* **Baseten account**: [Sign up for Baseten](https://app.baseten.co/).
* **API key**: Generate an API key from [Settings > API keys](https://app.baseten.co/settings/account/api_keys).
* **Hugging Face token**: Store a [Hugging Face access token](https://huggingface.co/settings/tokens) as a [Baseten secret](/organization/secrets) named `hf_access_token`. The deploy step at the end of this tutorial needs it to download the base model.
* **[uv](https://docs.astral.sh/uv/)**: The Baseten CLI runs training commands through Truss, which it launches with `uv`, so keep `uv` on your `PATH`. Install the Baseten CLI and log in:

  ```bash theme={"system"}
  brew tap basetenlabs/baseten
  brew trust --formula basetenlabs/baseten/baseten
  brew install baseten
  baseten auth login
  ```

For other install options, see [the Baseten CLI reference](/reference/cli/baseten/overview). The Baseten CLI forwards your credentials to Truss, so you don't need a separate `truss login`.

## Create a training project

```bash theme={"system"}
mkdir my-training-project && cd my-training-project
```

### Write the configuration file

Your configuration file uses the `truss_train` library to define your training
infrastructure as Python objects:

* [`TrainingProject`](/reference/sdk/training#trainingproject): the top-level container for your project.
* [`TrainingJob`](/reference/sdk/training#trainingjob): a single job within a project, combining:
  * [`Image`](/reference/sdk/training#image): what container to run.
  * [`Compute`](/reference/sdk/training#compute): what hardware to provision.
  * [`Runtime`](/reference/sdk/training#runtime): how to start training and what to persist.

This is the file Baseten reads when you submit a job. It tells the platform
which GPU to provision, which container image to use, and where to sync
checkpoints.

Create `config.py`:

```python config.py theme={"system"}
from truss_train import (
    TrainingProject,
    TrainingJob,
    Image,
    Compute,
    Runtime,
    CacheConfig,
    CheckpointingConfig,
)
from truss.base.truss_config import AcceleratorSpec

BASE_IMAGE = "pytorch/pytorch:2.7.0-cuda12.8-cudnn9-runtime"

training_runtime = Runtime(
    start_commands=[
        "chmod +x ./run.sh && ./run.sh",
    ],
    cache_config=CacheConfig(enabled=True),
    checkpointing_config=CheckpointingConfig(enabled=True),
)

training_compute = Compute(
    accelerator=AcceleratorSpec(accelerator="H100", count=1),
)

training_job = TrainingJob(
    image=Image(base_image=BASE_IMAGE),
    compute=training_compute,
    runtime=training_runtime,
)

training_project = TrainingProject(
    name="qwen3-4b-lora-sft",
    job=training_job,
)
```

`CacheConfig` avoids re-downloading models and datasets between jobs.
`CheckpointingConfig` tells Baseten to sync your saved checkpoints so you can
deploy them later.

### Write the training scripts

Create `run.sh` to install dependencies and launch training. This tutorial uses
`pip install` in the start command, but you can also pre-install dependencies in
a [custom base image](/training/concepts/basics#base-images).

```bash run.sh theme={"system"}
#!/bin/bash
set -eux

pip install "trl>=0.20.0" "peft>=0.17.0" "transformers>=4.55.0" "datasets"

python train.py
```

Your `train.py` is your own training code. Baseten runs it as-is, so you can use
any framework or training loop that works locally. This example fine-tunes
[Qwen3-4B](https://huggingface.co/Qwen/Qwen3-4B) on the
[pirate-ultrachat-10k](https://huggingface.co/datasets/winglian/pirate-ultrachat-10k)
dataset using LoRA with [TRL](https://huggingface.co/docs/trl) (Transformer
Reinforcement Learning). The dataset teaches the model to respond in pirate
dialect, so you'll know fine-tuning worked when the deployed model starts saying
"Ahoy, matey!"

```python train.py theme={"system"}
import os
from datasets import load_dataset
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch
from peft import LoraConfig
from trl import SFTConfig, SFTTrainer

MODEL_ID = "Qwen/Qwen3-4B"
DATASET_ID = "winglian/pirate-ultrachat-10k"

dataset = load_dataset(DATASET_ID, split="train")

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    use_cache=False,
)

peft_config = LoraConfig(
    r=8,
    lora_alpha=16,
    target_modules="all-linear",
    lora_dropout=0.05,
    task_type="CAUSAL_LM",
)

training_args = SFTConfig(
    learning_rate=2e-4,
    num_train_epochs=1,
    max_steps=50,
    logging_steps=5,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    gradient_checkpointing=True,
    max_length=1024,
    warmup_steps=5,
    lr_scheduler_type="cosine",
    save_steps=25,
    bf16=True,
    output_dir=os.getenv("BT_CHECKPOINT_DIR", "./checkpoints"),
)

trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=dataset,
    processing_class=tokenizer,
    peft_config=peft_config,
)
trainer.train()

trainer.save_model(training_args.output_dir)
print(f"Training complete. Model saved to {training_args.output_dir}")
```

<Note>
  Save checkpoints to `$BT_CHECKPOINT_DIR` so Baseten can sync and deploy them.
  Baseten sets this variable automatically when checkpointing is enabled.
</Note>

With `save_steps=25` and `max_steps=50`, the trainer saves LoRA checkpoints at
steps 25 and 50.

## Submit the training job

Now that your project is set up, submit your training job. The CLI packages your
files, creates the training project, and starts the job on your specified GPU.

```bash theme={"system"}
baseten train push --config config.py
```

You'll see:

```output theme={"system"}
✨ Training job successfully created!
🪵 View logs for your job via 'truss train logs --job-id q4ok7dw --tail'
🔍 View metrics for your job via 'truss train metrics --job-id q4ok7dw'
📁 View cache summary via 'truss train cache summarize "qwen3-4b-lora-sft"'
🌐 View job in the UI: https://app.baseten.co/training/4w51mm3/logs/q4ok7dw
```

The Baseten CLI delegates training jobs to Truss, so the hints above name the Truss commands. Use the Baseten equivalents instead, such as `baseten train job logs --job-id <job_id> --tail`.

Copy the `job_id` to use in the next steps.

## Monitor the training job

Tail logs in real time with the job ID from the previous step.

```bash theme={"system"}
baseten train job logs --job-id <job_id> --tail
```

You can also view logs, metrics, and job status in the [Baseten dashboard](https://app.baseten.co/training/).

## Deploy the trained model

Checkpoints sync while training runs: the logs show each one move from `SYNCING` to `COMPLETE` as it's saved. When the job finishes, you'll see:

```output theme={"system"}
{'train_runtime': '133.4', 'train_samples_per_second': '5.997', 'train_steps_per_second': '0.375', 'train_loss': '1.911', 'epoch': '0.0801'}
Training complete. Model saved to /mnt/ckpts
Job already completed successfully, terminating job...
```

<Tabs>
  <Tab title="CLI">
    **To deploy a checkpoint**:

    Deploy your checkpoint to Baseten's inference platform. The deployment downloads
    the base model weights and serves them with your LoRA adapter using vLLM. This
    step uses the `hf_access_token` secret from the prerequisites because the
    serving layer downloads the base model separately.

    ```bash theme={"system"}
    baseten train checkpoint deploy
    ```

    Follow the interactive prompts to select a checkpoint, name your model, and choose a GPU.

    ```output theme={"system"}
    Fetching checkpoints for training job <job_id>...
    ? Use spacebar to select/deselect checkpoints to deploy.
      ○ .
      ○ checkpoint-50
    ❯ ○ checkpoint-25

    ? Enter the model name for your deployment: my-fine-tuned-model
    ? Select the GPU type to use for deployment: H100
    ? Select the number of H100 GPUs to use for deployment: 1
    ? Enter the huggingface secret name: hf_access_token

    Writing truss config to truss_configs/deployment-1_<deployment_id>/config.yaml
    To call the deployment, set your `model` parameter to one of ['checkpoint-25']
    ```

    To script this step instead of answering prompts, pass `--config` with a `DeployCheckpointsConfig`; see [serve your trained model](/training/deployment#advanced-cli-deployment).
  </Tab>

  <Tab title="UI">
    **To deploy a checkpoint**:

    1. Sign in to your workspace and open the [training dashboard](https://app.baseten.co/training/), then select your training job.
    2. Open the **Checkpoints** tab.
    3. Choose a checkpoint.
    4. Choose **Deploy** and configure your model name, instance type, and scaling settings.
  </Tab>
</Tabs>

### Test the deployment

Call your deployed model using the OpenAI-compatible chat format. The `model` field matches the checkpoint name you selected during deployment.

<Tabs>
  <Tab title="cURL">
    **To call your deployed model**:

    Set your API key as an environment variable:

    <CodeGroup>
      ```bash macOS/Linux theme={"system"}
      export BASETEN_API_KEY="paste-your-api-key-here"
      ```

      ```powershell Windows theme={"system"}
      setx BASETEN_API_KEY "paste-your-api-key-here"
      ```
    </CodeGroup>

    Then send the request:

    ```bash theme={"system"}
    curl -X POST https://model-<model_id>.api.baseten.co/environments/production/sync/v1/chat/completions \
      -H "Authorization: Bearer $BASETEN_API_KEY" \
      -H "Content-Type: application/json" \
      -d '{"model": "checkpoint-25", "messages": [{"role": "user", "content": "What is the best way to learn Python programming?"}]}'
    ```
  </Tab>

  <Tab title="Python">
    **To call your deployed model**:

    ```python theme={"system"}
    import os
    from openai import OpenAI

    client = OpenAI(
        api_key=os.environ["BASETEN_API_KEY"],
        base_url="https://model-<model_id>.api.baseten.co/environments/production/sync/v1"
    )

    response = client.chat.completions.create(
        model="checkpoint-25",
        messages=[{"role": "user", "content": "What is the best way to learn Python programming?"}],
    )

    print(response.choices[0].message.content)
    ```
  </Tab>

  <Tab title="CLI">
    **To call your deployed model**:

    ```bash theme={"system"}
    baseten model predict --model-id <model-id> --data '{"model": "checkpoint-25", "messages": [{"role": "user", "content": "What is the best way to learn Python programming?"}]}'
    ```
  </Tab>
</Tabs>

The fine-tuned model responds in pirate dialect, confirming that the LoRA adapter is active. Qwen3 opens its reply with a (often empty) `<think>` block from its reasoning mode:

```output theme={"system"}
<think>

</think>

Ahoy there, matey!  Seeking to learn Python, ye be in for a grand adventure!
Here be some tips to guide ye:

**1. Start with the basics!**  Aye, like a pirate needs a ship to sail the
seven seas, a programmer needs to know the basics...
```

## Clean up

The deployed model bills for its GPU while it's live. When you finish testing, delete it with [`DELETE /v1/models/{model_id}`](/reference/management-api/models/deletes-a-model-by-id):

```bash theme={"system"}
curl -X DELETE "https://api.baseten.co/v1/models/<model_id>" \
  -H "Authorization: Bearer $BASETEN_API_KEY"
```

Deleting the model doesn't touch your synced checkpoints. You can redeploy any of them later with `baseten train checkpoint deploy --job-id <job_id>`.

## Next steps

* [Train on your data](/training/your-own-data): swap the dataset, mount weights through BDN, and scale the hardware.
* [Monitor and manage training jobs](/training/management): for logs, metrics, and job lifecycle commands.
* [Training SDK reference](/reference/sdk/training): for all configuration options, including [base images](/reference/sdk/training#image), [secrets](/reference/sdk/training#secretreference), [private registries](/reference/sdk/training#dockerauth), and [`.truss_ignore` syntax](/reference/cli/training/training-cli#ignore-files-and-folders).
* Browse the [ML Cookbook](https://github.com/basetenlabs/ml-cookbook): for framework examples and [advanced recipes](https://github.com/basetenlabs/ml-cookbook/tree/main/recipes).