Skip to main content
Once you’ve submitted training jobs, Baseten provides tools to manage your TrainingProjects and individual TrainingJobs. You can use the CLI or the API to manage your jobs.

​
TrainingProject management

  • List projects: To view all your training projects:
    This command will list all TrainingProjects you have access to, typically showing their names and IDs. Additionally, this command will show all active jobs.
  • View jobs within a project: To see all jobs associated with a specific project, use its project (obtained when creating the project or from truss train view):
  • Delete a TrainingProject: Delete a training project through the API or the dashboard. Using the API:
    From the Baseten dashboard:
    1. Select the training project you want to delete.
    2. Type the project name (for example, demo/qwen3-0.6b) to confirm.
    3. Choose Delete.
    When you delete a project, the following data is permanently deleted with no archival or recovery option:
    • All undeployed checkpoints from every job in the project
    • All data in the project’s training cache ($BT_PROJECT_CACHE_DIR)
    Checkpoints that have been deployed aren’t affected.

​
TrainingJob management

After submitting a job with truss train push config.py, you receive a project_id and job_id.
  • List jobs: List all jobs within a project using:
    This will typically show job IDs, statuses, creation times, etc.
  • Check status and retrieve logs: To view the logs for a specific job, tail them in real time or fetch existing logs.
    • To view logs for the most recently submitted job in the current context (for example, if you just pushed a job from your current terminal directory):
    • To view logs for a specific job using its job-id:
      Add --tail to follow the logs live.
  • Understand job statuses: The truss train view and truss train logs commands will help you track which status a job is in. For more on the job lifecycle, see the Lifecycle page.
  • Stop a TrainingJob: If you need to stop a running or pending job, use the stop command with the job’s project ID and job ID:
    This will transition the job to the TRAINING_JOB_STOPPED state.
  • Delete a TrainingJob: Delete a training job through the API or the dashboard. Using the API:
    From the Baseten dashboard:
    1. Select the project containing the job.
    2. Select the job you want to delete.
    3. Type the job name (for example, job-2) to confirm.
    4. Choose Delete.
    When you delete a job, all undeployed checkpoints are deleted permanently. There’s no archival or recovery option. Checkpoints that have been deployed aren’t affected.
  • Job outputs and checkpoints:
    • The primary outputs of a successful TrainingJob are model checkpoints (if checkpointing is enabled and configured).
    • These checkpoints are stored by Baseten. For more information on how CheckpointingConfig works, see Checkpoints.
    • When you are ready to deploy a model, you specify which checkpoints to use. The model_name you assign during deployment (using DeployCheckpointsConfig) becomes the identifier for this trained model version derived from your specific job’s checkpoints.
    • You can see the available checkpoints for a job through the Training API.

​
GPU capacity

Training jobs share your organization’s pool of GPUs. Whether a job starts right away depends on your team’s capacity limit, how much of the pool is already in use, and whether you run the job on spot capacity. By default, a job runs on dedicated capacity. Dedicated GPUs are reserved for the job and run it to completion without interruption, within your team’s limit. When you’re under the limit and GPUs are free, the job starts right away. When the pool is full, it waits in a queue. Spot capacity is how you run beyond your team’s limit. It borrows idle GPUs from the shared pool, within an organization-wide ceiling, at the cost of being interrupted when that capacity is needed elsewhere.
Spot capacity must be enabled for your Baseten workspace. Contact support to request access.

​
Team GPU capacity limits

Organizations can cap how many GPUs of each type a team can use at once. Dedicated jobs count against this limit; once a team reaches its ceiling, further dedicated jobs queue until capacity frees up. Spot jobs can run past it. To view your organization’s capacity and the per-team breakdown by GPU type:
The output shows organization-wide capacity, then each team’s limit and current usage:
Baseline is how many GPUs the organization can run on dedicated capacity. Limit caps all usage combined, dedicated and spot together, so the gap between the two is the most that can run as spot: here, up to 4 L4 GPUs. On-Demand and Spot show what’s in use now. Org admins can change a team’s limit at any time:
This sets the maximum number of concurrent H100 GPUs the ml-research team can use. You manage capacity limits through the Truss CLI; there’s no dashboard control for them yet.

​
Queue priority

When capacity is full, new training jobs wait in a queue (the TRAINING_JOB_PENDING state). Set --priority on truss train push to control the order jobs leave that queue:
Higher values run first. Jobs default to priority 0. Raise a job above 0 to run it sooner, or drop it below 0 to hold it behind default-priority work; the value is an unbounded integer. Priority only reorders jobs that are waiting for capacity. It doesn’t pause a running job or add capacity, so raising priority helps only when jobs are queued and capacity frees up. Set priority when you submit the job. To change a pending job’s priority, stop it with truss train stop and resubmit at the new value.

​
Spot capacity

Spot capacity runs a job on GPUs beyond your team’s limit, using capacity that would otherwise sit idle. Reach for it when you need to run more than your limit allows, or you don’t want to wait in the queue, for example to fire off extra experiments while your reserved GPUs are busy. Spot doesn’t create capacity, it borrows it, and it has its own ceiling: total usage stays within your organization’s Limit, so the most spot capacity you can get is the gap between Baseline and Limit in truss train capacity view. It only helps when the shared pool has idle GPUs to lend, so it’s most useful when other teams’ reserved GPUs are sitting unused. Think of spot capacity as borrowed, not owned. Baseten can reclaim it the moment that capacity is needed for dedicated work. Run a job on spot capacity by adding --spot to truss train push:
The flag overrides the availability_model field in your compute config. You can also set availability_model: SPOT in the config directly. When Baseten reclaims the capacity, the job is preempted and moves to the terminal TRAINING_JOB_PREEMPTED state. You can’t predict or control when this happens, or which of your spot jobs stops first, so treat every spot job as interruptible. Preemption isn’t a failure, and a preempted job doesn’t resume on its own. Checkpoint your progress as the job runs, then resubmit to continue from your latest checkpoint.