Skip to main content
Once you’ve submitted training jobs, Baseten provides tools to manage your TrainingProjects and individual TrainingJobs. You can use the CLI or the API to manage your jobs.

​
TrainingProject management

​
List projects

To view all your training projects:
This lists every TrainingProject you have access to, with its ID, name, latest job, and checkpoint sync status, followed by a table of all active jobs.

​
View jobs in a project

To see all jobs in a specific project, pass its project (which you get when you create the project, or from truss train view):

​
Delete a project

Delete a training project through the API or the dashboard. Using the API:
To delete a project from the Baseten dashboard:
  1. Select the training project you want to delete.
  2. Type the project name (for example, demo/qwen3-0.6b) to confirm.
  3. Choose Delete.
Deleting a project permanently deletes the following, with no archival or recovery option:
  • All undeployed checkpoints from every job in the project
  • All data in the project’s training cache ($BT_PROJECT_CACHE_DIR)
Deleting a project doesn’t touch checkpoints you’ve already deployed.

​
TrainingJob management

After submitting a job with truss train push config.py, you receive a project_id and job_id.

​
List jobs

List all jobs within a project, with each job’s ID, status, instance type, capacity type, and creation time:

​
Check status and retrieve logs

To view the logs for the job you submitted most recently in the current context (for example, if you just pushed a job from your current terminal directory):
To view logs for a specific job using its job-id:
Add --tail to follow the logs live. Both truss train view and truss train logs show which status a job is in. For more on the job lifecycle, see Lifecycle.

​
Stop a job

To stop a running or pending job, use the stop command with the job’s project ID and job ID:
This transitions the job to the TRAINING_JOB_STOPPED state.

​
Delete a job

Delete a training job through the API or the dashboard. Using the API:
To delete a job from the Baseten dashboard:
  1. Select the project containing the job.
  2. Select the job you want to delete.
  3. Type the job name (for example, job-2) to confirm.
  4. Choose Delete.
Deleting a job permanently deletes all its undeployed checkpoints, with no archival or recovery option. It doesn’t touch checkpoints you’ve already deployed.

​
Job outputs and checkpoints

The primary outputs of a successful TrainingJob are model checkpoints, when you enable and configure checkpointing. Baseten stores them for you. For more on how CheckpointingConfig works, see Checkpoints. When you’re ready to deploy a model, you specify which checkpoints to use. The model_name you assign during deployment (using DeployCheckpointsConfig) becomes the identifier for the model version those checkpoints produce. To see the available checkpoints for a job, use the Training API.

​
GPU capacity

Training jobs draw on your organization’s GPU capacity. Whether a job starts right away depends on your team’s capacity limit, how much of your organization’s capacity is already in use, and whether you run the job on spot capacity. By default, a job runs on dedicated capacity. The job holds its dedicated GPUs and runs to completion without interruption. A dedicated job draws first on your team’s own capacity limit, and in most configurations can run past that limit by borrowing from a shared pool, the organization capacity that no team’s limit claims. Some configurations have no pool, leaving the team limit a hard ceiling. When neither has room, the job waits in a queue. Spot capacity is opt-in and runs on whatever your organization isn’t using, including another team’s idle GPUs, at the cost of interruption when Baseten needs that capacity elsewhere.
Baseten turns on spot capacity per workspace. Contact support to request access.

​
Team GPU capacity limits

Organizations can set how many GPUs of each type a team can use at once. Whether that number is a guaranteed baseline or a hard ceiling depends on whether your organization has a shared pool:
  • With a shared pool, the limit is a baseline your team always keeps. Dedicated jobs can run past it by borrowing unclaimed organization capacity.
  • Without a shared pool, the limit is a hard ceiling. Once your team reaches it, further dedicated jobs queue until capacity frees up.
To view your organization’s capacity and the per-team breakdown by GPU type:
The output shows organization-wide capacity, then each team’s limit and current usage:
Baseline is how many GPUs the organization can run on dedicated capacity. Limit caps all usage combined, dedicated and spot together. On-Demand shows the dedicated GPUs in use now, and Spot shows the spot GPUs. Org admins can change a team’s limit at any time:
This sets the maximum number of concurrent H100 GPUs the ml-research team can use. You manage capacity limits through the Truss CLI; there’s no dashboard control for them yet.

​
Shared pool

The shared pool is the part of your organization’s dedicated capacity that no team’s limit claims: the organization Baseline minus the sum of every team Limit. In the output above, a Baseline of 4 L4 GPUs and a single team limit of 3 leave a pool of 1. Dedicated jobs that exceed their team’s baseline draw from the pool, first come, first served across teams. Baseten reserves every team’s baseline before carving out the pool, so borrowing never eats into another team’s guaranteed capacity. A team without a limit has a baseline of zero and runs entirely on the pool. truss train capacity view doesn’t report pool size directly, so compute it from the values it prints.

​
No shared pool

Team limits are hard ceilings, with nothing to borrow, when any of these are true:
  • Team limits already add up to your organization’s dedicated capacity, leaving nothing unallocated.
  • Your organization sets no organization-wide limit, or no team limits at all.
  • Your organization has asked Baseten to disable pool borrowing.
In these configurations a dedicated job never runs past its team’s limit; it queues until capacity frees up. Spot is the only way past.

​
Spot capacity

Spot capacity runs a job on GPUs that would otherwise sit idle, wherever they are in your organization. Reach for it when you need to run more than your dedicated capacity allows, or you don’t want to wait in the queue, for example to fire off extra experiments while your dedicated GPUs are busy. Your organization’s Limit is the only gate on a spot job. Spot runs whenever total live usage across your organization, dedicated and spot in every team, leaves room under that Limit. Your team’s limit and the shared pool don’t apply. Both of these follow. If your team is at its limit with an empty pool while another team sits idle, a spot job still runs, on capacity that belongs to that team’s baseline. If your organization’s entire dedicated capacity is in use, spot keeps going into the band between Baseline and Limit. Available spot capacity is your Limit minus everything running now, which truss train capacity view reports as On-Demand plus Spot. In the output above, spot reaches all 8 L4 GPUs when nothing else is running, and 4 when the organization’s dedicated capacity is full. You borrow spot capacity, you don’t own it: Baseten can reclaim it the moment it needs that capacity for dedicated work. Run a job on spot capacity by adding --spot to truss train push:
The flag overrides the availability_model field in your compute config. You can also set availability_model: SPOT in the config directly. To check which capacity a job runs on, read the Capacity Type column in truss train view. It reads Spot for spot jobs and On-demand for dedicated ones. The single-job view (truss train view --job-id <job_id>) reports the same thing as Availability Model, using the raw spot and dedicated values. When Baseten reclaims the capacity, the job stops and moves to the terminal TRAINING_JOB_PREEMPTED state. You can’t predict or control when this happens, or which of your spot jobs stops first, so treat every spot job as interruptible. Preemption isn’t a failure, and a preempted job doesn’t resume on its own. Checkpoint your progress as the job runs, then resubmit to continue from your latest checkpoint.

​
Queue priority

When capacity is full, new training jobs wait in a queue (the TRAINING_JOB_PENDING state). Set --priority on truss train push to control the order jobs leave that queue:
Higher values run first. Jobs default to priority 0. Raise a job above 0 to run it sooner, or drop it below 0 to hold it behind default-priority work; the value is an unbounded integer. Priority only reorders jobs that are waiting for capacity. It doesn’t pause a running job or add capacity, so raising priority helps only when jobs wait in the queue and capacity frees up. Set priority when you submit the job. To change a pending job’s priority, stop it with truss train stop and resubmit at the new value.