Skip to main content
Once you’ve submitted training jobs, Baseten provides tools to manage your TrainingProjects and individual TrainingJobs. You can use the CLI or the API to manage your jobs.

​
TrainingProject management

​
List projects

To view all your training projects:
This lists every TrainingProject you have access to, with its ID, name, latest job, and checkpoint sync status, followed by a table of all active jobs.

​
View jobs in a project

To see all jobs in a specific project, pass its project (which you get when you create the project, or from baseten train project list):

​
Delete a project

Delete a training project through the API or the dashboard. Using the API:
To delete a project from the Baseten dashboard:
  1. Select the training project you want to delete.
  2. Type the project name (for example, demo/qwen3-0.6b) to confirm.
  3. Choose Delete.
Deleting a project permanently deletes the following, with no archival or recovery option:
  • All undeployed checkpoints from every job in the project
  • All data in the project’s training cache ($BT_PROJECT_CACHE_DIR)
Deleting a project doesn’t touch checkpoints you’ve already deployed.

​
TrainingJob management

After submitting a job with baseten train push --config config.py, you receive a project_id and job_id.

​
List jobs

List all jobs within a project, with each job’s ID, status, instance type, capacity type, and creation time:

​
Check status and retrieve logs

Find the job ID, then stream its logs:
Add --tail to follow the logs live. Both baseten train job list and baseten train job logs show which status a job is in. For more on the job lifecycle, see Lifecycle.

​
Stop a job

To stop a running or pending job, use the stop command with the job’s project ID and job ID:
This transitions the job to the TRAINING_JOB_STOPPED state.

​
Delete a job

Delete a training job through the API or the dashboard. Using the API:
To delete a job from the Baseten dashboard:
  1. Select the project containing the job.
  2. Select the job you want to delete.
  3. Type the job name (for example, job-2) to confirm.
  4. Choose Delete.
Deleting a job permanently deletes all its undeployed checkpoints, with no archival or recovery option. It doesn’t touch checkpoints you’ve already deployed.

​
Job outputs and checkpoints

The primary outputs of a successful TrainingJob are model checkpoints, when you enable and configure checkpointing. Baseten stores them for you. For more on how CheckpointingConfig works, see Checkpoints. When you’re ready to deploy a model, you specify which checkpoints to use. The model_name you assign during deployment (using DeployCheckpointsConfig) becomes the identifier for the model version those checkpoints produce. To see the available checkpoints for a job, use the Training API.

​
GPU capacity

Baseten sets a concurrent GPU limit for each GPU type in your organization. Run baseten train capacity describe to see each type’s limit, baseline, and current usage; per-team limits appear when any are set. By default, a job runs on dedicated capacity. The job holds its dedicated GPUs and runs to completion without interruption. A dedicated job draws first on your team’s own capacity limit, and in most configurations can run past that limit by borrowing from a shared pool, the organization capacity that no team’s limit claims. Some configurations have no pool, leaving the team limit a hard ceiling. Spot capacity is opt-in and runs on idle GPUs in your organization. Baseten can reclaim spot capacity.
Baseten turns on spot capacity per workspace. Contact support to request access.

​
Team GPU capacity limits

Organizations can set how many GPUs of each type a team can use at once. Whether that number is a guaranteed baseline or a hard ceiling depends on whether your organization has a shared pool:
  • With a shared pool, the limit is a baseline your team always keeps. Dedicated jobs can run past it by borrowing unclaimed organization capacity.
  • Without a shared pool, the limit is a hard ceiling. Once your team reaches it, further dedicated jobs queue until capacity frees up.
To view your organization’s capacity and the per-team breakdown by GPU type:
The output shows organization-wide capacity, then each team’s limit and current usage:
Baseline is how many GPUs the organization can run on dedicated capacity. Limit caps all usage combined, dedicated and spot together. On-Demand shows the dedicated GPUs in use now, and Spot shows the spot GPUs. Org admins can change a team’s limit at any time:
This sets the maximum number of concurrent H100 GPUs the ml-research team can use. You manage capacity limits through the Baseten CLI; there’s no dashboard control for them yet.

​
Shared pool

The shared pool is the part of your organization’s dedicated capacity that no team’s limit claims: the organization Baseline minus the sum of every team Limit. In the output above, a Baseline of 4 L4 GPUs and a single team limit of 3 leave a pool of 1. Dedicated jobs that exceed their team’s baseline draw from the pool, first come, first served across teams. Baseten reserves every team’s baseline before carving out the pool, so borrowing never eats into another team’s guaranteed capacity. A team without a limit has a baseline of zero and runs entirely on the pool. baseten train capacity describe doesn’t report pool size directly, so compute it from the values it prints.

​
No shared pool

Team limits are hard ceilings, with nothing to borrow, when any of these are true:
  • Team limits already add up to your organization’s dedicated capacity, leaving nothing unallocated.
  • Your organization sets no organization-wide limit, or no team limits at all.
  • Your organization has asked Baseten to disable pool borrowing.
In these configurations a dedicated job never runs past its team’s limit; it queues until capacity frees up. Spot is the only way past.

​
Spot capacity

Spot capacity runs a job on GPUs that would otherwise sit idle, wherever they are in your organization. Reach for it when you need to run more than your dedicated capacity allows, or you don’t want to wait in the queue, for example to fire off extra experiments while your dedicated GPUs are busy. Your organization’s Limit is the only gate on a spot job. Spot runs whenever total live usage across your organization, dedicated and spot in every team, leaves room under that Limit. Your team’s limit and the shared pool don’t apply. Both of these follow. If your team is at its limit with an empty pool while another team sits idle, a spot job still runs, on capacity that belongs to that team’s baseline. If your organization’s entire dedicated capacity is in use, spot keeps going into the band between Baseline and Limit. Available spot capacity is your Limit minus everything running now, which baseten train capacity describe reports as On-Demand plus Spot. In the output above, spot reaches all 8 L4 GPUs when nothing else is running, and 4 when the organization’s dedicated capacity is full. You borrow spot capacity, you don’t own it: Baseten can reclaim it the moment it needs that capacity for dedicated work. Run a job on spot capacity by adding --spot to baseten train push:
The flag overrides the availability_model field in your compute config. You can also set availability_model: SPOT in the config directly. To check which capacity a job runs on, read the Capacity Type column in baseten train job list. It reads Spot for spot jobs and On-demand for dedicated ones. The single-job view (baseten train job describe --job-id <job_id>) reports the same thing as Availability Model, using the raw spot and dedicated values. When Baseten reclaims the capacity, the job stops and moves to the terminal TRAINING_JOB_PREEMPTED state. You can’t predict or control when this happens, or which of your spot jobs stops first, so treat every spot job as interruptible. Preemption isn’t a failure, and a preempted job doesn’t resume on its own. Checkpoint your progress as the job runs, then resubmit to continue from your latest checkpoint.

​
Job admission and queueing

Baseten compares each job’s required GPU count with the applicable organization and team limits, current usage, and pending jobs.
  • If capacity is available and no pending job ranks ahead of the submission, Baseten starts the job.
  • If the job fits within the organization limit but the required capacity is unavailable, Baseten places the job in TRAINING_JOB_PENDING. The job starts when capacity becomes available and the job reaches the front of its queue.
  • If the job requires more GPUs than the organization limit, Baseten rejects the submission. The job can’t fit within the configured limit.
  • If the pending-job queue is full, Baseten rejects the submission.
For example, an organization has a limit of 16 H100 GPUs, uses 12 of them, and has no pending jobs: A job that exceeds the organization limit returns an error that identifies the requested GPU count and the applicable limit:

​
Queue priority

When a job fits within the organization limit but the required capacity is unavailable, Baseten places it in a queue with the TRAINING_JOB_PENDING state. Set --priority on baseten train push to control the order jobs leave that queue:
Higher values run first. Jobs default to priority 0. Raise a job above 0 to run it sooner, or drop it below 0 to hold it behind default-priority work; the value is an unbounded integer. Priority only reorders jobs that are waiting for capacity. It doesn’t pause a running job or add capacity, so raising priority helps only when jobs wait in the queue and capacity frees up. Set priority when you submit the job. To change a pending job’s priority without resubmitting, use baseten train job update. This works only on jobs still in the TRAINING_JOB_PENDING state: