Skip to main content
Once you’ve submitted training jobs, Baseten provides tools to manage your TrainingProjects and individual TrainingJobs. You can use the CLI or the API to manage your jobs.

TrainingProject management

List projects

To view all your training projects:
This lists every TrainingProject you have access to, with its ID, name, latest job, and checkpoint sync status, followed by a table of all active jobs.

View jobs in a project

To see all jobs in a specific project, pass its project (which you get when you create the project, or from baseten train project list):

Delete a project

Delete a training project through the API or the dashboard. Using the API:
To delete a project from the Baseten dashboard:
  1. Select the training project you want to delete.
  2. Type the project name (for example, demo/qwen3-0.6b) to confirm.
  3. Choose Delete.
Deleting a project permanently deletes the following, with no archival or recovery option:
  • All undeployed checkpoints from every job in the project
  • All data in the project’s training cache ($BT_PROJECT_CACHE_DIR)
Deleting a project doesn’t touch checkpoints you’ve already deployed.

TrainingJob management

After submitting a job with baseten train push --config config.py, you receive a project_id and job_id.

List jobs

List all jobs within a project, with each job’s ID, status, instance type, capacity type, and creation time:

Check status and retrieve logs

Find the job ID, then stream its logs:
Add --tail to follow the logs live. Both baseten train job list and baseten train job logs show which status a job is in. For more on the job lifecycle, see Lifecycle.

Stop a job

To stop a running or pending job, use the stop command with the job’s project ID and job ID:
This transitions the job to the TRAINING_JOB_STOPPED state.

Delete a job

Delete a training job through the API or the dashboard. Using the API:
To delete a job from the Baseten dashboard:
  1. Select the project containing the job.
  2. Select the job you want to delete.
  3. Type the job name (for example, job-2) to confirm.
  4. Choose Delete.
Deleting a job permanently deletes all its undeployed checkpoints, with no archival or recovery option. It doesn’t touch checkpoints you’ve already deployed.

Job outputs and checkpoints

The primary outputs of a successful TrainingJob are model checkpoints, when you enable and configure checkpointing. Baseten stores them for you. For more on how CheckpointingConfig works, see Checkpoints. When you’re ready to deploy a model, you specify which checkpoints to use. The model_name you assign during deployment (using DeployCheckpointsConfig) becomes the identifier for the model version those checkpoints produce. To see the available checkpoints for a job, use the Training API.

GPU capacity

Baseten sets a GPU quota for each GPU type in your organization: the most GPUs of that type your jobs can use at once. Run baseten train capacity describe to see each type’s quotas and current usage; per-team quotas appear when any are set. A GPU quota controls how much capacity your organization can use; it doesn’t reserve hardware. A job also needs available GPUs on a cluster eligible for its configuration. Contact support to raise an organization quota or enable a GPU type. By default, a job runs on dedicated capacity. The job holds its dedicated GPUs and runs to completion without interruption. A dedicated job draws first on your team’s own quota, and in most configurations can run past that quota by borrowing from a shared pool, the organization capacity that no team’s quota claims. Some configurations have no pool, leaving the team quota a hard ceiling. Spot capacity is opt-in and runs on idle GPUs in your organization. Baseten can reclaim spot capacity.
Baseten turns on spot capacity per workspace. Contact support to request access.

Team GPU quotas

Organizations can set a GPU quota for each team: how many GPUs of each type the team can use at once. Whether that quota is guaranteed or a hard ceiling depends on whether your organization has a shared pool:
  • With a shared pool, your team always keeps its quota. Dedicated jobs can run past it by borrowing unclaimed organization capacity.
  • Without a shared pool, the quota is a hard ceiling. Once your team reaches it, further dedicated jobs queue until capacity frees up.
To view your organization’s capacity and the per-team breakdown by GPU type:
The output shows organization-wide quotas and usage, then each team’s:
IN USE counts the GPUs running dedicated and spot jobs. In the organization table, BASELINE is the on-demand quota: how many GPUs the organization can run on dedicated capacity. LIMIT is the spot quota, which covers dedicated and spot usage together, so spot can use whatever dedicated jobs leave free under it. When BASELINE is 0, dedicated jobs can use up to LIMIT. In the team table, LIMIT is the team’s quota. Org admins can change a team’s quota at any time:
This sets the ml-research team’s H100 quota to 32 GPUs. You manage quotas through the Baseten CLI; there’s no dashboard control for them yet.

Shared pool

The shared pool is the part of your organization’s dedicated capacity that no team’s quota claims: the organization’s on-demand quota minus the sum of every team’s quota. In the output above, an on-demand quota of 4 L4 GPUs and a single team quota of 3 leave a pool of 1. Dedicated jobs that exceed their team’s quota draw from the pool, first come, first served across teams. Baseten reserves every team’s quota before carving out the pool, so borrowing never eats into another team’s guaranteed capacity. A team without a quota runs entirely on the pool. baseten train capacity describe doesn’t report pool size directly, so compute it from the values it prints.

No shared pool

Team quotas are hard ceilings, with nothing to borrow, when any of these are true:
  • Team quotas already add up to your organization’s dedicated capacity, leaving nothing unallocated.
  • Your organization sets no organization-wide quota, or no team quotas at all.
  • Your organization has asked Baseten to disable pool borrowing.
In these configurations a dedicated job never runs past its team’s quota; it queues until capacity frees up. Spot is the only way past.

Spot capacity

Spot capacity runs a job on GPUs that would otherwise sit idle, wherever they are in your organization. Reach for it when you need to run more than your dedicated capacity allows, or you don’t want to wait in the queue, for example to fire off extra experiments while your dedicated GPUs are busy. Spot jobs must fit within your organization’s spot quota, counting dedicated and spot usage across every team. They also need available GPUs on an eligible cluster. Your team’s quota and the shared pool don’t apply. Both of these follow. If your team is at its quota with an empty pool while another team sits idle, a spot job still runs, on capacity that belongs to that team’s quota. If your organization’s entire dedicated capacity is in use, spot keeps going into the band between the on-demand quota and the spot quota. Available spot capacity is your spot quota minus everything running now, on-demand and spot. In the output above, 3 of 8 L4 GPUs are in use, so spot can add 5 more. Spot reaches all 8 when nothing else is running, and 4 when the organization’s on-demand capacity is full. You borrow spot capacity, you don’t own it: Baseten can reclaim it the moment it needs that capacity for dedicated work. Run a job on spot capacity by adding --spot to baseten train push:
The flag overrides the availability_model field in your compute config. You can also set availability_model: SPOT in the config directly. To check which capacity a job runs on, read the Capacity Type column in baseten train job list. It reads Spot for spot jobs and On-demand for dedicated ones. The single-job view (baseten train job describe --job-id <job_id>) reports the same thing as Availability Model, using the raw spot and dedicated values. When Baseten reclaims the capacity, the job stops and moves to the terminal TRAINING_JOB_PREEMPTED state. You can’t predict or control when this happens, or which of your spot jobs stops first, so treat every spot job as interruptible. Preemption isn’t a failure, and a preempted job doesn’t resume on its own. Checkpoint your progress as the job runs, then resubmit to continue from your latest checkpoint.

Job admission and queueing

Baseten compares each job’s required GPU count with the applicable organization and team quotas, current usage, and pending jobs.
  • If no eligible cluster has the required GPUs, Baseten rejects the submission with HTTP 429, even when queueing is enabled. Queueing doesn’t handle cluster capacity shortages.
  • If the required GPUs are available, the job fits within the applicable GPU quotas, and no pending job ranks ahead of the submission, Baseten starts the job.
  • If running jobs leave too little room under your organization’s GPU quota, Baseten places the job in TRAINING_JOB_PENDING when queueing is enabled for your organization. The job starts when quota becomes available and the job reaches the front of its queue.
  • If the job requires more GPUs than the organization quota, Baseten rejects the submission. The job can’t fit within the configured quota.
  • If a job must wait for organization GPU quota, Baseten rejects the submission when queueing is disabled or the pending-job queue is full. Wait for running jobs to release quota before resubmitting, or contact support about queueing and GPU quotas for your workload.
For example, an organization has a quota of 16 H100 GPUs, uses 12 of them, and has no pending jobs. Queueing is enabled, and the cluster has enough available GPUs: A job that exceeds the organization quota returns an error that identifies the requested GPU count and the applicable quota:

Resolve capacity errors

Read the submission error to choose whether to reduce the job’s GPU request, wait for running work to finish, or request a higher quota: A pending job has been accepted; it doesn’t need another submission. For a spot job that reaches TRAINING_JOB_PREEMPTED, resubmit from a checkpoint instead of waiting for it to restart.

Queue priority

When queueing is enabled, jobs waiting for organization GPU quota use the TRAINING_JOB_PENDING state. Set --priority on baseten train push to control the order jobs leave that queue:
Higher values run first. Jobs default to priority 0. Raise a job above 0 to run it sooner, or drop it below 0 to hold it behind default-priority work; the value is an unbounded integer. Priority only reorders jobs that are waiting for organization GPU quota. It doesn’t pause a running job or add cluster capacity, so raising priority helps only when jobs wait in the queue and quota becomes available. Set priority when you submit the job. To change a pending job’s priority without resubmitting, use baseten train job update. This works only on jobs still in the TRAINING_JOB_PENDING state: