> ## Documentation Index > Fetch the complete documentation index at: https://docs.baseten.co/llms.txt > Use this file to discover all available pages before exploring further. # Management > How to monitor, manage, and interact with your training projects and jobs. Once you've submitted training jobs, Baseten provides tools to manage your `TrainingProject`s and individual `TrainingJob`s. You can use the [CLI](/reference/cli/training/training-cli) or the [API](/reference/training-api/overview) to manage your jobs. ## `TrainingProject` management ### List projects To view all your training projects: ```bash theme={"system"} truss train view ``` This lists every `TrainingProject` you have access to, with its ID, name, latest job, and checkpoint sync status, followed by a table of all active jobs. ### View jobs in a project To see all jobs in a specific project, pass its `project` (which you get when you create the project, or from `truss train view`): ```bash theme={"system"} truss train view --project ``` ### Delete a project Delete a training project through the API or the dashboard. Using the API: ```bash theme={"system"} curl -X DELETE https://api.baseten.co/v1/training_projects/ \ -H "Authorization: Bearer YOUR_API_KEY" ``` **To delete a project from the Baseten dashboard**: 1. Select the training project you want to delete. 2. Type the project name (for example, `demo/qwen3-0.6b`) to confirm. 3. Choose **Delete**. Deleting a project permanently deletes the following, with no archival or recovery option: * All undeployed [checkpoints](/training/concepts/checkpoints) from every job in the project * All data in the project's [training cache](/training/concepts/cache) (`$BT_PROJECT_CACHE_DIR`) Deleting a project doesn't touch checkpoints you've already [deployed](/training/deployment). ## `TrainingJob` management After submitting a job with `truss train push config.py`, you receive a `project_id` and `job_id`. ### List jobs List all jobs within a project, with each job's ID, status, instance type, and creation time: ```bash theme={"system"} truss train view --project ``` ### Check status and retrieve logs To view the logs for the job you submitted most recently in the current context (for example, if you just pushed a job from your current terminal directory): ```bash theme={"system"} truss train logs --tail ``` To view logs for a specific job using its `job-id`: ```bash theme={"system"} truss train logs --job-id [--tail] ``` Add `--tail` to follow the logs live. Both `truss train view` and `truss train logs` show which status a job is in. For more on the job lifecycle, see [Lifecycle](/training/lifecycle). ### Stop a job To stop a running or pending job, use the `stop` command with the job's project ID and job ID: ```bash theme={"system"} truss train stop --job-id truss train stop --all # Stops all active jobs; Will prompt the user for confirmation. ``` This transitions the job to the `TRAINING_JOB_STOPPED` state. ### Delete a job Delete a training job through the API or the dashboard. Using the API: ```bash theme={"system"} curl -X DELETE https://api.baseten.co/v1/training_projects//jobs/ \ -H "Authorization: Bearer YOUR_API_KEY" ``` **To delete a job from the Baseten dashboard**: 1. Select the project containing the job. 2. Select the job you want to delete. 3. Type the job name (for example, `job-2`) to confirm. 4. Choose **Delete**. Deleting a job permanently deletes all its undeployed checkpoints, with no archival or recovery option. It doesn't touch checkpoints you've already [deployed](/training/deployment). ### Job outputs and checkpoints The primary outputs of a successful `TrainingJob` are model checkpoints, when you enable and configure checkpointing. Baseten stores them for you. For more on how `CheckpointingConfig` works, see [Checkpoints](/training/concepts/checkpoints). When you're ready to [deploy a model](/training/deployment), you specify which checkpoints to use. The `model_name` you assign during deployment (using `DeployCheckpointsConfig`) becomes the identifier for the model version those checkpoints produce. To see the available checkpoints for a job, use the [Training API](/reference/training-api/get-training-job-checkpoints). ## GPU capacity Training jobs draw on your organization's GPU capacity. Whether a job starts right away depends on your team's capacity limit, how much of your organization's capacity is already in use, and whether you run the job on spot capacity. By default, a job runs on dedicated capacity. The job holds its dedicated GPUs and runs to completion without interruption. A dedicated job draws first on your team's own capacity limit, and in most configurations can run past that limit by borrowing from a [shared pool](#shared-pool), the organization capacity that no team's limit claims. [Some configurations have no pool](#no-shared-pool), leaving the team limit a hard ceiling. When neither has room, the job waits in a queue. [Spot capacity](#spot-capacity) is opt-in and runs on whatever your organization isn't using, including another team's idle GPUs, at the cost of interruption when Baseten needs that capacity elsewhere. Baseten turns on spot capacity per workspace. [Contact support](mailto:support@baseten.co) to request access. ### Team GPU capacity limits Organizations can set how many GPUs of each type a team can use at once. Whether that number is a guaranteed baseline or a hard ceiling depends on whether your organization has a [shared pool](#shared-pool): * **With a shared pool**, the limit is a baseline your team always keeps. Dedicated jobs can run past it by borrowing unclaimed organization capacity. * **Without a shared pool**, the limit is a hard ceiling. Once your team reaches it, further dedicated jobs queue until capacity frees up. To view your organization's capacity and the per-team breakdown by GPU type: ```bash theme={"system"} truss train capacity view ``` The output shows organization-wide capacity, then each team's limit and current usage: ```text theme={"system"} Training GPU Capacity │ GPU Type │ Baseline │ Limit │ On-Demand │ Spot │ │ L4 │ 4 │ 8 │ 0 │ 0 │ Team Training GPU Capacity │ Team │ GPU Type │ Limit │ On-Demand │ Spot │ │ ml-research │ L4 │ 3 │ 0 │ 0 │ ``` **Baseline** is how many GPUs the organization can run on dedicated capacity. **Limit** caps all usage combined, dedicated and [spot](#spot-capacity) together. **On-Demand** shows the dedicated GPUs in use now, and **Spot** shows the spot GPUs. Org admins can change a team's limit at any time: ```bash theme={"system"} truss train capacity update --team ml-research --gpu-type H100 --capacity 32 ``` This sets the maximum number of concurrent H100 GPUs the `ml-research` team can use. You manage capacity limits through the [Truss CLI](/reference/cli/training/training-cli#capacity); there's no dashboard control for them yet. ### Shared pool The shared pool is the part of your organization's dedicated capacity that no team's limit claims: the organization **Baseline** minus the sum of every team **Limit**. In the output above, a Baseline of 4 L4 GPUs and a single team limit of 3 leave a pool of 1. Dedicated jobs that exceed their team's baseline draw from the pool, first come, first served across teams. Baseten reserves every team's baseline before carving out the pool, so borrowing never eats into another team's guaranteed capacity. A team without a limit has a baseline of zero and runs entirely on the pool. `truss train capacity view` doesn't report pool size directly, so compute it from the values it prints. ### No shared pool Team limits are hard ceilings, with nothing to borrow, when any of these are true: * Team limits already add up to your organization's dedicated capacity, leaving nothing unallocated. * Your organization sets no organization-wide limit, or no team limits at all. * Your organization has asked Baseten to disable pool borrowing. In these configurations a dedicated job never runs past its team's limit; it queues until capacity frees up. [Spot](#spot-capacity) is the only way past. ### Spot capacity Spot capacity runs a job on GPUs that would otherwise sit idle, wherever they are in your organization. Reach for it when you need to run more than your dedicated capacity allows, or you don't want to wait in the queue, for example to fire off extra experiments while your dedicated GPUs are busy. Your organization's **Limit** is the only gate on a spot job. Spot runs whenever total live usage across your organization, dedicated and spot in every team, leaves room under that Limit. Your team's limit and the [shared pool](#shared-pool) don't apply. Both of these follow. If your team is at its limit with an empty pool while another team sits idle, a spot job still runs, on capacity that belongs to that team's baseline. If your organization's entire dedicated capacity is in use, spot keeps going into the band between **Baseline** and **Limit**. Available spot capacity is your **Limit** minus everything running now, which [`truss train capacity view`](#team-gpu-capacity-limits) reports as **On-Demand** plus **Spot**. In the output above, spot reaches all 8 L4 GPUs when nothing else is running, and 4 when the organization's dedicated capacity is full. You borrow spot capacity, you don't own it: Baseten can reclaim it the moment it needs that capacity for dedicated work. Run a job on spot capacity by adding `--spot` to [`truss train push`](/reference/cli/training/training-cli#push): ```bash theme={"system"} truss train push config.py --spot ``` The flag overrides the `availability_model` field in your [compute config](/reference/sdk/training). You can also set `availability_model: SPOT` in the config directly. When Baseten reclaims the capacity, the job stops and moves to the terminal [`TRAINING_JOB_PREEMPTED`](/training/lifecycle) state. You can't predict or control when this happens, or which of your spot jobs stops first, so treat every spot job as interruptible. Preemption isn't a failure, and a preempted job doesn't resume on its own. [Checkpoint](/training/concepts/checkpoints) your progress as the job runs, then resubmit to continue from your latest checkpoint. ### Queue priority When capacity is full, new training jobs wait in a queue (the [`TRAINING_JOB_PENDING`](/training/lifecycle) state). Set `--priority` on `truss train push` to control the order jobs leave that queue: ```bash theme={"system"} truss train push config.py --priority 100 ``` Higher values run first. Jobs default to priority `0`. Raise a job above `0` to run it sooner, or drop it below `0` to hold it behind default-priority work; the value is an unbounded integer. Priority only reorders jobs that are waiting for capacity. It doesn't pause a running job or add capacity, so raising priority helps only when jobs wait in the queue and capacity frees up. Set priority when you submit the job. To change a pending job's priority, stop it with `truss train stop` and resubmit at the new value.