Skip to main content
Every deployed model runs on an instance, a dedicated set of CPU, memory, and optional GPU resources. Choose an instance that has enough memory and compute for the model without provisioning unused capacity.
  • Insufficient resources can cause slow inference, out-of-memory errors, or failed deployments.
  • Excess resources increase cost without improving performance.
Baseten dashboard showing CPU, memory, and GPU resource settings for a deployment

Instance type resource components

  • Instance: The hardware allocated to an inference replica.
  • Node: A compute server that provides the instance’s CPU, RAM, GPUs, and VRAM.
  • vCPU: Virtual CPU cores for general computation.
  • RAM: Memory available to the CPU.
  • GPU: Hardware for accelerated model execution.
  • VRAM: Memory available to the GPU.

Configure model resources

Define resources in config.yaml before deployment, or update a published deployment through the Baseten dashboard.

Define resources in Truss

Define resource requirements in config.yaml before running baseten model push. The push creates one of two deployment types:
  • Published deployment: baseten model push creates a published deployment with the resources in config.yaml.
  • Development deployment: baseten model push --watch replaces the development deployment with the specified resources and watches for changes. Use baseten model watch to resume watching it.
You can also push straight to a custom environment, such as staging, with baseten model push --environment <name>. To promote an existing deployment to production, use baseten model deployment promote. The Truss CLI combines both steps in truss push --promote.
Changes to config.yaml only affect new deployments. To update resources on an existing published deployment, edit resources in the Baseten dashboard.
You can configure resources in two ways:

Individual resource fields

config.yaml
Baseten provisions the smallest instance that meets the specified constraints:
  • cpu: "3" or cpu: "4" maps to a 4-core instance.
  • Values from cpu: "5" through cpu: "8" map to an 8-core instance.
Gi in resources.memory means gibibytes rather than gigabytes.

Exact instance type

An instance type is the SKU for a specific hardware configuration. When you set individual fields such as cpu and accelerator, Baseten selects the smallest matching instance. Set instance_type to choose an exact SKU. Use instance_type when you:
  • Know the exact hardware configuration you need.
  • Want to ensure consistent instance selection across deployments.
  • Are following a recommendation for a specific model (for example, “use an L4 with 4 vCPUs and 16 GiB RAM”).
config.yaml
The format encodes the hardware specs. For example, L4:4x16 means an L4 GPU with 4 vCPUs and 16 GiB of RAM. Naming conventions vary by GPU family, so copy the exact instance type from the instance type reference. When instance_type is specified, other resource fields (cpu, memory, accelerator, use_gpu) are ignored.

Update resources in the Baseten dashboard

Update resources on an existing published deployment through the Baseten dashboard. Changing the instance type creates a copy of the deployment on the selected hardware. For a list of available instance types, see the instance type reference.

Multi-node deployments

For models that don’t fit on a single node, set node_count in resources to provision multiple identical nodes for one deployment. Each node gets the resources you specify, and Baseten connects the nodes with a high-speed fabric for inter-node communication.
config.yaml
To constrain which fabric Baseten schedules the deployment on, set resources.fabric. Use use_rdma: true to require any RDMA-capable fabric, or preferences to list acceptable fabrics in preference order:
config.yaml
Supported fabrics are infiniband and roce. Set either use_rdma or preferences, not both. See fabric for the full field reference. Multi-node inference is typically used with TensorRT-LLM’s v2 inference stack, which supports MoE and multi-node setups. Set trt_llm.inference_stack: v2 in config.yaml when compiling the engine for a multi-node deployment.

Instance type reference

Compare specifications and list prices for the instance types Baseten offers. Your workspace’s instance types and prices depend on your plan, so list the instance types available to your workspace before you choose one. To request an instance type that your workspace doesn’t list, Contact us.

CPU-only instances

CPU-only instances suit workloads that don’t require GPU acceleration, such as Transformers pipelines, small QA models, and text embeddings. To select a CPU-only instance, use the bare <vCPU>x<MEMORY> SKU (for example, instance_type: "4x16"). Example workloads:
  • 1x2: Text classification (for example, Truss quickstart)
  • 4x16: LayoutLM Document QA
  • 4x16+: Sentence Transformers embeddings on larger corpora

GPU instances

GPU instances suit any inference workload that benefits from GPU acceleration, whatever the model architecture or serving engine. To deploy on an instance, copy its Instance value into resources.instance_type. VRAM is the total across all GPUs: A100:2x24x288 has two 80 GiB A100s, so it shows 160 GiB. For memory per GPU, see GPU details and workloads.
To select a GPU instance with instance_type:
  • Single L4 or A100: <GPU>:<vCPU>x<MEMORY> (for example, "L4:4x16").
  • Single T4 or A10G: <GPU>x<vCPU>x<MEMORY>, with no colon (for example, "T4x4x16", "A10Gx8x32").
  • Multi-GPU: <GPU>:<COUNT>x<vCPU>x<MEMORY> (for example, "A100:2x24x288").
  • H100/H200/B200/RTX-PRO-6000: <GPU> or <GPU>:<COUNT> (for example, "H100:2", "RTX-PRO-6000:4").
  • Fractional H100: "H100MIG".

List the instance types available to your workspace

The Management API returns the instance types your workspace can deploy on and the price your workspace pays for each. Use an instance type’s id as the instance_type value in config.yaml.
To list your workspace’s instance types and prices:Run the following command:
The response contains one entry per instance type, with price in USD per minute:
To list instance types without prices, request /v1/instance_types. See the instance type prices reference for every field.

GPU details and workloads

T4

Turing-series GPU
  • 2,560 CUDA / 320 Tensor cores
  • 16 GiB VRAM
  • Best for: small speech and vision models, such as Kokoro text-to-speech

L4

Ada Lovelace-series GPU
  • 7,680 CUDA / 240 Tensor cores
  • 24 GiB VRAM, 300 GiB/s
  • 121 TFLOPS (fp16)
  • Best for: small LLMs and embedding models, such as Gemma 4 E2B and Qwen3 Embedding 0.6B

A10G

Ampere-series GPU
  • 9,216 CUDA / 288 Tensor cores
  • 24 GiB VRAM, 600 GiB/s
  • 70 TFLOPS (fp16)
  • Best for: 7B-class LLMs, such as Mistral 7B

A100

Ampere-series GPU
  • 6,912 CUDA / 432 Tensor cores
  • 80 GiB VRAM, 1.94 TB/s
  • 312 TFLOPS (fp16)
  • Best for: models that need 80 GiB of VRAM per GPU

H100

Hopper-series GPU

H100MIG

Fractional H100 (3/7 compute, ½ memory)

H200

Hopper-series GPU

B200

Blackwell-series GPU

RTX Pro 6000

Blackwell-series GPU

Sandbox resources

Sandboxes don’t use instance types. Set a sandbox’s memory when you create it, and Baseten gives it the matching number of vCPUs. To set memory, choose a custom vCPU count, or raise your account’s per-sandbox limits, see Configure sandboxes.

Sandbox prices

Baseten bills an active sandbox for its vCPUs and memory. On standby, you pay only for storage. Snapshots, volumes, and images are billed for the storage they use.
Each organization can run up to 10 sandboxes at once. To raise this limit, contact support.
For example, a sandbox with 4,096 MiB of memory and 2 vCPUs costs $0.2154 per hour: 2 × $0.0655 for vCPUs plus 4 × $0.0211 for 4 GB of memory. See Baseten pricing for current rates.