Skip to main content
Every AI/ML model on Baseten runs on an instance, a dedicated set of hardware allocated to the model server. Selecting the right instance type ensures optimal performance while controlling compute costs.
  • Insufficient resources: Slow inference or failures.
  • Excess resources: Higher costs without added benefit.

​
Instance type resource components

  • Instance: The allocated hardware for inference.
  • Node: The compute unit within an instance, comprising 8 GPUs with associated vCPU, RAM, and VRAM.
  • vCPU: Virtual CPU cores for general computing.
  • RAM: Memory available to the CPU.
  • GPU: Specialized hardware for accelerated ML workloads.
  • VRAM: Dedicated GPU memory for model execution.

​
Configure model resources

Define resources before deployment in Truss or adjust them later through the Baseten UI.

​
Define resources in Truss

Define resource requirements in config.yaml before running truss push.
  • Published deployment (truss push): Creates a new deployment (named sequentially: deployment-1, deployment-2, and so on) using the resources in config.yaml.
  • Development deployment (truss push --watch): Overwrites the existing development deployment with the specified resource configuration and starts watching for changes. Use truss watch to resume watching an existing development deployment.
  • Production deployment (truss push --promote): Creates a new deployment and promotes it to production, replacing the active deployment.
  • Environment deployment (truss push --environment <name>): Deploys directly to a custom environment like staging.
Changes to config.yaml only affect new deployments. To update resources on an existing published deployment, edit resources in the Baseten UI.
You can configure resources in two ways:

​
Individual resource fields

config.yaml
Baseten provisions the smallest instance that meets the specified constraints:
  • cpu: “3” or “4” → Maps to a 4-core instance.
  • cpu: “5” to “8” → Maps to an 8-core instance.
Gi in resources.memory refers to Gibibytes, which are slightly larger than Gigabytes.

​
Exact instance type

An instance type is the full SKU name that uniquely identifies a specific hardware configuration. When you specify individual resource fields like cpu and accelerator, Baseten selects the smallest instance that meets your requirements. With instance_type, you specify exactly which instance you want, no guessing required. Use instance_type when you:
  • Know the exact hardware configuration you need.
  • Want to ensure consistent instance selection across deployments.
  • Are following a recommendation for a specific model (for example, “use an L4 with 4 vCPUs and 16 GiB RAM”).
config.yaml
The format encodes the hardware specs. For example, L4:4x16 means an L4 GPU with 4 vCPUs and 16 GiB of RAM. Naming conventions vary by GPU family, so copy the exact instance type from the instance type reference. When instance_type is specified, other resource fields (cpu, memory, accelerator, use_gpu) are ignored.

​
Update resources in the Baseten UI

Once deployed, you can only update resource configurations through the Baseten UI. Changing the instance type deploys a copy of the deployment using the specified instance type. For a list of available instance types, see the instance type reference.

​
Instance type reference

Specs and benchmarks for every Baseten instance type.

​
CPU-only instances

Cost-effective options for lighter workloads. No GPU.
  • Starts at: $0.00058/min
  • Best for: Transformers pipelines, small QA models, text embeddings
To select a CPU-only instance, use the bare <vCPU>x<MEMORY> SKU (for example, instance_type: "4x16"). Example workloads:
  • 1x2: Text classification (for example, Truss quickstart)
  • 4x16: LayoutLM Document QA
  • 4x16+: Sentence Transformers embeddings on larger corpora

​
GPU instances

Accelerated inference for LLMs, diffusion models, and Whisper. To select a GPU instance with instance_type:
  • Single L4 or A100: <GPU>:<vCPU>x<MEMORY> (for example, "L4:4x16").
  • Single T4 or A10G: <GPU>x<vCPU>x<MEMORY>, with no colon (for example, "T4x4x16", "A10Gx8x32").
  • Multi-GPU: <GPU>:<COUNT>x<vCPU>x<MEMORY> (for example, "A100:2x24x288").
  • H100/H200/B200/RTX-PRO-6000: <GPU> or <GPU>:<COUNT> (for example, "H100:2", "RTX-PRO-6000:4").
  • Fractional H100: "H100MIG".
Naming is not uniform across GPU families, so copy the exact SKU from the tables above.

​
GPU details and workloads

​
T4

Turing-series GPU
  • 2,560 CUDA / 320 Tensor cores
  • 16 GiB VRAM
  • Best for: Whisper, small LLMs like StableLM 3B

​
L4

Ada Lovelace-series GPU
  • 7,680 CUDA / 240 Tensor cores
  • 24 GiB VRAM, 300 GiB/s
  • 121 TFLOPS (fp16)
  • Best for: Stable Diffusion XL
  • Limit: Not suitable for LLMs due to bandwidth

​
A10G

Ampere-series GPU
  • 9,216 CUDA / 288 Tensor cores
  • 24 GiB VRAM, 600 GiB/s
  • 70 TFLOPS (fp16)
  • Best for: Mistral 7B, Whisper, Stable Diffusion/SDXL

​
A100

Ampere-series GPU
  • 6,912 CUDA / 432 Tensor cores
  • 80 GiB VRAM, 1.94 TB/s
  • 312 TFLOPS (fp16)
  • Best for: Mixtral, Llama 2 70B (2 A100s), Falcon 180B (5 A100s), SDXL

​
H100

Hopper-series GPU
  • 16,896 CUDA / 640 Tensor cores
  • 80 GiB VRAM, 3.35 TB/s
  • 990 TFLOPS (fp16)
  • Best for: Mixtral 8x7B, Llama 2 70B (2xH100), SDXL

​
H100MIG

Fractional H100 (3/7 compute, ½ memory)
  • 7,242 CUDA cores, 40 GiB VRAM
  • 1.675 TB/s bandwidth
  • Best for: Efficient LLM inference at lower cost than A100

​
RTX Pro 6000

Blackwell-series GPU
  • 96 GiB VRAM
  • Best for: vision-language models and mid-size LLMs at lower cost than a datacenter GPU