Every AI/ML model on Baseten runs on an instance, a dedicated set of hardware allocated to the model server. Selecting the right instance type ensures optimal performance while controlling compute costs.

• Insufficient resources → Slow inference or failures.
• Excess resources → Higher costs without added benefit.

​

Instance type resource components

• Instance → The allocated hardware for inference.
• vCPU → Virtual CPU cores for general computing.
• RAM → Memory available to the CPU.
• GPU → Specialized hardware for accelerated ML workloads.
• VRAM → Dedicated GPU memory for model execution.

​

Configuring model resources

Resources can be defined before deployment in Truss or adjusted later via the Baseten UI.

​

Defining resources in Truss

Define resource requirements in config.yaml before running truss push. Any changes after deployment will not impact previous deployments. Running truss push again will create a new deployment using the resources specified in the config.yaml.

Example (Stable Diffusion XL):

resources:
  accelerator: A10G
  cpu: "4"
  memory: 16Gi
  use_gpu: true

Baseten provisions the smallest instance that meets the specified constraints:

• **cpu: “3” or “4” → **Maps to a 4-core instance.
• **cpu: “5” to “8” → **Maps to an 8-core instance.

​

Updating resources in the Baseten UI

Once deployed, resource configurations can only be updated through the Baseten UI. Changing the instance type will deploy a new copy of the deployment using the new specified instance type.

For a list of available instance types, see the instance type reference.

​

Instance Type Reference

Specs and benchmarks for every Baseten instance type.

Choosing the right instance for model inference means balancing performance and cost. This page lists all available instance types on Baseten to help you deploy and serve models effectively.

​

CPU-only Instances

Cost-effective options for lighter workloads. No GPU.

• Starts at: $0.00058/min
• Best for: Transformers pipelines, small QA models, text embeddings

Example workloads:

• 1x2: Text classification (e.g., Truss quickstart)
• 4x16: LayoutLM Document QA
• 4x16+: Sentence Transformers embeddings on larger corpora

​

GPU Instances

Accelerated inference for LLMs, diffusion models, and Whisper.

​

GPU Details & Workloads

​

T4

Turing-series GPU

• 2,560 CUDA / 320 Tensor cores
• 16 GiB VRAM
• Best for: Whisper, small LLMs like StableLM 3B

​

L4

Ada Lovelace-series GPU

• 7,680 CUDA / 240 Tensor cores
• 24 GiB VRAM, 300 GiB/s
• 24 GiB VRAM, 300 GiB/s
• 121 TFLOPS (fp16)
• Best for: Stable Diffusion XL
• Limit: Not suitable for LLMs due to bandwidth

​

A10G

Ampere-series GPU

• 9,216 CUDA / 288 Tensor cores
• 24 GiB VRAM, 600 GiB/s
• 70 TFLOPS (fp16)
• Best for: Mistral 7B, Whisper, Stable Diffusion/SDXL

​

V100

Volta-series GPU

• 16 GiB VRAM
• Best for: Legacy workloads needing V100-specific support

​

A100

Ampere-series GPU

• 6,912 CUDA / 432 Tensor cores
• 80 GiB VRAM, 1.94 TB/s
• 312 TFLOPS (fp16)
• Best for: Mixtral, Llama 2 70B (2 A100s), Falcon 180B (5 A100s), SDXL

​

H100

Hopper-series GPU

• 16,896 CUDA / 640 Tensor cores
• 80 GiB VRAM, 3.35 TB/s
• 990 TFLOPS (fp16)
• Best for: Mixtral 8x7B, Llama 2 70B (2×H100), SDXL

​

H100MIG

Fractional H100 (3/7 compute, ½ memory)

• 7,242 CUDA cores, 40 GiB VRAM
• 1.675 TB/s bandwidth
• Best for: Efficient LLM inference at lower cost than A100