Llama 3.2

Tool calling Long context

Meta’s compact Llama 3.2 instruction-tuned model. Runs on a single H100 40GB for low-cost chat and edge-adjacent workloads.

Setup

Install the Baseten CLI and sign in, then install the OpenAI SDK.

Install and sign in to Baseten

macOS or Linux
Windows

Terminal

brew tap basetenlabs/baseten
brew trust --formula basetenlabs/baseten/baseten
brew install baseten

Download and extract the Windows archive for the latest Baseten CLI release:

Terminal

$release = Invoke-RestMethod https://api.github.com/repos/basetenlabs/baseten-cli/releases/latest
$asset = $release.assets | Where-Object { $_.name -like "*_windows_amd64.zip" }
Invoke-WebRequest $asset.browser_download_url -OutFile baseten.zip
Expand-Archive -Force baseten.zip .

Move baseten.exe to a directory on your PATH.

For other platforms or a specific version, see the Baseten CLI install reference.

baseten auth login

Install the OpenAI SDK

uv pip install openai

To install without Homebrew, see the Baseten CLI install reference. This preset serves Llama 3.2 3B Instruct on a single H100 40GB through Baseten Inference Stack (TensorRT-LLM), optimized for the lowest Llama 3.2 latency on Baseten.

Hardware

H100_40GB

Engine

TRT-LLM

Context

128K

Write the config

Create and move into the project directory:

mkdir llama-3.2-3b-instruct-latency && cd llama-3.2-3b-instruct-latency

Then create a file named config.yaml and paste the following:

config.yaml

model_metadata:
  example_model_input:
    max_tokens: 512
    messages:
      - content: Tell me everything you know about optimized inference.
        role: user
    stream: true
    temperature: 0.5
  tags:
    - openai-compatible
model_name: "model:llama-3.2-3b-instruct preset:latency"
python_version: py311
resources:
  accelerator: H100_40GB
  cpu: "1"
  memory: 10Gi
  use_gpu: true
trt_llm:
  build:
    base_model: decoder
    checkpoint_repository:
      repo: meta-llama/Llama-3.2-3B-Instruct
      revision: main
      source: HF
    max_seq_len: 131072
    quantization_type: fp8_kv
    tensor_parallel_count: 1
  runtime:
    enable_chunked_context: true

Key parameters

Baseten Inference Stack (BIS) reads these fields from the trt_llm block. Each one shapes how the engine is built and served:

Parameter	Value
Max sequence length	`131072`
Chunked prefill	`enabled`
Quantization	`fp8_kv`
Base model type	`decoder`

Deploy

Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:

baseten model push

uvx truss push

You should see output similar to:

Pushing model "llama-3.2-3b-instruct-latency"...
Uploading model...
Uploaded model in 0s
✨ Model llama-3.2-3b-instruct-latency was successfully pushed ✨

  Model:       llama-3.2-3b-instruct-latency (abc1d2ef)
  Deployment:  xyz123
  Environment: production

🪵 View logs:
   deployment:   baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
   environment:  baseten model environment logs --model-id abc1d2ef --environment production  (once deployed)
   app:          https://app.baseten.co/models/abc1d2ef/logs/xyz123

🚀 Invoke your model:
   URL:  https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
   CLI:  baseten model predict --model-id abc1d2ef

baseten model push prints your model ID (abc1d2ef in the example). The examples below use it wherever you see {model_id}, and read your API key from the BASETEN_API_KEY environment variable.

Call the model

Your deployment serves an OpenAI-compatible API. Now call your deployment to run inference:

Python
cURL

main.py

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["BASETEN_API_KEY"],
    base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)

response = client.chat.completions.create(
    model="",
    messages=[
        {"role": "user", "content": "What is machine learning?"}
    ],
)

print(response.choices[0].message.content)

curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '{
    "model": "",
    "messages": [
      {"role": "user", "content": "What is machine learning?"}
    ]
  }'

Examples

Models

Engines

Custom Docker servers

Custom Python models

Chains

API workflows

Baseten for Model Labs

Setup

Hardware

Engine

Context

Write the config

Key parameters

Deploy

Call the model

Next steps

Call your model

Autoscaling

​Setup

Hardware

Engine

Context

​Write the config

​Key parameters

​Deploy

​Call the model

​Next steps

Call your model

Autoscaling

Setup

Write the config

Key parameters

Deploy

Call the model

Next steps