> ## Documentation Index
> Fetch the complete documentation index at: https://docs.baseten.co/llms.txt
> Use this file to discover all available pages before exploring further.

# Llama 3.2

> Meta's compact Llama 3.2 instruction-tuned model.

[Tool calling](https://docs.baseten.co/examples/models/capabilities/tool-calling)
[Long context](https://docs.baseten.co/examples/models/capabilities/long-context)

Meta's compact Llama 3.2 instruction-tuned model. Runs on a single H100 40GB for low-cost chat and edge-adjacent workloads.

## Setup

Install the Baseten CLI and sign in, then install the OpenAI SDK.

**Install and sign in to Baseten**

**macOS or Linux**

```bash Terminal theme={"system"}
brew tap basetenlabs/baseten
brew trust --formula basetenlabs/baseten/baseten
brew install baseten
```

**Windows**

Download and extract the Windows archive for the latest Baseten CLI release:

```powershell Terminal theme={"system"}
$release = Invoke-RestMethod https://api.github.com/repos/basetenlabs/baseten-cli/releases/latest
$asset = $release.assets | Where-Object { $_.name -like "*_windows_amd64.zip" }
Invoke-WebRequest $asset.browser_download_url -OutFile baseten.zip
Expand-Archive -Force baseten.zip .
```

Move `baseten.exe` to a directory on your `PATH`.

For other platforms or a specific version, see the [Baseten CLI install reference](https://docs.baseten.co/reference/cli/baseten/overview.md#install).

```sh theme={"system"}
baseten auth login
```

**Install the OpenAI SDK**

```sh theme={"system"}
uv pip install openai
```

To install without Homebrew, see the [Baseten CLI install reference](https://docs.baseten.co/reference/cli/baseten/overview.md#install).

This preset serves Llama 3.2 3B Instruct on a single H100 40GB through [Baseten Inference Stack](https://docs.baseten.co/engines/bis-llm/overview.md) (TensorRT-LLM), optimized for the lowest Llama 3.2 latency on Baseten.

> **Hardware**
>
> H100\_40GB

> **Engine**
>
> TRT-LLM

> **Context**
>
> 128K

## Write the config

Create and move into the project directory:

```sh theme={"system"}
mkdir llama-3.2-3b-instruct-latency && cd llama-3.2-3b-instruct-latency
```

Then create a file named `config.yaml` and paste the following:

```yaml config.yaml theme={"system"}
model_metadata:
  example_model_input:
    max_tokens: 512
    messages:
      - content: Tell me everything you know about optimized inference.
        role: user
    stream: true
    temperature: 0.5
  tags:
    - openai-compatible
model_name: "model:llama-3.2-3b-instruct preset:latency"
python_version: py311
resources:
  accelerator: H100_40GB
  cpu: "1"
  memory: 10Gi
  use_gpu: true
trt_llm:
  build:
    base_model: decoder
    checkpoint_repository:
      repo: meta-llama/Llama-3.2-3B-Instruct
      revision: main
      source: HF
    max_seq_len: 131072
    quantization_type: fp8_kv
    tensor_parallel_count: 1
  runtime:
    enable_chunked_context: true
```

## Key parameters

[Baseten Inference Stack](https://docs.baseten.co/engines/bis-llm/overview.md) (BIS) reads these fields from the `trt_llm` block. Each one shapes how the engine is built and served:

| Parameter | Value |
| - | - |
| Max sequence length | `131072` |
| Chunked prefill | `enabled` |
| Quantization | `fp8_kv` |
| Base model type | `decoder` |

## Deploy

Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:

```sh Baseten CLI theme={"system"}
baseten model push
```

```sh Truss CLI theme={"system"}
uvx truss push
```

You should see output similar to:

```output theme={"system"}
Pushing model "llama-3.2-3b-instruct-latency"...
Uploading model...
Uploaded model in 0s
✨ Model llama-3.2-3b-instruct-latency was successfully pushed ✨

  Model:       llama-3.2-3b-instruct-latency (abc1d2ef)
  Deployment:  xyz123
  Environment: production

🪵 View logs:
   deployment:   baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
   environment:  baseten model environment logs --model-id abc1d2ef --environment production  (once deployed)
   app:          https://app.baseten.co/models/abc1d2ef/logs/xyz123

🚀 Invoke your model:
   URL:  https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
   CLI:  baseten model predict --model-id abc1d2ef
```

`baseten model push` prints your **model ID** (`abc1d2ef` in the example). The examples below use it wherever you see `{model_id}`, and read your API key from the `BASETEN_API_KEY` environment variable.

## Call the model

Your deployment serves an OpenAI-compatible API.

Now call your deployment to run inference:

**Python**

```python main.py theme={"system"}
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["BASETEN_API_KEY"],
    base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)

response = client.chat.completions.create(
    model="",
    messages=[
        {"role": "user", "content": "What is machine learning?"}
    ],
)

print(response.choices[0].message.content)
```

**cURL**

```sh theme={"system"}
curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '{
    "model": "",
    "messages": [
      {"role": "user", "content": "What is machine learning?"}
    ]
  }'
```

## Next steps

> **[Call your model](https://docs.baseten.co/inference/calling-your-model.md)**
>
> Endpoint anatomy, authentication, and sync versus async inference

> **[Autoscaling](https://docs.baseten.co/deployment/autoscaling/overview.md)**
>
> Scale replicas with traffic, including scale to zero