> ## Documentation Index
> Fetch the complete documentation index at: https://docs.baseten.co/llms.txt
> Use this file to discover all available pages before exploring further.

# Llama 4

> Meta's Llama 4 Scout is a 17B-active MoE with native multimodal support and a 10M token context window.

<div className="capability-pills">
  <a href="/examples/models/capabilities/tool-calling" className="capability-pill">Tool calling</a>
  <a href="/examples/models/capabilities/long-context" className="capability-pill">Long context</a>
  <a href="/examples/models/capabilities/multimodal-image" className="capability-pill">Multimodal (image)</a>
</div>

Meta's Llama 4 Scout is a 17B-active MoE with native multimodal support and a 10M token context window.

## Setup

Install the Baseten CLI and sign in, then install the OpenAI SDK.

<Columns cols={2}>
  <Column>
    **Install and sign in to Baseten**

    <Tabs>
      <Tab title="macOS or Linux">
        ```bash Terminal theme={"system"}
        brew tap basetenlabs/baseten
        brew trust --formula basetenlabs/baseten/baseten
        brew install baseten
        ```
      </Tab>

      <Tab title="Windows">
        Download and extract the Windows archive for the latest Baseten CLI release:

        ```powershell Terminal theme={"system"}
        $release = Invoke-RestMethod https://api.github.com/repos/basetenlabs/baseten-cli/releases/latest
        $asset = $release.assets | Where-Object { $_.name -like "*_windows_amd64.zip" }
        Invoke-WebRequest $asset.browser_download_url -OutFile baseten.zip
        Expand-Archive -Force baseten.zip .
        ```

        Move `baseten.exe` to a directory on your `PATH`.
      </Tab>
    </Tabs>

    For other platforms or a specific version, see the [Baseten CLI install reference](/reference/cli/baseten/overview#install).

    ```sh theme={"system"}
    baseten auth login
    ```
  </Column>

  <Column>
    **Install the OpenAI SDK**

    ```sh theme={"system"}
    uv pip install openai
    ```
  </Column>
</Columns>

To install without Homebrew, see the [Baseten CLI install reference](/reference/cli/baseten/overview#install).

This preset serves Llama 4 Scout on H100:4 with a 128K serving context and native multimodal support.

<CardGroup cols={4}>
  <Card title="Hardware" icon="microchip">H100 × 4</Card>
  <Card title="Engine" icon="server">vLLM (0.22.0-cu129 build)</Card>
  <Card title="Context" icon="ruler-horizontal">128K</Card>
  <Card title="Concurrency" icon="layer-group">256</Card>
</CardGroup>

## Write the config

Create and move into the project directory:

```sh theme={"system"}
mkdir llama-4-scout-latency && cd llama-4-scout-latency
```

Then create a file named `config.yaml` and paste the following:

```yaml config.yaml theme={"system"}
model_name: "model:llama-4-scout preset:latency"
model_metadata:
  description: >-
    Llama 4 Scout 17B multimodal instruct (RedHat FP8-dynamic), long-context with TP=4 FP8 KV,
    OpenAI-compatible chat using vLLM.
  repo_id: RedHatAI/Llama-4-Scout-17B-16E-Instruct-FP8-dynamic
  example_model_input:
    model: llama
    messages:
      - role: user
        content: "Given an array of integers nums and an integer target, return indices of the two numbers such that they add up to target. You may assume that each input would have exactly one solution, and you may not use the same element twice. You can return the answer in any order. class Solution: def twoSum(self, nums: List[int], target: int) -> List[int]:"
    stream: true
    max_tokens: 512
    temperature: 0.5
  tags:
    - openai-compatible
base_image:
  image: vllm/vllm-openai:v0.22.0-cu129
weights:
  - source: "hf://RedHatAI/Llama-4-Scout-17B-16E-Instruct-FP8-dynamic@fc1b4f4cea92cfc13570f504a209eb4d4ca81534"
    mount_location: "/app/checkpoint/model"
environment_variables:
  VLLM_LOGGING_LEVEL: WARNING
  VLLM_ENGINE_READY_TIMEOUT_S: "3600"
docker_server:
  start_command: >-
    sh -c 'GPU_COUNT=$(nvidia-smi --list-gpus | wc -l) && vllm serve /app/checkpoint/model
    --served-model-name llama
    --host 0.0.0.0
    --port 8000
    --trust-remote-code
    --max-model-len 131072
    --tensor-parallel-size $GPU_COUNT
    --distributed-executor-backend mp
    --gpu-memory-utilization 0.95
    --kv-cache-dtype fp8
    --limit-mm-per-prompt.image 10
    --override-generation-config.attn_temperature_tuning true
    --enable-prefix-caching'
    --load-format runai_streamer
  readiness_endpoint: /health
  liveness_endpoint: /health
  predict_endpoint: /v1/chat/completions
  server_port: 8000
resources:
  accelerator: H100:4
  use_gpu: true
runtime:
  predict_concurrency: 256
  health_checks:
    startup_threshold_seconds: 1800
    restart_threshold_seconds: 1200
    stop_traffic_threshold_seconds: 120
```

## Flags

The `start_command` passes these flags to the engine. Each one controls a runtime or serving behavior:

| Flag | Value | What it does |
| - | - | - |
| `--trust-remote-code` | (no value) | Execute model-specific Python from the checkpoint (required for many Qwen, Phi, and custom architectures). |
| `--max-model-len` | `131072` | Maximum context length (tokens) the server accepts per request. |
| `--tensor-parallel-size` | `$GPU_COUNT` | Number of GPUs to shard the model across. |
| `--distributed-executor-backend` | `mp` | How vLLM coordinates tensor-parallel workers across processes. **mp:** Python multiprocessing (single-node default). |
| `--gpu-memory-utilization` | `0.95` | Fraction of GPU memory vLLM may use for weights and KV cache. |
| `--kv-cache-dtype` | `fp8` | KV cache numeric precision. **fp8:** \~2× KV cache density with negligible quality impact on most models. |
| `--limit-mm-per-prompt.image` | `10` | Maximum number of image inputs per prompt. |
| `--override-generation-config.attn_temperature_tuning` | `true` | Sets the `attn_temperature_tuning` field in the model's generation config. |
| `--enable-prefix-caching` | (no value) | Reuse KV cache across requests that share a prefix. |

## Deploy

Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:

<CodeGroup>
  ```sh Baseten CLI theme={"system"}
  baseten model push
  ```

  ```sh Truss CLI theme={"system"}
  uvx truss push
  ```
</CodeGroup>

You should see output similar to:

```output theme={"system"}
Pushing model "llama-4-scout-latency"...
Uploading model...
Uploaded model in 0s
✨ Model llama-4-scout-latency was successfully pushed ✨

  Model:       llama-4-scout-latency (abc1d2ef)
  Deployment:  xyz123
  Environment: production

🪵 View logs:
   deployment:   baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
   environment:  baseten model environment logs --model-id abc1d2ef --environment production  (once deployed)
   app:          https://app.baseten.co/models/abc1d2ef/logs/xyz123

🚀 Invoke your model:
   URL:  https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
   CLI:  baseten model predict --model-id abc1d2ef
```

`baseten model push` prints your **model ID** (`abc1d2ef` in the example). The examples below use it wherever you see `{model_id}`, and read your API key from the `BASETEN_API_KEY` environment variable.

## Call the model

Your deployment serves an OpenAI-compatible API.

Now call your deployment to run inference:

<Tabs>
  <Tab title="Python">
    ```python main.py theme={"system"}
    import os
    from openai import OpenAI

    client = OpenAI(
        api_key=os.environ["BASETEN_API_KEY"],
        base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
    )

    response = client.chat.completions.create(
        model="llama",
        messages=[
            {"role": "user", "content": "What is machine learning?"}
        ],
    )

    print(response.choices[0].message.content)
    ```
  </Tab>

  <Tab title="cURL">
    ```sh theme={"system"}
    curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
      -H "Content-Type: application/json" \
      -H "Authorization: Bearer $BASETEN_API_KEY" \
      -d '{
        "model": "llama",
        "messages": [
          {"role": "user", "content": "What is machine learning?"}
        ]
      }'
    ```
  </Tab>
</Tabs>

## Next steps

<CardGroup cols={2}>
  <Card title="Call your model" icon="code" href="/inference/calling-your-model">
    Endpoint anatomy, authentication, and sync versus async inference
  </Card>

  <Card title="Autoscaling" icon="arrow-up-right-dots" href="/deployment/autoscaling/overview">
    Scale replicas with traffic, including scale to zero
  </Card>
</CardGroup>