Llama 3.3

Tool calling Long context

Meta’s Llama 3.3 70B instruction-tuned model. Both presets run on H100:4 under vLLM from NVIDIA’s FP8 checkpoint: one tuned for low time-to-first-token, one for total token throughput.

Setup

Install the Baseten CLI and sign in, then install the OpenAI SDK.

Install and sign in to Baseten

macOS or Linux
Windows

Terminal

brew tap basetenlabs/baseten
brew install baseten

Download and extract the binary, then move baseten.exe to a directory on your PATH:

Terminal

Invoke-WebRequest `
  https://github.com/basetenlabs/baseten-cli/releases/download/v0.4.0/baseten_0.4.0_windows_amd64.zip `
  -OutFile baseten.zip; Expand-Archive -Force baseten.zip .

For other platforms or a specific version, see the Baseten CLI install reference.

baseten auth login

Install the OpenAI SDK

uv pip install openai

Prefer not to install? Sign in with uvx truss login --browser and deploy with uvx truss push. This variant ships in 2 presets tuned for different goals: Latency for lowest time-to-first-token, and Throughput for highest tokens per second. Pick the tab that matches your workload.

Latency
Throughput

This preset serves Llama 3.3 70B Instruct on H100:4 under vLLM with FP8 weights and tensor parallelism. It targets low time-to-first-token on the 70B chat model.

Hardware

H100 × 4

Engine

vLLM 0.26.0

Context

128K

Concurrency

128

Write the config

Create and move into the project directory:

mkdir llama-3.3-70b-instruct-latency && cd llama-3.3-70b-instruct-latency

Then create a file named config.yaml and paste the following:

config.yaml

model_name: "model:llama-3.3-70b-instruct preset:latency"

model_metadata:
  repo_id: nvidia/Llama-3.3-70B-Instruct-FP8
  tags:
    - openai-compatible
    - vllm
  example_model_input:
    stream: true
    model: nvidia/Llama-3.3-70B-Instruct-FP8
    messages:
      - role: user
        content: Tell me everything you know about optimized inference.
    max_tokens: 512
    temperature: 0.5

base_image:
  image: vllm/vllm-openai:v0.26.0

docker_server:
  start_command: >-
    vllm serve /app/model_cache/llama-3-3-70b-instruct
    --served-model-name nvidia/Llama-3.3-70B-Instruct-FP8
    --host 0.0.0.0
    --port 8000
    --tensor-parallel-size 4
    --distributed-executor-backend mp
    --max-model-len 131072
    --max-num-seqs 128
    --max-num-batched-tokens 8192
    --enable-chunked-prefill
    --enable-prefix-caching
    --gpu-memory-utilization 0.90
    --load-format runai_streamer
  readiness_endpoint: /health
  liveness_endpoint: /health
  predict_endpoint: /v1/chat/completions
  server_port: 8000

secrets:
  hf_access_token: null

weights:
  - source: hf://nvidia/Llama-3.3-70B-Instruct-FP8@main
    allow_patterns:
      - "*.safetensors"
      - "*.json"
      - "*.model"
      - tokenizer.model
      - "*.tiktoken"
      - "*.jinja"
    mount_location: /app/model_cache/llama-3-3-70b-instruct
    ignore_patterns:
      - original/*
      - "*.pth"
    auth_secret_name: hf_access_token

resources:
  cpu: "4"
  memory: 40Gi
  use_gpu: true
  accelerator: H100:4

runtime:
  predict_concurrency: 128
  streaming_read_timeout: 60
  health_checks:
    startup_threshold_seconds: 1800
    restart_threshold_seconds: 1200
    stop_traffic_threshold_seconds: 120

environment_variables:
  VLLM_LOGGING_LEVEL: INFO
  VLLM_ENGINE_READY_TIMEOUT_S: "3600"

This config runs Llama 3.3 70B Instruct on four H100 GPUs with vLLM, loading FP8 weights from nvidia/Llama-3.3-70B-Instruct-FP8 and sharding them across the four ranks. The server holds the batch to 128 sequences and 8192 batched tokens per scheduler step, and chunked prefill keeps long prompts from stalling the requests already decoding.

Flags

The start_command passes these flags to the engine. Each one controls a runtime or serving behavior:

Flag	Value	What it does
`--tensor-parallel-size`	`4`	Number of GPUs to shard the model across.
`--distributed-executor-backend`	`mp`	How vLLM coordinates tensor-parallel workers across processes. mp: Python multiprocessing (single-node default).
`--max-model-len`	`131072`	Maximum context length (tokens) the server accepts per request.
`--max-num-seqs`	`128`	Maximum number of concurrent sequences in the batch.
`--max-num-batched-tokens`	`8192`	Maximum total tokens processed per scheduler step.
`--enable-chunked-prefill`	(no value)	Process long prompts in chunks so decode requests keep running.
`--enable-prefix-caching`	(no value)	Reuse KV cache across requests that share a prefix.
`--gpu-memory-utilization`	`0.90`	Fraction of GPU memory vLLM may use for weights and KV cache.
`--load-format`	`runai_streamer`	Weight loading backend. runai_streamer: Stream weights from object storage without materializing to disk.

Deploy

Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:

baseten model push

uvx truss push

You should see output similar to:

Pushing model "llama-3.3-70b-instruct-latency"...
Uploading model...
Uploaded model in 0s
✨ Model llama-3.3-70b-instruct-latency was successfully pushed ✨

  Model:       llama-3.3-70b-instruct-latency (abc1d2ef)
  Deployment:  xyz123
  Environment: production

🪵 View logs:
   deployment:   baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
   environment:  baseten model environment logs --model-id abc1d2ef --environment production  (once deployed)
   app:          https://app.baseten.co/models/abc1d2ef/logs/xyz123

🚀 Invoke your model:
   URL:  https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
   CLI:  baseten model predict --model-id abc1d2ef

baseten model push prints your model ID (abc1d2ef in the example). The examples below use it wherever you see {model_id}, and read your API key from the BASETEN_API_KEY environment variable.

Call the model

Your deployment serves an OpenAI-compatible API.Now call your deployment to run inference:

Python
cURL

main.py

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["BASETEN_API_KEY"],
    base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)

response = client.chat.completions.create(
    model="nvidia/Llama-3.3-70B-Instruct-FP8",
    messages=[
        {"role": "user", "content": "What is machine learning?"}
    ],
)

print(response.choices[0].message.content)

curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '{
    "model": "nvidia/Llama-3.3-70B-Instruct-FP8",
    "messages": [
      {"role": "user", "content": "What is machine learning?"}
    ]
  }'

This preset serves Llama 3.3 70B Instruct on H100:4 under vLLM with FP8 weights, optimized for total token throughput on long-context workloads.

Hardware

H100 × 4

Engine

vLLM 0.26.0

Context

128K

Concurrency

256

Write the config

Create and move into the project directory:

mkdir llama-3.3-70b-instruct-throughput && cd llama-3.3-70b-instruct-throughput

Then create a file named config.yaml and paste the following:

config.yaml

########################################################
# Throughput preset for Llama 3.3 70B Instruct (FP8), H100:4, vLLM.
#
# Tuned for high total token throughput on long-context workloads.
# Key throughput levers compared with the latency preset:
# gpu-memory-utilization 0.95 and max-num-seqs 256 for more KV cache headroom
# and larger batches, plus prefix caching for shared prefixes. FP8 (ModelOpt)
# weights and fp8 KV cache are auto-detected from the checkpoint.
# Raising max-num-batched-tokens beyond 16384 added no further throughput.
########################################################

model_name: "model:llama-3.3-70b-instruct preset:throughput"

model_metadata:
  repo_id: nvidia/Llama-3.3-70B-Instruct-FP8
  tags:
    - openai-compatible
    - vllm
  example_model_input:
    stream: true
    model: nvidia/Llama-3.3-70B-Instruct-FP8
    messages:
      - role: user
        content: Tell me everything you know about optimized inference.
    max_tokens: 512
    temperature: 0.5

base_image:
  image: vllm/vllm-openai:v0.26.0

docker_server:
  start_command: >-
    vllm serve /app/model_cache/llama-3-3-70b-instruct
    --served-model-name nvidia/Llama-3.3-70B-Instruct-FP8
    --host 0.0.0.0
    --port 8000
    --tensor-parallel-size 4
    --distributed-executor-backend mp
    --max-model-len 131072
    --max-num-seqs 256
    --max-num-batched-tokens 16384
    --enable-chunked-prefill
    --enable-prefix-caching
    --gpu-memory-utilization 0.95
  readiness_endpoint: /health
  liveness_endpoint: /health
  predict_endpoint: /v1/chat/completions
  server_port: 8000

weights:
  - source: hf://nvidia/Llama-3.3-70B-Instruct-FP8@main
    allow_patterns:
      - "*.safetensors"
      - "*.json"
      - "*.model"
      - tokenizer.model
      - "*.tiktoken"
      - "*.jinja"
    mount_location: /app/model_cache/llama-3-3-70b-instruct
    ignore_patterns:
      - original/*
      - "*.pth"
    auth_secret_name: hf_access_token

secrets:
  hf_access_token: null

resources:
  cpu: "4"
  memory: 40Gi
  use_gpu: true
  accelerator: H100:4

runtime:
  predict_concurrency: 256
  streaming_read_timeout: 60
  health_checks:
    restart_check_delay_seconds: 1800
    restart_threshold_seconds: 1200
    stop_traffic_threshold_seconds: 120

environment_variables:
  VLLM_LOGGING_LEVEL: INFO
  VLLM_ENGINE_READY_TIMEOUT_S: "3600"

This config gives vLLM more room to batch than the latency preset does: 256 concurrent sequences, 16384 batched tokens per scheduler step, and 95% of each H100’s memory for weights and KV cache. Prefix caching keeps the shared portion of repeated prompts resident, which pays off most when many requests carry the same long system prompt.

Flags

The start_command passes these flags to the engine. Each one controls a runtime or serving behavior:

Flag	Value	What it does
`--tensor-parallel-size`	`4`	Number of GPUs to shard the model across.
`--distributed-executor-backend`	`mp`	How vLLM coordinates tensor-parallel workers across processes. mp: Python multiprocessing (single-node default).
`--max-model-len`	`131072`	Maximum context length (tokens) the server accepts per request.
`--max-num-seqs`	`256`	Maximum number of concurrent sequences in the batch.
`--max-num-batched-tokens`	`16384`	Maximum total tokens processed per scheduler step.
`--enable-chunked-prefill`	(no value)	Process long prompts in chunks so decode requests keep running.
`--enable-prefix-caching`	(no value)	Reuse KV cache across requests that share a prefix.
`--gpu-memory-utilization`	`0.95`	Fraction of GPU memory vLLM may use for weights and KV cache.

Deploy

Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:

baseten model push

uvx truss push

You should see output similar to:

Pushing model "llama-3.3-70b-instruct-throughput"...
Uploading model...
Uploaded model in 0s
✨ Model llama-3.3-70b-instruct-throughput was successfully pushed ✨

  Model:       llama-3.3-70b-instruct-throughput (abc1d2ef)
  Deployment:  xyz123
  Environment: production

🪵 View logs:
   deployment:   baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
   environment:  baseten model environment logs --model-id abc1d2ef --environment production  (once deployed)
   app:          https://app.baseten.co/models/abc1d2ef/logs/xyz123

🚀 Invoke your model:
   URL:  https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
   CLI:  baseten model predict --model-id abc1d2ef

Call the model

Your deployment serves an OpenAI-compatible API.Now call your deployment to run inference:

Python
cURL

main.py

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["BASETEN_API_KEY"],
    base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)

response = client.chat.completions.create(
    model="nvidia/Llama-3.3-70B-Instruct-FP8",
    messages=[
        {"role": "user", "content": "What is machine learning?"}
    ],
)

print(response.choices[0].message.content)

curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '{
    "model": "nvidia/Llama-3.3-70B-Instruct-FP8",
    "messages": [
      {"role": "user", "content": "What is machine learning?"}
    ]
  }'

Examples

Models

Engines

Custom Docker servers

Custom Python models

Chains

Setup

Hardware

Engine

Context

Concurrency

Write the config

Flags

Deploy

Call the model

Hardware

Engine

Context

Concurrency

Write the config

Flags

Deploy

Call the model

Next steps

Call your model

Autoscaling

​Setup

Hardware

Engine

Context

Concurrency

​Write the config

​Flags

​Deploy

​Call the model

Hardware

Engine

Context

Concurrency

​Write the config

​Flags

​Deploy

​Call the model

​Next steps

Call your model

Autoscaling

Setup

Write the config

Flags

Deploy

Call the model

Write the config

Flags

Deploy

Call the model

Next steps