> ## Documentation Index
> Fetch the complete documentation index at: https://docs.baseten.co/llms.txt
> Use this file to discover all available pages before exploring further.
# Llama 3.1
> Meta's Llama 3.1 8B instruction-tuned model.
Meta's Llama 3.1 8B instruction-tuned model. Runs on a single B200 from NVIDIA's FP8 checkpoint with EAGLE3 speculative decoding for high concurrent throughput.
## Setup
Install the Baseten CLI and sign in, then install the OpenAI SDK.
**Install and sign in to Baseten**
```bash Terminal theme={"system"}
brew tap basetenlabs/baseten
brew trust --formula basetenlabs/baseten/baseten
brew install baseten
```
Download and extract the binary, then move `baseten.exe` to a directory on your `PATH`:
```powershell Terminal theme={"system"}
Invoke-WebRequest `
https://github.com/basetenlabs/baseten-cli/releases/download/v1.0.0/baseten_1.0.0_windows_amd64.zip `
-OutFile baseten.zip; Expand-Archive -Force baseten.zip .
```
For other platforms or a specific version, see the [Baseten CLI install reference](/reference/cli/baseten/overview#install).
```sh theme={"system"}
baseten auth login
```
**Install the OpenAI SDK**
```sh theme={"system"}
uv pip install openai
```
To install without Homebrew, see the [Baseten CLI install reference](/reference/cli/baseten/overview#install).
This preset serves Llama 3.1 8B Instruct on a single B200 through [Baseten Inference Stack](/engines/bis-llm/overview) (TensorRT-LLM) with FP8 weights, an FP8 KV cache, and EAGLE3 speculative decoding. It targets high concurrent throughput.
B200
TRT-LLM v2
128K
512
## Write the config
Create and move into the project directory:
```sh theme={"system"}
mkdir llama-3.1-8b-instruct-throughput && cd llama-3.1-8b-instruct-throughput
```
Then create a file named `config.yaml` and paste the following:
```yaml config.yaml theme={"system"}
model_name: "model:llama-3.1-8b-instruct preset:throughput"
build_commands:
# The overlap scheduler uses two sequence-slot pools, but this runtime sizes
# CapturableGuidedDecoder to one pool. Match its buffers to the sampler's
# doubled slot capacity (NVIDIA/TensorRT-LLM#16279).
- >-
python3 -c 'from pathlib import Path;
p=next(Path("/usr/local/lib").glob("python*/dist-packages/tensorrt_llm/_torch/pyexecutor/py_executor_creator.py"));
s=p.read_text();
old=" \"max_num_sequences\": max_batch_size,";
new=" \"max_num_sequences\": max_batch_size * (1 if llm_args.disable_overlap_scheduler else 2),";
assert s.count(old)==1; p.write_text(s.replace(old,new,1))'
# Truss synthesizes this contract for inference_stack v2 during image builds.
# Keep it explicit so slim-image metadata makes the operator probe Dynamo on
# port 8000 instead of expecting the legacy nginx proxy on port 8080.
docker_server:
start_command: /workspace/trtllm/standalone/launch.sh
server_port: 8000
predict_endpoint: /v1/chat/completions
readiness_endpoint: /health_file
liveness_endpoint: /health_file
model_metadata:
example_model_input:
messages:
- role: user
content: "Write FizzBuzz in Python"
stream: true
model: "nvidia/Llama-3.1-8B-Instruct-FP8"
max_tokens: 512
temperature: 0.5
tags:
- openai-compatible
resources:
accelerator: B200
cpu: "1"
memory: 10Gi
use_gpu: true
weights:
- source: "hf://nvidia/Llama-3.1-8B-Instruct-FP8@main"
mount_location: "/app/model_cache/trt_model"
- source: "hf://yuhuili/EAGLE3-LLaMA3.1-Instruct-8B@main"
mount_location: "/app/model_cache/eagle3_draft"
trt_llm:
build:
checkpoint_repository:
repo: michaelfeil/empty-model
revision: main
source: HF
inference_stack: v2
runtime:
enable_chunked_prefill: true
max_batch_size: 512
max_num_tokens: 16384
max_seq_len: 131072
tensor_parallel_size: 1
served_model_name: nvidia/Llama-3.1-8B-Instruct-FP8
patch_kwargs:
model_path: /app/model_cache/trt_model
backend: pytorch
sampler_type: TorchSampler
guided_decoding_backend: xgrammar
max_beam_width: 1
max_input_len: 131072
trust_remote_code: 1
cuda_graph_config:
enable_padding: true
max_batch_size: 512
kv_cache_config:
dtype: fp8
enable_block_reuse: true
free_gpu_memory_fraction: 0.9
speculative_config:
decoding_type: Eagle
max_draft_len: 3
speculative_model_dir: /app/model_cache/eagle3_draft
eagle3_one_model: true
version_overrides:
# Pin a schema-compatible runtime whose Dynamo frontend exposes the
# /health_file endpoint expected by Truss readiness checks.
v2_llm_version: trtllm-gpu-f83d539b2a-00b032f66-c62638826d
runtime:
predict_concurrency: 512
```
This config tells Baseten to compile a TensorRT-LLM engine for Llama 3.1 8B Instruct on a single B200, pulling FP8 weights from `nvidia/Llama-3.1-8B-Instruct-FP8` and an EAGLE3 draft speculator from `yuhuili/EAGLE3-LLaMA3.1-Instruct-8B`. The runtime is tuned for high concurrent throughput: 512 in-flight requests, chunked prefill, an FP8 KV cache, and CUDA graphs sized to the same batch ceiling so the engine stays hot under load.
## Key parameters
[Baseten Inference Stack](/engines/bis-llm/overview) (BIS) reads these fields from the `trt_llm` block. Each one shapes how the engine is built and served:
| Parameter | Value |
| -------------------- | ---------------------------------- |
| Tensor parallel size | `1` |
| Max sequence length | `131072` |
| Max batch size | `512` |
| Max batched tokens | `16384` |
| Chunked prefill | `enabled` |
| Inference stack | `v2` |
| Served model name | `nvidia/Llama-3.1-8B-Instruct-FP8` |
## Deploy
Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:
```sh Baseten CLI theme={"system"}
baseten model push
```
```sh Truss CLI theme={"system"}
uvx truss push
```
You should see output similar to:
```output theme={"system"}
Pushing model "llama-3.1-8b-instruct-throughput"...
Uploading model...
Uploaded model in 0s
✨ Model llama-3.1-8b-instruct-throughput was successfully pushed ✨
Model: llama-3.1-8b-instruct-throughput (abc1d2ef)
Deployment: xyz123
Environment: production
🪵 View logs:
deployment: baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
environment: baseten model environment logs --model-id abc1d2ef --environment production (once deployed)
app: https://app.baseten.co/models/abc1d2ef/logs/xyz123
🚀 Invoke your model:
URL: https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
CLI: baseten model predict --model-id abc1d2ef
```
`baseten model push` prints your **model ID** (`abc1d2ef` in the example). The examples below use it wherever you see `{model_id}`, and read your API key from the `BASETEN_API_KEY` environment variable.
## Call the model
Your deployment serves an OpenAI-compatible API.
Now call your deployment to run inference:
```python main.py theme={"system"}
import os
from openai import OpenAI
client = OpenAI(
api_key=os.environ["BASETEN_API_KEY"],
base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)
response = client.chat.completions.create(
model="nvidia/Llama-3.1-8B-Instruct-FP8",
messages=[
{"role": "user", "content": "What is machine learning?"}
],
)
print(response.choices[0].message.content)
```
```sh theme={"system"}
curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $BASETEN_API_KEY" \
-d '{
"model": "nvidia/Llama-3.1-8B-Instruct-FP8",
"messages": [
{"role": "user", "content": "What is machine learning?"}
]
}'
```
## Next steps
Endpoint anatomy, authentication, and sync versus async inference
Scale replicas with traffic, including scale to zero