Setup
Install the Baseten CLI and sign in, then install the OpenAI SDK.Install and sign in to BasetenFor other platforms or a specific version, see the Baseten CLI install reference.
- macOS or Linux
- Windows
Terminal
brew tap basetenlabs/baseten
brew trust --formula basetenlabs/baseten/baseten
brew install baseten
Download and extract the Windows archive for the latest Baseten CLI release:Move
Terminal
$release = Invoke-RestMethod https://api.github.com/repos/basetenlabs/baseten-cli/releases/latest
$asset = $release.assets | Where-Object { $_.name -like "*_windows_amd64.zip" }
Invoke-WebRequest $asset.browser_download_url -OutFile baseten.zip
Expand-Archive -Force baseten.zip .
baseten.exe to a directory on your PATH.baseten auth login
Install the OpenAI SDK
uv pip install openai
- E2B
- E4B
- 26B A4B
- 31B
google/gemma-4-E2B-it is a 2B-parameter dense model with up to 125K context.This preset serves Gemma 4 E2B on a single L4, the lowest-cost deployment in the Model Library. Then create a file named
You should see output similar to:The server parses the model’s chain of thought into a separate To let the model call tools, pass a
Hardware
L4
Engine
vLLM (0.22.0-cu129 build)
Context
125K
Concurrency
16
Write the config
Create and move into the project directory:mkdir gemma-4-E2B-it-latency && cd gemma-4-E2B-it-latency
config.yaml and paste the following:config.yaml
model_name: model:gemma-4-E2B-it preset:latency
model_metadata:
description: >-
Gemma 4 multimodal instruct (E2B), OpenAI-compatible chat with vision using vLLM on L4.
repo_id: google/gemma-4-E2B-it
example_model_input:
model: google/gemma-4-E2B-it
messages:
- role: user
content:
- type: text
text: "Describe this image in one sentence."
- type: image_url
image_url:
url: "https://picsum.photos/id/237/200/300"
stream: true
max_tokens: 512
temperature: 1.0
tags:
- openai-compatible
base_image:
image: vllm/vllm-openai:v0.22.0-cu129
weights:
- source: "hf://google/gemma-4-E2B-it@3e22461f65e89153144f8adb70e3b8c2cc9845a7"
mount_location: "/app/checkpoint/model"
docker_server:
start_command: >-
sh -c 'GPU_COUNT=$(nvidia-smi --list-gpus | wc -l) && vllm serve /app/checkpoint/model
--tensor-parallel-size $GPU_COUNT
--served-model-name google/gemma-4-E2B-it
--max-num-seqs 16
--max-model-len auto
--limit-mm-per-prompt.image 1
--gpu-memory-utilization 0.9
--async-scheduling
--trust-remote-code
--enable-auto-tool-choice
--enable-prefix-caching
--reasoning-parser gemma4
--tool-call-parser gemma4'
readiness_endpoint: /health
liveness_endpoint: /health
predict_endpoint: /v1/chat/completions
server_port: 8000
environment_variables:
VLLM_LOGGING_LEVEL: WARNING
VLLM_ENGINE_READY_TIMEOUT_S: "3600"
resources:
accelerator: L4
use_gpu: true
runtime:
health_checks:
startup_threshold_seconds: 1800
restart_threshold_seconds: 1200
stop_traffic_threshold_seconds: 120
predict_concurrency: 16
Flags
Thestart_command passes these flags to the engine. Each one controls a runtime or serving behavior:| Flag | Value | What it does |
|---|---|---|
--tensor-parallel-size | $GPU_COUNT | Number of GPUs to shard the model across. |
--max-num-seqs | 16 | Maximum number of concurrent sequences in the batch. |
--max-model-len | auto | Maximum context length (tokens) the server accepts per request. |
--limit-mm-per-prompt.image | 1 | Maximum number of image inputs per prompt. |
--gpu-memory-utilization | 0.9 | Fraction of GPU memory vLLM may use for weights and KV cache. |
--async-scheduling | (no value) | Overlap scheduling with GPU execution to hide scheduler latency. |
--trust-remote-code | (no value) | Execute model-specific Python from the checkpoint (required for many Qwen, Phi, and custom architectures). |
--enable-auto-tool-choice | (no value) | Let the model choose when to call tools without requiring tool_choice: "required". |
--enable-prefix-caching | (no value) | Reuse KV cache across requests that share a prefix. |
--reasoning-parser | gemma4 | Server-side parser that separates reasoning output into reasoning_content. |
--tool-call-parser | gemma4 | Server-side parser that emits structured tool_calls on the response. |
Deploy
Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:baseten model push
uvx truss push
Pushing model "gemma-4-E2B-it-latency"...
Uploading model...
Uploaded model in 0s
✨ Model gemma-4-E2B-it-latency was successfully pushed ✨
Model: gemma-4-E2B-it-latency (abc1d2ef)
Deployment: xyz123
Environment: production
🪵 View logs:
deployment: baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
environment: baseten model environment logs --model-id abc1d2ef --environment production (once deployed)
app: https://app.baseten.co/models/abc1d2ef/logs/xyz123
🚀 Invoke your model:
URL: https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
CLI: baseten model predict --model-id abc1d2ef
baseten model push prints your model ID (abc1d2ef in the example). The examples below use it wherever you see {model_id}, and read your API key from the BASETEN_API_KEY environment variable.Call the model
Your deployment serves an OpenAI-compatible API.Now call your deployment to run inference:- Python
- cURL
main.py
import os
from openai import OpenAI
client = OpenAI(
api_key=os.environ["BASETEN_API_KEY"],
base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)
response = client.chat.completions.create(
model="google/gemma-4-E2B-it",
messages=[
{"role": "user", "content": "What is machine learning?"}
],
)
print(response.choices[0].message.content)
curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $BASETEN_API_KEY" \
-d '{
"model": "google/gemma-4-E2B-it",
"messages": [
{"role": "user", "content": "What is machine learning?"}
]
}'
reasoning_content field on the response. Read it alongside the final answer:response = client.chat.completions.create(
model="google/gemma-4-E2B-it",
messages=[
{"role": "user", "content": "How many r's in strawberry?"}
],
)
print(response.choices[0].message.reasoning_content) # chain of thought
print(response.choices[0].message.content) # final answer
tools array. The server returns structured tool_calls on the response:tools = [{
"type": "function",
"function": {
"name": "get_weather",
"parameters": {
"type": "object",
"properties": {"location": {"type": "string"}},
"required": ["location"],
},
},
}]
response = client.chat.completions.create(
model="google/gemma-4-E2B-it",
messages=[
{"role": "user", "content": "What's the weather in Paris?"}
],
tools=tools,
)
print(response.choices[0].message.tool_calls)
google/gemma-4-E4B-it is a 4B-parameter dense model with up to 125K context.This preset serves Gemma 4 E4B on a single H100. Then create a file named
You should see output similar to:The server parses the model’s chain of thought into a separate To let the model call tools, pass a
Hardware
H100
Engine
vLLM (0.22.0-cu129 build)
Context
125K
Concurrency
16
Write the config
Create and move into the project directory:mkdir gemma-4-E4B-it-latency && cd gemma-4-E4B-it-latency
config.yaml and paste the following:config.yaml
model_name: model:gemma-4-E4B-it preset:latency
model_metadata:
description: >-
Gemma 4 multimodal instruct (E4B), OpenAI-compatible chat with vision using vLLM on H100.
repo_id: google/gemma-4-E4B-it
example_model_input:
model: google/gemma-4-E4B-it
messages:
- role: user
content:
- type: text
text: "Describe this image in one sentence."
- type: image_url
image_url:
url: "https://picsum.photos/id/237/200/300"
stream: true
max_tokens: 512
temperature: 1.0
tags:
- openai-compatible
base_image:
image: vllm/vllm-openai:v0.22.0-cu129
weights:
- source: "hf://google/gemma-4-E4B-it@ee0ef6023621cff504d758262d4e04895a5af4a2"
mount_location: "/app/checkpoint/model"
docker_server:
start_command: >-
sh -c 'GPU_COUNT=$(nvidia-smi --list-gpus | wc -l) && vllm serve /app/checkpoint/model
--tensor-parallel-size $GPU_COUNT
--served-model-name google/gemma-4-E4B-it
--max-num-seqs 16
--max-model-len auto
--limit-mm-per-prompt.image 1
--gpu-memory-utilization 0.9
--async-scheduling
--trust-remote-code
--enable-auto-tool-choice
--enable-prefix-caching
--reasoning-parser gemma4
--tool-call-parser gemma4
--load-format runai_streamer'
readiness_endpoint: /health
liveness_endpoint: /health
predict_endpoint: /v1/chat/completions
server_port: 8000
environment_variables:
VLLM_LOGGING_LEVEL: WARNING
VLLM_ENGINE_READY_TIMEOUT_S: "3600"
resources:
accelerator: H100
use_gpu: true
runtime:
health_checks:
startup_threshold_seconds: 1800
restart_threshold_seconds: 1200
stop_traffic_threshold_seconds: 120
predict_concurrency: 16
Flags
Thestart_command passes these flags to the engine. Each one controls a runtime or serving behavior:| Flag | Value | What it does |
|---|---|---|
--tensor-parallel-size | $GPU_COUNT | Number of GPUs to shard the model across. |
--max-num-seqs | 16 | Maximum number of concurrent sequences in the batch. |
--max-model-len | auto | Maximum context length (tokens) the server accepts per request. |
--limit-mm-per-prompt.image | 1 | Maximum number of image inputs per prompt. |
--gpu-memory-utilization | 0.9 | Fraction of GPU memory vLLM may use for weights and KV cache. |
--async-scheduling | (no value) | Overlap scheduling with GPU execution to hide scheduler latency. |
--trust-remote-code | (no value) | Execute model-specific Python from the checkpoint (required for many Qwen, Phi, and custom architectures). |
--enable-auto-tool-choice | (no value) | Let the model choose when to call tools without requiring tool_choice: "required". |
--enable-prefix-caching | (no value) | Reuse KV cache across requests that share a prefix. |
--reasoning-parser | gemma4 | Server-side parser that separates reasoning output into reasoning_content. |
--tool-call-parser | gemma4 | Server-side parser that emits structured tool_calls on the response. |
--load-format | runai_streamer | Weight loading backend. runai_streamer: Stream weights from object storage without materializing to disk. |
Deploy
Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:baseten model push
uvx truss push
Pushing model "gemma-4-E4B-it-latency"...
Uploading model...
Uploaded model in 0s
✨ Model gemma-4-E4B-it-latency was successfully pushed ✨
Model: gemma-4-E4B-it-latency (abc1d2ef)
Deployment: xyz123
Environment: production
🪵 View logs:
deployment: baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
environment: baseten model environment logs --model-id abc1d2ef --environment production (once deployed)
app: https://app.baseten.co/models/abc1d2ef/logs/xyz123
🚀 Invoke your model:
URL: https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
CLI: baseten model predict --model-id abc1d2ef
baseten model push prints your model ID (abc1d2ef in the example). The examples below use it wherever you see {model_id}, and read your API key from the BASETEN_API_KEY environment variable.Call the model
Your deployment serves an OpenAI-compatible API.Now call your deployment to run inference:- Python
- cURL
main.py
import os
from openai import OpenAI
client = OpenAI(
api_key=os.environ["BASETEN_API_KEY"],
base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)
response = client.chat.completions.create(
model="google/gemma-4-E4B-it",
messages=[
{"role": "user", "content": "What is machine learning?"}
],
)
print(response.choices[0].message.content)
curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $BASETEN_API_KEY" \
-d '{
"model": "google/gemma-4-E4B-it",
"messages": [
{"role": "user", "content": "What is machine learning?"}
]
}'
reasoning_content field on the response. Read it alongside the final answer:response = client.chat.completions.create(
model="google/gemma-4-E4B-it",
messages=[
{"role": "user", "content": "How many r's in strawberry?"}
],
)
print(response.choices[0].message.reasoning_content) # chain of thought
print(response.choices[0].message.content) # final answer
tools array. The server returns structured tool_calls on the response:tools = [{
"type": "function",
"function": {
"name": "get_weather",
"parameters": {
"type": "object",
"properties": {"location": {"type": "string"}},
"required": ["location"],
},
},
}]
response = client.chat.completions.create(
model="google/gemma-4-E4B-it",
messages=[
{"role": "user", "content": "What's the weather in Paris?"}
],
tools=tools,
)
print(response.choices[0].message.tool_calls)
google/gemma-4-26B-A4B-it is a 26B-parameter MoE model (4B active per token) with up to 256K context.This preset serves Gemma 4 26B A4B on H100:2 with FP8 dynamic quantization. Then create a file named
You should see output similar to:The server parses the model’s chain of thought into a separate To let the model call tools, pass a
Hardware
H100 × 2
Engine
vLLM (0.22.0-cu129 build)
Context
256K
Concurrency
16
Write the config
Create and move into the project directory:mkdir gemma-4-26B-A4B-it-latency && cd gemma-4-26B-A4B-it-latency
config.yaml and paste the following:config.yaml
model_name: model:gemma-4-26B-A4B-it preset:latency
model_metadata:
description: >-
Gemma 4 multimodal instruct (26B MOE FP8 dynamique), speculative decoding Eagle3 using vLLM.
repo_id: RedHatAI/gemma-4-26B-A4B-it-FP8-Dynamic
example_model_input:
model: google/gemma-4-26B-A4B-it
messages:
- role: user
content:
- type: text
text: "Describe this image in one sentence."
- type: image_url
image_url:
url: "https://picsum.photos/id/237/200/300"
stream: true
max_tokens: 512
temperature: 1.0
tags:
- openai-compatible
base_image:
image: vllm/vllm-openai:v0.22.0-cu129
weights:
- source: "hf://RedHatAI/gemma-4-26B-A4B-it-FP8-Dynamic@ed35d7abe5d940da41b4ff06eb482feb0be8cb44"
mount_location: "/app/checkpoint/model"
- source: "hf://RedHatAI/gemma-4-26B-A4B-it-speculator.eagle3@226b2692d48029aae52adc1d0021a9d58709d33c"
mount_location: "/app/checkpoint/speculator"
docker_server:
start_command: >-
sh -c 'GPU_COUNT=$(nvidia-smi --list-gpus | wc -l) && vllm serve /app/checkpoint/model
--tensor-parallel-size $GPU_COUNT
--served-model-name google/gemma-4-26B-A4B-it
--max-num-seqs 16
--max-model-len auto
--limit-mm-per-prompt.image 1
--gpu-memory-utilization 0.9
--enable-prefix-caching
--speculative-config.model /app/checkpoint/speculator
--speculative-config.num_speculative_tokens 3
--speculative-config.method eagle3
--trust-remote-code
--enable-auto-tool-choice
--reasoning-parser gemma4
--tool-call-parser gemma4
--load-format runai_streamer'
readiness_endpoint: /health
liveness_endpoint: /health
predict_endpoint: /v1/chat/completions
server_port: 8000
environment_variables:
VLLM_LOGGING_LEVEL: WARNING
VLLM_ENGINE_READY_TIMEOUT_S: "3600"
resources:
accelerator: H100:2
use_gpu: true
runtime:
health_checks:
startup_threshold_seconds: 1800
restart_threshold_seconds: 1200
stop_traffic_threshold_seconds: 120
predict_concurrency: 16
Flags
Thestart_command passes these flags to the engine. Each one controls a runtime or serving behavior:| Flag | Value | What it does |
|---|---|---|
--tensor-parallel-size | $GPU_COUNT | Number of GPUs to shard the model across. |
--max-num-seqs | 16 | Maximum number of concurrent sequences in the batch. |
--max-model-len | auto | Maximum context length (tokens) the server accepts per request. |
--limit-mm-per-prompt.image | 1 | Maximum number of image inputs per prompt. |
--gpu-memory-utilization | 0.9 | Fraction of GPU memory vLLM may use for weights and KV cache. |
--enable-prefix-caching | (no value) | Reuse KV cache across requests that share a prefix. |
--speculative-config.model | /app/checkpoint/speculator | Path to the draft speculator checkpoint (HF repo or local mount path). |
--speculative-config.num_speculative_tokens | 3 | Number of tokens the draft speculator proposes per step. |
--speculative-config.method | eagle3 | Speculative decoding method. eagle3: EAGLE v3 speculative decoding. |
--trust-remote-code | (no value) | Execute model-specific Python from the checkpoint (required for many Qwen, Phi, and custom architectures). |
--enable-auto-tool-choice | (no value) | Let the model choose when to call tools without requiring tool_choice: "required". |
--reasoning-parser | gemma4 | Server-side parser that separates reasoning output into reasoning_content. |
--tool-call-parser | gemma4 | Server-side parser that emits structured tool_calls on the response. |
--load-format | runai_streamer | Weight loading backend. runai_streamer: Stream weights from object storage without materializing to disk. |
Deploy
Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:baseten model push
uvx truss push
Pushing model "gemma-4-26B-A4B-it-latency"...
Uploading model...
Uploaded model in 0s
✨ Model gemma-4-26B-A4B-it-latency was successfully pushed ✨
Model: gemma-4-26B-A4B-it-latency (abc1d2ef)
Deployment: xyz123
Environment: production
🪵 View logs:
deployment: baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
environment: baseten model environment logs --model-id abc1d2ef --environment production (once deployed)
app: https://app.baseten.co/models/abc1d2ef/logs/xyz123
🚀 Invoke your model:
URL: https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
CLI: baseten model predict --model-id abc1d2ef
baseten model push prints your model ID (abc1d2ef in the example). The examples below use it wherever you see {model_id}, and read your API key from the BASETEN_API_KEY environment variable.Call the model
Your deployment serves an OpenAI-compatible API.Now call your deployment to run inference:- Python
- cURL
main.py
import os
from openai import OpenAI
client = OpenAI(
api_key=os.environ["BASETEN_API_KEY"],
base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)
response = client.chat.completions.create(
model="google/gemma-4-26B-A4B-it",
messages=[
{"role": "user", "content": "What is machine learning?"}
],
)
print(response.choices[0].message.content)
curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $BASETEN_API_KEY" \
-d '{
"model": "google/gemma-4-26B-A4B-it",
"messages": [
{"role": "user", "content": "What is machine learning?"}
]
}'
reasoning_content field on the response. Read it alongside the final answer:response = client.chat.completions.create(
model="google/gemma-4-26B-A4B-it",
messages=[
{"role": "user", "content": "How many r's in strawberry?"}
],
)
print(response.choices[0].message.reasoning_content) # chain of thought
print(response.choices[0].message.content) # final answer
tools array. The server returns structured tool_calls on the response:tools = [{
"type": "function",
"function": {
"name": "get_weather",
"parameters": {
"type": "object",
"properties": {"location": {"type": "string"}},
"required": ["location"],
},
},
}]
response = client.chat.completions.create(
model="google/gemma-4-26B-A4B-it",
messages=[
{"role": "user", "content": "What's the weather in Paris?"}
],
tools=tools,
)
print(response.choices[0].message.tool_calls)
google/gemma-4-31B-it is a 31B-parameter dense model with up to 256K context.This variant ships in 2 presets tuned for different goals: Latency for lowest time-to-first-token, and Throughput for highest tokens per second. Pick the tab that matches your workload. Then create a file named You should see output similar to:
- Latency
- Throughput
This preset serves Gemma 4 31B on H100:2 with FP8 block quantization. Then create a file named
You should see output similar to:The server parses the model’s chain of thought into a separate To let the model call tools, pass a
Hardware
H100 × 2
Engine
vLLM 0.31.0
Context
256K
Concurrency
16
Write the config
Create and move into the project directory:mkdir gemma-4-31B-it-latency && cd gemma-4-31B-it-latency
config.yaml and paste the following:config.yaml
model_name: model:gemma-4-31B-it preset:latency
model_metadata:
description: >-
Gemma 4 multimodal instruct (FP8), OpenAI-compatible chat with vision using vLLM.
repo_id: RedHatAI/gemma-4-31B-it-FP8-block
example_model_input:
model: google/gemma-4-31B-it
messages:
- role: user
content:
- type: text
text: "Describe this image in one sentence."
- type: image_url
image_url:
url: "https://picsum.photos/id/237/200/300"
stream: true
max_tokens: 512
temperature: 1.0
tags:
- openai-compatible
base_image:
image: vllm/vllm-openai:v0.31.0
weights:
- source: "hf://RedHatAI/gemma-4-31B-it-FP8-block@d7242548c457ab4b45bd3adb8937f2659af4739d"
mount_location: "/app/checkpoint/model"
- source: "hf://RedHatAI/gemma-4-31B-it-speculator.eagle3@28a1c8b4bb64dbaee883ba35341841138bdf1fe3"
mount_location: "/app/checkpoint/speculator"
docker_server:
start_command: >-
sh -c 'GPU_COUNT=$(nvidia-smi --list-gpus | wc -l) && vllm serve /app/checkpoint/model
--tensor-parallel-size $GPU_COUNT
--served-model-name google/gemma-4-31B-it
--max-num-seqs 16
--max-model-len auto
--limit-mm-per-prompt.image 1
--gpu-memory-utilization 0.9
--enable-prefix-caching
--speculative-config.model /app/checkpoint/speculator
--speculative-config.num_speculative_tokens 3
--speculative-config.method eagle3
--trust-remote-code
--enable-auto-tool-choice
--reasoning-parser gemma4
--tool-call-parser gemma4
--load-format runai_streamer'
readiness_endpoint: /health
liveness_endpoint: /health
predict_endpoint: /v1/chat/completions
server_port: 8000
environment_variables:
VLLM_LOGGING_LEVEL: WARNING
VLLM_ENGINE_READY_TIMEOUT_S: "3600"
resources:
accelerator: H100:2
use_gpu: true
runtime:
health_checks:
startup_threshold_seconds: 1800
restart_threshold_seconds: 1200
stop_traffic_threshold_seconds: 120
predict_concurrency: 16
Flags
Thestart_command passes these flags to the engine. Each one controls a runtime or serving behavior:| Flag | Value | What it does |
|---|---|---|
--tensor-parallel-size | $GPU_COUNT | Number of GPUs to shard the model across. |
--max-num-seqs | 16 | Maximum number of concurrent sequences in the batch. |
--max-model-len | auto | Maximum context length (tokens) the server accepts per request. |
--limit-mm-per-prompt.image | 1 | Maximum number of image inputs per prompt. |
--gpu-memory-utilization | 0.9 | Fraction of GPU memory vLLM may use for weights and KV cache. |
--enable-prefix-caching | (no value) | Reuse KV cache across requests that share a prefix. |
--speculative-config.model | /app/checkpoint/speculator | Path to the draft speculator checkpoint (HF repo or local mount path). |
--speculative-config.num_speculative_tokens | 3 | Number of tokens the draft speculator proposes per step. |
--speculative-config.method | eagle3 | Speculative decoding method. eagle3: EAGLE v3 speculative decoding. |
--trust-remote-code | (no value) | Execute model-specific Python from the checkpoint (required for many Qwen, Phi, and custom architectures). |
--enable-auto-tool-choice | (no value) | Let the model choose when to call tools without requiring tool_choice: "required". |
--reasoning-parser | gemma4 | Server-side parser that separates reasoning output into reasoning_content. |
--tool-call-parser | gemma4 | Server-side parser that emits structured tool_calls on the response. |
--load-format | runai_streamer | Weight loading backend. runai_streamer: Stream weights from object storage without materializing to disk. |
Deploy
Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:baseten model push
uvx truss push
Pushing model "gemma-4-31B-it-latency"...
Uploading model...
Uploaded model in 0s
✨ Model gemma-4-31B-it-latency was successfully pushed ✨
Model: gemma-4-31B-it-latency (abc1d2ef)
Deployment: xyz123
Environment: production
🪵 View logs:
deployment: baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
environment: baseten model environment logs --model-id abc1d2ef --environment production (once deployed)
app: https://app.baseten.co/models/abc1d2ef/logs/xyz123
🚀 Invoke your model:
URL: https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
CLI: baseten model predict --model-id abc1d2ef
baseten model push prints your model ID (abc1d2ef in the example). The examples below use it wherever you see {model_id}, and read your API key from the BASETEN_API_KEY environment variable.Call the model
Your deployment serves an OpenAI-compatible API.Now call your deployment to run inference:- Python
- cURL
main.py
import os
from openai import OpenAI
client = OpenAI(
api_key=os.environ["BASETEN_API_KEY"],
base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)
response = client.chat.completions.create(
model="google/gemma-4-31B-it",
messages=[
{"role": "user", "content": "What is machine learning?"}
],
)
print(response.choices[0].message.content)
curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $BASETEN_API_KEY" \
-d '{
"model": "google/gemma-4-31B-it",
"messages": [
{"role": "user", "content": "What is machine learning?"}
]
}'
reasoning_content field on the response. Read it alongside the final answer:response = client.chat.completions.create(
model="google/gemma-4-31B-it",
messages=[
{"role": "user", "content": "How many r's in strawberry?"}
],
)
print(response.choices[0].message.reasoning_content) # chain of thought
print(response.choices[0].message.content) # final answer
tools array. The server returns structured tool_calls on the response:tools = [{
"type": "function",
"function": {
"name": "get_weather",
"parameters": {
"type": "object",
"properties": {"location": {"type": "string"}},
"required": ["location"],
},
},
}]
response = client.chat.completions.create(
model="google/gemma-4-31B-it",
messages=[
{"role": "user", "content": "What's the weather in Paris?"}
],
tools=tools,
)
print(response.choices[0].message.tool_calls)
Hardware
RTX_PRO_6000
Context
128K
Write the config
Create and move into the project directory:mkdir gemma-4-31B-it && cd gemma-4-31B-it
config.yaml and paste the following:config.yaml
# Gemma 4 31B (NVIDIA NVFP4) on BIS-LLM with the vLLM backend, one RTX PRO 6000, throughput.
# Converted in place from the vLLM docker_server preset: same checkpoint, 64 sequences,
# 131,072 context, FP8 KV cache, chunked prefill and prefix caching.
#
# Serving image: the 2026-11-02 MP release (baseten/bis-llm-vllm:mp-v2026.11.02), pinned by its
# green tag; includes the Gemma 4 reasoning-parser fix (baseten#37383). Platform-default chart.
#
# Text only (language_model_only: true). Current BIS plans images in the frontend with
# mm-libtorch-encoder, which has no Gemma 4 family yet, so image requests would fail; the
# vLLM preset accepted images.
# Weights pinned to the revision in variant.json (the vLLM preset used @main).
model_name: "model:gemma-4-31B-it preset:bis-throughput"
model_metadata:
description: >-
Gemma 4 instruct (NVFP4), OpenAI-compatible chat (text only), served with vLLM through the
Baseten Inference Stack on RTX PRO 6000.
repo_id: nvidia/Gemma-4-31B-IT-NVFP4
example_model_input:
model: google/gemma-4-31B-it
messages:
- role: user
content: "Summarize the benefits of solar power in two sentences."
stream: true
max_tokens: 512
temperature: 1.0
tags:
- openai-compatible
bis_llm:
config:
additional_environment_variables:
Worker:
HF_HUB_ENABLE_HF_TRANSFER: "1"
SAFETENSORS_FAST_GPU: "1"
VLLM_ENGINE_READY_TIMEOUT_S: "3600"
arguments_as_json: true
b10_routing_config:
algo_selector: B10
durable_kv_events: false
enable_worker_tree: true
router_cache_miss_weight: 0
router_overlap_score_weight: 1000
router_temperature: 0.001
checkpoint_name: nvidia/Gemma-4-31B-IT-NVFP4
engine_backend: vllm
engine_config:
dtype: auto
enable_chunked_prefill: true
enable_prefix_caching: true
enforce_eager: false
kv_cache_dtype: fp8
kv_events_endpoint: "tcp://*:5557"
max_model_len: 131072
max_num_seqs: 64
trust_remote_code: true
gpuTRTImage: baseten/bis-llm-vllm:green-mp--v0.26.0-d9e38b3e4-b6573aaba5
language_model_only: true
model_name: google/gemma-4-31B-it
model_path: /models/gemma-4-31b
reasoning_parser: gemma_4
served_model_name: google/gemma-4-31B-it
served_model_name_response: google/gemma-4-31B-it
tensor_parallel_size: 1
tokenizer_limit_length: 131072
tool_call_parser: gemma_4
environment_variables: {}
resources:
accelerator: RTX_PRO_6000
use_gpu: true
weights:
- source: hf://nvidia/Gemma-4-31B-IT-NVFP4@4135a98a9b728a548947683219633b25682223ac
mount_location: /models/gemma-4-31b
Deploy
Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:baseten model push
uvx truss push
Pushing model "gemma-4-31B-it"...
Uploading model...
Uploaded model in 0s
✨ Model gemma-4-31B-it was successfully pushed ✨
Model: gemma-4-31B-it (abc1d2ef)
Deployment: xyz123
Environment: production
🪵 View logs:
deployment: baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
environment: baseten model environment logs --model-id abc1d2ef --environment production (once deployed)
app: https://app.baseten.co/models/abc1d2ef/logs/xyz123
🚀 Invoke your model:
URL: https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
CLI: baseten model predict --model-id abc1d2ef
baseten model push prints your model ID (abc1d2ef in the example). The examples below use it wherever you see {model_id}, and read your API key from the BASETEN_API_KEY environment variable.Call the model
Your deployment serves an OpenAI-compatible API.Now call your deployment to run inference:- Python
- cURL
main.py
import os
from openai import OpenAI
client = OpenAI(
api_key=os.environ["BASETEN_API_KEY"],
base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)
response = client.chat.completions.create(
model="google/gemma-4-31B-it",
messages=[
{"role": "user", "content": "What is machine learning?"}
],
)
print(response.choices[0].message.content)
curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $BASETEN_API_KEY" \
-d '{
"model": "google/gemma-4-31B-it",
"messages": [
{"role": "user", "content": "What is machine learning?"}
]
}'
Next steps
Call your model
Endpoint anatomy, authentication, and sync versus async inference
Autoscaling
Scale replicas with traffic, including scale to zero