Gemma 4 - Baseten

Reasoning Tool calling Multimodal (image)Long context

Setup

Install the Baseten CLI and sign in, then install the OpenAI SDK.

Install and sign in to Baseten

macOS or Linux
Windows

Terminal

brew tap basetenlabs/baseten
brew trust --formula basetenlabs/baseten/baseten
brew install baseten

Download and extract the Windows archive for the latest Baseten CLI release:

Terminal

$release = Invoke-RestMethod https://api.github.com/repos/basetenlabs/baseten-cli/releases/latest
$asset = $release.assets | Where-Object { $_.name -like "*_windows_amd64.zip" }
Invoke-WebRequest $asset.browser_download_url -OutFile baseten.zip
Expand-Archive -Force baseten.zip .

Move baseten.exe to a directory on your PATH.

For other platforms or a specific version, see the Baseten CLI install reference.

baseten auth login

Install the OpenAI SDK

uv pip install openai

To install without Homebrew, see the Baseten CLI install reference. Pick the model you want to deploy. Each tab is a self-contained recipe.

E2B
E4B
26B A4B
31B

google/gemma-4-E2B-it is a 2B-parameter dense model with up to 125K context.This preset serves Gemma 4 E2B on a single L4, the lowest-cost deployment in the Model Library.

Hardware

Engine

vLLM (0.22.0-cu129 build)

Context

125K

Concurrency

Write the config

Create and move into the project directory:

mkdir gemma-4-E2B-it-latency && cd gemma-4-E2B-it-latency

Then create a file named config.yaml and paste the following:

config.yaml

model_name: model:gemma-4-E2B-it preset:latency
model_metadata:
  description: >-
    Gemma 4 multimodal instruct (E2B), OpenAI-compatible chat with vision using vLLM on L4.
  repo_id: google/gemma-4-E2B-it
  example_model_input:
    model: google/gemma-4-E2B-it
    messages:
      - role: user
        content:
          - type: text
            text: "Describe this image in one sentence."
          - type: image_url
            image_url:
              url: "https://picsum.photos/id/237/200/300"
    stream: true
    max_tokens: 512
    temperature: 1.0
  tags:
    - openai-compatible
base_image:
  image: vllm/vllm-openai:v0.22.0-cu129
weights:
  - source: "hf://google/gemma-4-E2B-it@3e22461f65e89153144f8adb70e3b8c2cc9845a7"
    mount_location: "/app/checkpoint/model"
docker_server:
  start_command: >-
    sh -c 'GPU_COUNT=$(nvidia-smi --list-gpus | wc -l) && vllm serve /app/checkpoint/model
    --tensor-parallel-size $GPU_COUNT
    --served-model-name google/gemma-4-E2B-it
    --max-num-seqs 16
    --max-model-len auto
    --limit-mm-per-prompt.image 1
    --gpu-memory-utilization 0.9
    --async-scheduling
    --trust-remote-code
    --enable-auto-tool-choice
    --enable-prefix-caching
    --reasoning-parser gemma4
    --tool-call-parser gemma4'
  readiness_endpoint: /health
  liveness_endpoint: /health
  predict_endpoint: /v1/chat/completions
  server_port: 8000
environment_variables:
  VLLM_LOGGING_LEVEL: WARNING
  VLLM_ENGINE_READY_TIMEOUT_S: "3600"
resources:
  accelerator: L4
  use_gpu: true
runtime:
  health_checks:
    startup_threshold_seconds: 1800
    restart_threshold_seconds: 1200
    stop_traffic_threshold_seconds: 120
  predict_concurrency: 16

Flags

The start_command passes these flags to the engine. Each one controls a runtime or serving behavior:

Flag	Value	What it does
`--tensor-parallel-size`	`$GPU_COUNT`	Number of GPUs to shard the model across.
`--max-num-seqs`	`16`	Maximum number of concurrent sequences in the batch.
`--max-model-len`	`auto`	Maximum context length (tokens) the server accepts per request.
`--limit-mm-per-prompt.image`	`1`	Maximum number of image inputs per prompt.
`--gpu-memory-utilization`	`0.9`	Fraction of GPU memory vLLM may use for weights and KV cache.
`--async-scheduling`	(no value)	Overlap scheduling with GPU execution to hide scheduler latency.
`--trust-remote-code`	(no value)	Execute model-specific Python from the checkpoint (required for many Qwen, Phi, and custom architectures).
`--enable-auto-tool-choice`	(no value)	Let the model choose when to call tools without requiring `tool_choice: "required"`.
`--enable-prefix-caching`	(no value)	Reuse KV cache across requests that share a prefix.
`--reasoning-parser`	`gemma4`	Server-side parser that separates reasoning output into `reasoning_content`.
`--tool-call-parser`	`gemma4`	Server-side parser that emits structured `tool_calls` on the response.

Deploy

Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:

baseten model push

uvx truss push

You should see output similar to:

Pushing model "gemma-4-E2B-it-latency"...
Uploading model...
Uploaded model in 0s
✨ Model gemma-4-E2B-it-latency was successfully pushed ✨

  Model:       gemma-4-E2B-it-latency (abc1d2ef)
  Deployment:  xyz123
  Environment: production

🪵 View logs:
   deployment:   baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
   environment:  baseten model environment logs --model-id abc1d2ef --environment production  (once deployed)
   app:          https://app.baseten.co/models/abc1d2ef/logs/xyz123

🚀 Invoke your model:
   URL:  https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
   CLI:  baseten model predict --model-id abc1d2ef

baseten model push prints your model ID (abc1d2ef in the example). The examples below use it wherever you see {model_id}, and read your API key from the BASETEN_API_KEY environment variable.

Call the model

Your deployment serves an OpenAI-compatible API.Now call your deployment to run inference:

Python
cURL

main.py

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["BASETEN_API_KEY"],
    base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)

response = client.chat.completions.create(
    model="google/gemma-4-E2B-it",
    messages=[
        {"role": "user", "content": "What is machine learning?"}
    ],
)

print(response.choices[0].message.content)

curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '{
    "model": "google/gemma-4-E2B-it",
    "messages": [
      {"role": "user", "content": "What is machine learning?"}
    ]
  }'

The server parses the model’s chain of thought into a separate reasoning_content field on the response. Read it alongside the final answer:

response = client.chat.completions.create(
    model="google/gemma-4-E2B-it",
    messages=[
        {"role": "user", "content": "How many r's in strawberry?"}
    ],
)
print(response.choices[0].message.reasoning_content)  # chain of thought
print(response.choices[0].message.content)            # final answer

To let the model call tools, pass a tools array. The server returns structured tool_calls on the response:

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "parameters": {
            "type": "object",
            "properties": {"location": {"type": "string"}},
            "required": ["location"],
        },
    },
}]

response = client.chat.completions.create(
    model="google/gemma-4-E2B-it",
    messages=[
        {"role": "user", "content": "What's the weather in Paris?"}
    ],
    tools=tools,
)
print(response.choices[0].message.tool_calls)

google/gemma-4-E4B-it is a 4B-parameter dense model with up to 125K context.This preset serves Gemma 4 E4B on a single H100.

Hardware

H100

Engine

vLLM (0.22.0-cu129 build)

Context

125K

Concurrency

Write the config

Create and move into the project directory:

mkdir gemma-4-E4B-it-latency && cd gemma-4-E4B-it-latency

Then create a file named config.yaml and paste the following:

config.yaml

model_name: model:gemma-4-E4B-it preset:latency
model_metadata:
  description: >-
    Gemma 4 multimodal instruct (E4B), OpenAI-compatible chat with vision using vLLM on H100.
  repo_id: google/gemma-4-E4B-it
  example_model_input:
    model: google/gemma-4-E4B-it
    messages:
      - role: user
        content:
          - type: text
            text: "Describe this image in one sentence."
          - type: image_url
            image_url:
              url: "https://picsum.photos/id/237/200/300"
    stream: true
    max_tokens: 512
    temperature: 1.0
  tags:
    - openai-compatible
base_image:
  image: vllm/vllm-openai:v0.22.0-cu129
weights:
  - source: "hf://google/gemma-4-E4B-it@ee0ef6023621cff504d758262d4e04895a5af4a2"
    mount_location: "/app/checkpoint/model"
docker_server:
  start_command: >-
    sh -c 'GPU_COUNT=$(nvidia-smi --list-gpus | wc -l) && vllm serve /app/checkpoint/model
    --tensor-parallel-size $GPU_COUNT
    --served-model-name google/gemma-4-E4B-it
    --max-num-seqs 16
    --max-model-len auto
    --limit-mm-per-prompt.image 1
    --gpu-memory-utilization 0.9
    --async-scheduling
    --trust-remote-code
    --enable-auto-tool-choice
    --enable-prefix-caching
    --reasoning-parser gemma4
    --tool-call-parser gemma4
    --load-format runai_streamer'
  readiness_endpoint: /health
  liveness_endpoint: /health
  predict_endpoint: /v1/chat/completions
  server_port: 8000
environment_variables:
  VLLM_LOGGING_LEVEL: WARNING
  VLLM_ENGINE_READY_TIMEOUT_S: "3600"
resources:
  accelerator: H100
  use_gpu: true
runtime:
  health_checks:
    startup_threshold_seconds: 1800
    restart_threshold_seconds: 1200
    stop_traffic_threshold_seconds: 120
  predict_concurrency: 16

Flags

The start_command passes these flags to the engine. Each one controls a runtime or serving behavior:

Flag	Value	What it does
`--tensor-parallel-size`	`$GPU_COUNT`	Number of GPUs to shard the model across.
`--max-num-seqs`	`16`	Maximum number of concurrent sequences in the batch.
`--max-model-len`	`auto`	Maximum context length (tokens) the server accepts per request.
`--limit-mm-per-prompt.image`	`1`	Maximum number of image inputs per prompt.
`--gpu-memory-utilization`	`0.9`	Fraction of GPU memory vLLM may use for weights and KV cache.
`--async-scheduling`	(no value)	Overlap scheduling with GPU execution to hide scheduler latency.
`--trust-remote-code`	(no value)	Execute model-specific Python from the checkpoint (required for many Qwen, Phi, and custom architectures).
`--enable-auto-tool-choice`	(no value)	Let the model choose when to call tools without requiring `tool_choice: "required"`.
`--enable-prefix-caching`	(no value)	Reuse KV cache across requests that share a prefix.
`--reasoning-parser`	`gemma4`	Server-side parser that separates reasoning output into `reasoning_content`.
`--tool-call-parser`	`gemma4`	Server-side parser that emits structured `tool_calls` on the response.
`--load-format`	`runai_streamer`	Weight loading backend. runai_streamer: Stream weights from object storage without materializing to disk.

Deploy

Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:

baseten model push

uvx truss push

You should see output similar to:

Pushing model "gemma-4-E4B-it-latency"...
Uploading model...
Uploaded model in 0s
✨ Model gemma-4-E4B-it-latency was successfully pushed ✨

  Model:       gemma-4-E4B-it-latency (abc1d2ef)
  Deployment:  xyz123
  Environment: production

🪵 View logs:
   deployment:   baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
   environment:  baseten model environment logs --model-id abc1d2ef --environment production  (once deployed)
   app:          https://app.baseten.co/models/abc1d2ef/logs/xyz123

🚀 Invoke your model:
   URL:  https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
   CLI:  baseten model predict --model-id abc1d2ef

Call the model

Your deployment serves an OpenAI-compatible API.Now call your deployment to run inference:

Python
cURL

main.py

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["BASETEN_API_KEY"],
    base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)

response = client.chat.completions.create(
    model="google/gemma-4-E4B-it",
    messages=[
        {"role": "user", "content": "What is machine learning?"}
    ],
)

print(response.choices[0].message.content)

curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '{
    "model": "google/gemma-4-E4B-it",
    "messages": [
      {"role": "user", "content": "What is machine learning?"}
    ]
  }'

The server parses the model’s chain of thought into a separate reasoning_content field on the response. Read it alongside the final answer:

response = client.chat.completions.create(
    model="google/gemma-4-E4B-it",
    messages=[
        {"role": "user", "content": "How many r's in strawberry?"}
    ],
)
print(response.choices[0].message.reasoning_content)  # chain of thought
print(response.choices[0].message.content)            # final answer

To let the model call tools, pass a tools array. The server returns structured tool_calls on the response:

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "parameters": {
            "type": "object",
            "properties": {"location": {"type": "string"}},
            "required": ["location"],
        },
    },
}]

response = client.chat.completions.create(
    model="google/gemma-4-E4B-it",
    messages=[
        {"role": "user", "content": "What's the weather in Paris?"}
    ],
    tools=tools,
)
print(response.choices[0].message.tool_calls)

google/gemma-4-26B-A4B-it is a 26B-parameter MoE model (4B active per token) with up to 256K context.This preset serves Gemma 4 26B A4B on H100:2 with FP8 dynamic quantization.

Hardware

H100 × 2

Engine

vLLM (0.22.0-cu129 build)

Context

256K

Concurrency

Write the config

Create and move into the project directory:

mkdir gemma-4-26B-A4B-it-latency && cd gemma-4-26B-A4B-it-latency

Then create a file named config.yaml and paste the following:

config.yaml

model_name: model:gemma-4-26B-A4B-it preset:latency
model_metadata:
  description: >-
    Gemma 4 multimodal instruct (26B MOE FP8 dynamique), speculative decoding Eagle3 using vLLM.
  repo_id: RedHatAI/gemma-4-26B-A4B-it-FP8-Dynamic
  example_model_input:
    model: google/gemma-4-26B-A4B-it
    messages:
      - role: user
        content:
          - type: text
            text: "Describe this image in one sentence."
          - type: image_url
            image_url:
              url: "https://picsum.photos/id/237/200/300"
    stream: true
    max_tokens: 512
    temperature: 1.0
  tags:
    - openai-compatible
base_image:
  image: vllm/vllm-openai:v0.22.0-cu129
weights:
  - source: "hf://RedHatAI/gemma-4-26B-A4B-it-FP8-Dynamic@ed35d7abe5d940da41b4ff06eb482feb0be8cb44"
    mount_location: "/app/checkpoint/model"
  - source: "hf://RedHatAI/gemma-4-26B-A4B-it-speculator.eagle3@226b2692d48029aae52adc1d0021a9d58709d33c"
    mount_location: "/app/checkpoint/speculator"
docker_server:
  start_command: >-
    sh -c 'GPU_COUNT=$(nvidia-smi --list-gpus | wc -l) && vllm serve /app/checkpoint/model
    --tensor-parallel-size $GPU_COUNT
    --served-model-name google/gemma-4-26B-A4B-it
    --max-num-seqs 16
    --max-model-len auto
    --limit-mm-per-prompt.image 1
    --gpu-memory-utilization 0.9
    --enable-prefix-caching
    --speculative-config.model /app/checkpoint/speculator
    --speculative-config.num_speculative_tokens 3
    --speculative-config.method eagle3
    --trust-remote-code
    --enable-auto-tool-choice
    --reasoning-parser gemma4
    --tool-call-parser gemma4
    --load-format runai_streamer'
  readiness_endpoint: /health
  liveness_endpoint: /health
  predict_endpoint: /v1/chat/completions
  server_port: 8000
environment_variables:
  VLLM_LOGGING_LEVEL: WARNING
  VLLM_ENGINE_READY_TIMEOUT_S: "3600"
resources:
  accelerator: H100:2
  use_gpu: true
runtime:
  health_checks:
    startup_threshold_seconds: 1800
    restart_threshold_seconds: 1200
    stop_traffic_threshold_seconds: 120
  predict_concurrency: 16

Flags

The start_command passes these flags to the engine. Each one controls a runtime or serving behavior:

Flag	Value	What it does
`--tensor-parallel-size`	`$GPU_COUNT`	Number of GPUs to shard the model across.
`--max-num-seqs`	`16`	Maximum number of concurrent sequences in the batch.
`--max-model-len`	`auto`	Maximum context length (tokens) the server accepts per request.
`--limit-mm-per-prompt.image`	`1`	Maximum number of image inputs per prompt.
`--gpu-memory-utilization`	`0.9`	Fraction of GPU memory vLLM may use for weights and KV cache.
`--enable-prefix-caching`	(no value)	Reuse KV cache across requests that share a prefix.
`--speculative-config.model`	`/app/checkpoint/speculator`	Path to the draft speculator checkpoint (HF repo or local mount path).
`--speculative-config.num_speculative_tokens`	`3`	Number of tokens the draft speculator proposes per step.
`--speculative-config.method`	`eagle3`	Speculative decoding method. eagle3: EAGLE v3 speculative decoding.
`--trust-remote-code`	(no value)	Execute model-specific Python from the checkpoint (required for many Qwen, Phi, and custom architectures).
`--enable-auto-tool-choice`	(no value)	Let the model choose when to call tools without requiring `tool_choice: "required"`.
`--reasoning-parser`	`gemma4`	Server-side parser that separates reasoning output into `reasoning_content`.
`--tool-call-parser`	`gemma4`	Server-side parser that emits structured `tool_calls` on the response.
`--load-format`	`runai_streamer`	Weight loading backend. runai_streamer: Stream weights from object storage without materializing to disk.

Deploy

Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:

baseten model push

uvx truss push

You should see output similar to:

Pushing model "gemma-4-26B-A4B-it-latency"...
Uploading model...
Uploaded model in 0s
✨ Model gemma-4-26B-A4B-it-latency was successfully pushed ✨

  Model:       gemma-4-26B-A4B-it-latency (abc1d2ef)
  Deployment:  xyz123
  Environment: production

🪵 View logs:
   deployment:   baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
   environment:  baseten model environment logs --model-id abc1d2ef --environment production  (once deployed)
   app:          https://app.baseten.co/models/abc1d2ef/logs/xyz123

🚀 Invoke your model:
   URL:  https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
   CLI:  baseten model predict --model-id abc1d2ef

Call the model

Your deployment serves an OpenAI-compatible API.Now call your deployment to run inference:

Python
cURL

main.py

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["BASETEN_API_KEY"],
    base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)

response = client.chat.completions.create(
    model="google/gemma-4-26B-A4B-it",
    messages=[
        {"role": "user", "content": "What is machine learning?"}
    ],
)

print(response.choices[0].message.content)

curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '{
    "model": "google/gemma-4-26B-A4B-it",
    "messages": [
      {"role": "user", "content": "What is machine learning?"}
    ]
  }'

The server parses the model’s chain of thought into a separate reasoning_content field on the response. Read it alongside the final answer:

response = client.chat.completions.create(
    model="google/gemma-4-26B-A4B-it",
    messages=[
        {"role": "user", "content": "How many r's in strawberry?"}
    ],
)
print(response.choices[0].message.reasoning_content)  # chain of thought
print(response.choices[0].message.content)            # final answer

To let the model call tools, pass a tools array. The server returns structured tool_calls on the response:

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "parameters": {
            "type": "object",
            "properties": {"location": {"type": "string"}},
            "required": ["location"],
        },
    },
}]

response = client.chat.completions.create(
    model="google/gemma-4-26B-A4B-it",
    messages=[
        {"role": "user", "content": "What's the weather in Paris?"}
    ],
    tools=tools,
)
print(response.choices[0].message.tool_calls)

google/gemma-4-31B-it is a 31B-parameter dense model with up to 256K context.This variant ships in 2 presets tuned for different goals: Latency for lowest time-to-first-token, and Throughput for highest tokens per second. Pick the tab that matches your workload.

Latency
Throughput

This preset serves Gemma 4 31B on H100:2 with FP8 block quantization.

Hardware

H100 × 2

Engine

vLLM 0.31.0

Context

256K

Concurrency

Write the config

Create and move into the project directory:

mkdir gemma-4-31B-it-latency && cd gemma-4-31B-it-latency

Then create a file named config.yaml and paste the following:

config.yaml

model_name: model:gemma-4-31B-it preset:latency
model_metadata:
  description: >-
    Gemma 4 multimodal instruct (FP8), OpenAI-compatible chat with vision using vLLM.
  repo_id: RedHatAI/gemma-4-31B-it-FP8-block
  example_model_input:
    model: google/gemma-4-31B-it
    messages:
      - role: user
        content:
          - type: text
            text: "Describe this image in one sentence."
          - type: image_url
            image_url:
              url: "https://picsum.photos/id/237/200/300"
    stream: true
    max_tokens: 512
    temperature: 1.0
  tags:
    - openai-compatible
base_image:
  image: vllm/vllm-openai:v0.31.0
weights:
  - source: "hf://RedHatAI/gemma-4-31B-it-FP8-block@d7242548c457ab4b45bd3adb8937f2659af4739d"
    mount_location: "/app/checkpoint/model"
  - source: "hf://RedHatAI/gemma-4-31B-it-speculator.eagle3@28a1c8b4bb64dbaee883ba35341841138bdf1fe3"
    mount_location: "/app/checkpoint/speculator"
docker_server:
  start_command: >-
    sh -c 'GPU_COUNT=$(nvidia-smi --list-gpus | wc -l) && vllm serve /app/checkpoint/model
    --tensor-parallel-size $GPU_COUNT
    --served-model-name google/gemma-4-31B-it
    --max-num-seqs 16
    --max-model-len auto
    --limit-mm-per-prompt.image 1
    --gpu-memory-utilization 0.9
    --enable-prefix-caching
    --speculative-config.model /app/checkpoint/speculator
    --speculative-config.num_speculative_tokens 3
    --speculative-config.method eagle3
    --trust-remote-code
    --enable-auto-tool-choice
    --reasoning-parser gemma4
    --tool-call-parser gemma4
    --load-format runai_streamer'
  readiness_endpoint: /health
  liveness_endpoint: /health
  predict_endpoint: /v1/chat/completions
  server_port: 8000
environment_variables:
  VLLM_LOGGING_LEVEL: WARNING
  VLLM_ENGINE_READY_TIMEOUT_S: "3600"
resources:
  accelerator: H100:2
  use_gpu: true
runtime:
  health_checks:
    startup_threshold_seconds: 1800
    restart_threshold_seconds: 1200
    stop_traffic_threshold_seconds: 120
  predict_concurrency: 16

Flags

The start_command passes these flags to the engine. Each one controls a runtime or serving behavior:

Flag	Value	What it does
`--tensor-parallel-size`	`$GPU_COUNT`	Number of GPUs to shard the model across.
`--max-num-seqs`	`16`	Maximum number of concurrent sequences in the batch.
`--max-model-len`	`auto`	Maximum context length (tokens) the server accepts per request.
`--limit-mm-per-prompt.image`	`1`	Maximum number of image inputs per prompt.
`--gpu-memory-utilization`	`0.9`	Fraction of GPU memory vLLM may use for weights and KV cache.
`--enable-prefix-caching`	(no value)	Reuse KV cache across requests that share a prefix.
`--speculative-config.model`	`/app/checkpoint/speculator`	Path to the draft speculator checkpoint (HF repo or local mount path).
`--speculative-config.num_speculative_tokens`	`3`	Number of tokens the draft speculator proposes per step.
`--speculative-config.method`	`eagle3`	Speculative decoding method. eagle3: EAGLE v3 speculative decoding.
`--trust-remote-code`	(no value)	Execute model-specific Python from the checkpoint (required for many Qwen, Phi, and custom architectures).
`--enable-auto-tool-choice`	(no value)	Let the model choose when to call tools without requiring `tool_choice: "required"`.
`--reasoning-parser`	`gemma4`	Server-side parser that separates reasoning output into `reasoning_content`.
`--tool-call-parser`	`gemma4`	Server-side parser that emits structured `tool_calls` on the response.
`--load-format`	`runai_streamer`	Weight loading backend. runai_streamer: Stream weights from object storage without materializing to disk.

Deploy

Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:

baseten model push

uvx truss push

You should see output similar to:

Pushing model "gemma-4-31B-it-latency"...
Uploading model...
Uploaded model in 0s
✨ Model gemma-4-31B-it-latency was successfully pushed ✨

  Model:       gemma-4-31B-it-latency (abc1d2ef)
  Deployment:  xyz123
  Environment: production

🪵 View logs:
   deployment:   baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
   environment:  baseten model environment logs --model-id abc1d2ef --environment production  (once deployed)
   app:          https://app.baseten.co/models/abc1d2ef/logs/xyz123

🚀 Invoke your model:
   URL:  https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
   CLI:  baseten model predict --model-id abc1d2ef

Call the model

Your deployment serves an OpenAI-compatible API.Now call your deployment to run inference:

Python
cURL

main.py

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["BASETEN_API_KEY"],
    base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)

response = client.chat.completions.create(
    model="google/gemma-4-31B-it",
    messages=[
        {"role": "user", "content": "What is machine learning?"}
    ],
)

print(response.choices[0].message.content)

curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '{
    "model": "google/gemma-4-31B-it",
    "messages": [
      {"role": "user", "content": "What is machine learning?"}
    ]
  }'

The server parses the model’s chain of thought into a separate reasoning_content field on the response. Read it alongside the final answer:

response = client.chat.completions.create(
    model="google/gemma-4-31B-it",
    messages=[
        {"role": "user", "content": "How many r's in strawberry?"}
    ],
)
print(response.choices[0].message.reasoning_content)  # chain of thought
print(response.choices[0].message.content)            # final answer

To let the model call tools, pass a tools array. The server returns structured tool_calls on the response:

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "parameters": {
            "type": "object",
            "properties": {"location": {"type": "string"}},
            "required": ["location"],
        },
    },
}]

response = client.chat.completions.create(
    model="google/gemma-4-31B-it",
    messages=[
        {"role": "user", "content": "What's the weather in Paris?"}
    ],
    tools=tools,
)
print(response.choices[0].message.tool_calls)

Hardware

RTX-PRO-6000

Engine

vLLM 0.22.1

Context

128K

Concurrency

Write the config

Create and move into the project directory:

mkdir gemma-4-31B-it && cd gemma-4-31B-it

Then create a file named config.yaml and paste the following:

config.yaml

model_name: model:gemma-4-31B-it preset:throughput
model_metadata:
  description: >-
    Gemma 4 multimodal instruct (NVFP4), OpenAI-compatible chat with vision using vLLM on RTX PRO 6000.
  repo_id: nvidia/Gemma-4-31B-IT-NVFP4
  example_model_input:
    model: google/gemma-4-31B-it
    messages:
      - role: user
        content:
          - type: text
            text: "Describe this image in one sentence."
          - type: image_url
            image_url:
              url: "https://picsum.photos/id/237/200/300"
    stream: true
    max_tokens: 512
    temperature: 1.0
  tags:
    - openai-compatible
base_image:
  image: vllm/vllm-openai:v0.22.1
weights:
  - source: "hf://nvidia/Gemma-4-31B-IT-NVFP4@main"
    mount_location: "/app/checkpoint/gemma"
docker_server:
  start_command: >-
    sh -c 'GPU_COUNT=$(nvidia-smi --list-gpus | wc -l) && vllm serve /app/checkpoint/gemma
    --tensor-parallel-size $GPU_COUNT
    --served-model-name google/gemma-4-31B-it
    --max-num-seqs 64
    --max-model-len 131072
    --kv-cache-dtype fp8
    --enable-chunked-prefill
    --enable-prefix-caching
    --trust-remote-code
    --enable-auto-tool-choice
    --reasoning-parser gemma4
    --tool-call-parser gemma4
    --load-format runai_streamer'
  readiness_endpoint: /health
  liveness_endpoint: /health
  predict_endpoint: /v1/chat/completions
  server_port: 8000
resources:
  instance_type: RTX-PRO-6000
runtime:
  health_checks:
    startup_threshold_seconds: 1800
    restart_threshold_seconds: 1200
    stop_traffic_threshold_seconds: 120
  predict_concurrency: 64

Flags

The start_command passes these flags to the engine. Each one controls a runtime or serving behavior:

Flag	Value	What it does
`--tensor-parallel-size`	`$GPU_COUNT`	Number of GPUs to shard the model across.
`--max-num-seqs`	`64`	Maximum number of concurrent sequences in the batch.
`--max-model-len`	`131072`	Maximum context length (tokens) the server accepts per request.
`--kv-cache-dtype`	`fp8`	KV cache numeric precision. fp8: ~2× KV cache density with negligible quality impact on most models.
`--enable-chunked-prefill`	(no value)	Process long prompts in chunks so decode requests keep running.
`--enable-prefix-caching`	(no value)	Reuse KV cache across requests that share a prefix.
`--trust-remote-code`	(no value)	Execute model-specific Python from the checkpoint (required for many Qwen, Phi, and custom architectures).
`--enable-auto-tool-choice`	(no value)	Let the model choose when to call tools without requiring `tool_choice: "required"`.
`--reasoning-parser`	`gemma4`	Server-side parser that separates reasoning output into `reasoning_content`.
`--tool-call-parser`	`gemma4`	Server-side parser that emits structured `tool_calls` on the response.
`--load-format`	`runai_streamer`	Weight loading backend. runai_streamer: Stream weights from object storage without materializing to disk.

Deploy

Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:

baseten model push

uvx truss push

You should see output similar to:

Pushing model "gemma-4-31B-it"...
Uploading model...
Uploaded model in 0s
✨ Model gemma-4-31B-it was successfully pushed ✨

  Model:       gemma-4-31B-it (abc1d2ef)
  Deployment:  xyz123
  Environment: production

🪵 View logs:
   deployment:   baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
   environment:  baseten model environment logs --model-id abc1d2ef --environment production  (once deployed)
   app:          https://app.baseten.co/models/abc1d2ef/logs/xyz123

🚀 Invoke your model:
   URL:  https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
   CLI:  baseten model predict --model-id abc1d2ef

Call the model

Your deployment serves an OpenAI-compatible API.Now call your deployment to run inference:

Python
cURL

main.py

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["BASETEN_API_KEY"],
    base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)

response = client.chat.completions.create(
    model="google/gemma-4-31B-it",
    messages=[
        {"role": "user", "content": "What is machine learning?"}
    ],
)

print(response.choices[0].message.content)

curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '{
    "model": "google/gemma-4-31B-it",
    "messages": [
      {"role": "user", "content": "What is machine learning?"}
    ]
  }'

The server parses the model’s chain of thought into a separate reasoning_content field on the response. Read it alongside the final answer:

response = client.chat.completions.create(
    model="google/gemma-4-31B-it",
    messages=[
        {"role": "user", "content": "How many r's in strawberry?"}
    ],
)
print(response.choices[0].message.reasoning_content)  # chain of thought
print(response.choices[0].message.content)            # final answer

To let the model call tools, pass a tools array. The server returns structured tool_calls on the response:

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "parameters": {
            "type": "object",
            "properties": {"location": {"type": "string"}},
            "required": ["location"],
        },
    },
}]

response = client.chat.completions.create(
    model="google/gemma-4-31B-it",
    messages=[
        {"role": "user", "content": "What's the weather in Paris?"}
    ],
    tools=tools,
)
print(response.choices[0].message.tool_calls)

​Setup

Hardware

Engine

Context

Concurrency

​Write the config

​Flags

​Deploy

​Call the model

Hardware

Engine

Context

Concurrency

​Write the config

​Flags

​Deploy

​Call the model

Hardware

Engine

Context

Concurrency

​Write the config

​Flags

​Deploy

​Call the model

Hardware

Engine

Context

Concurrency

​Write the config

​Flags

​Deploy

​Call the model

Hardware

Engine

Context

Concurrency

​Write the config

​Flags

​Deploy

​Call the model

​Next steps

Call your model

Autoscaling

Setup

Write the config

Flags

Deploy

Call the model

Write the config

Flags

Deploy

Call the model

Write the config

Flags

Deploy

Call the model

Write the config

Flags

Deploy

Call the model

Write the config

Flags

Deploy

Call the model

Next steps