GPT-OSS

Reasoning Tool calling Agentic Long context

Setup

Install the Baseten CLI and sign in, then install the OpenAI SDK.

Install and sign in to Baseten

macOS or Linux
Windows

Terminal

brew tap basetenlabs/baseten
brew trust --formula basetenlabs/baseten/baseten
brew install baseten

Download and extract the Windows archive for the latest Baseten CLI release:

Terminal

$release = Invoke-RestMethod https://api.github.com/repos/basetenlabs/baseten-cli/releases/latest
$asset = $release.assets | Where-Object { $_.name -like "*_windows_amd64.zip" }
Invoke-WebRequest $asset.browser_download_url -OutFile baseten.zip
Expand-Archive -Force baseten.zip .

Move baseten.exe to a directory on your PATH.

For other platforms or a specific version, see the Baseten CLI install reference.

baseten auth login

Install the OpenAI SDK

uv pip install openai

To install without Homebrew, see the Baseten CLI install reference. Pick the model you want to deploy. Each tab is a self-contained recipe.

20B
120B

openai/gpt-oss-20b is a 20B-parameter dense model with up to 128K context.This preset serves GPT-OSS 20B on a single H100 using the Harmony response format, tuned for low time-to-first-token.

Hardware

H100

Engine

TRT-LLM v2

Context

128K

Concurrency

Write the config

Create and move into the project directory:

mkdir gpt-oss-20b-latency && cd gpt-oss-20b-latency

Then create a file named config.yaml and paste the following:

config.yaml

model_name: "model:gpt-oss-20b preset:latency"
build_commands:
  - python -c 'from openai_harmony import load_harmony_encoding; load_harmony_encoding("HarmonyGptOss")'
model_metadata:
  repo_id: openai/gpt-oss-20b
  example_model_input:
    {
      "model": "openai/gpt-oss-20b",
      "messages":
        [
          {
            "role": "user",
            "content": "Given an array of integers nums and an integer target, return indices of the two numbers such that they add up to target. You may assume that each input would have exactly one solution, and you may not use the same element twice. You can return the answer in any order. class Solution: def twoSum(self, nums: List[int], target: int) -> List[int]:",
          },
        ],
      "stream": true,
      "max_tokens": 4096,
      "temperature": 0.5,
    }
  tags:
    - openai-compatible
resources:
  accelerator: H100
  cpu: "1"
  memory: 10Gi
  use_gpu: true
weights:
  - source: "hf://openai/gpt-oss-20b@main"
    mount_location: "/app/model_cache/trt_model"
trt_llm:
  build:
    checkpoint_repository:
      repo: michaelfeil/empty-model
      revision: main
      source: HF
  inference_stack: v2
  runtime:
    enable_chunked_prefill: true
    max_batch_size: 64
    max_num_tokens: 8192
    max_seq_len: 131072
    patch_kwargs:
      model_path: /app/model_cache/trt_model
      chat_processor: harmony
      moe_expert_parallel_size: 1
      backend: pytorch
      cuda_graph_config:
        enable_padding: true
      disable_overlap_scheduler: 1
      enable_autotuner: 0
      enable_iter_perf_stats: 0
      enable_trtllm_sampler: 1
      guided_decoding_backend: xgrammar
      kv_cache_config:
        enable_block_reuse: true
        free_gpu_memory_fraction: 0.8
        event_buffer_max_size: 1024
      max_beam_width: 1
      max_input_len: 131072
      model_level_stop_words:
        - "<|call|>"
      tokenizer_limit_length: 131072
      trust_remote_code: 1
      moe_config:
        backend: CUTLASS
    served_model_name: openai/gpt-oss-20b
    tensor_parallel_size: 1
  version_overrides:
    v2_llm_version: null

Key parameters

Baseten Inference Stack (BIS) reads these fields from the trt_llm block. Each one shapes how the engine is built and served:

Parameter	Value
Tensor parallel size	`1`
Max sequence length	`131072`
Max batch size	`64`
Max batched tokens	`8192`
Chunked prefill	`enabled`
Inference stack	`v2`
Served model name	`openai/gpt-oss-20b`

Deploy

Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:

baseten model push

uvx truss push

You should see output similar to:

Pushing model "gpt-oss-20b-latency"...
Uploading model...
Uploaded model in 0s
✨ Model gpt-oss-20b-latency was successfully pushed ✨

  Model:       gpt-oss-20b-latency (abc1d2ef)
  Deployment:  xyz123
  Environment: production

🪵 View logs:
   deployment:   baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
   environment:  baseten model environment logs --model-id abc1d2ef --environment production  (once deployed)
   app:          https://app.baseten.co/models/abc1d2ef/logs/xyz123

🚀 Invoke your model:
   URL:  https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
   CLI:  baseten model predict --model-id abc1d2ef

baseten model push prints your model ID (abc1d2ef in the example). The examples below use it wherever you see {model_id}, and read your API key from the BASETEN_API_KEY environment variable.

Call the model

Your deployment serves an OpenAI-compatible API.Now call your deployment to run inference:

Python
cURL

main.py

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["BASETEN_API_KEY"],
    base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)

response = client.chat.completions.create(
    model="openai/gpt-oss-20b",
    messages=[
        {"role": "user", "content": "What is machine learning?"}
    ],
)

print(response.choices[0].message.content)

curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '{
    "model": "openai/gpt-oss-20b",
    "messages": [
      {"role": "user", "content": "What is machine learning?"}
    ]
  }'

openai/gpt-oss-120b is a 120B-parameter MoE model with up to 128K context.This variant ships in 3 presets tuned for different goals: Latency for lowest time-to-first-token, Throughput for highest tokens per second, and H100 Throughput for high throughput on H100 hardware. Pick the tab that matches your workload.

Latency
Throughput
H100 Throughput

Hardware

B200 × 4

Context

128K

Write the config

Create and move into the project directory:

mkdir gpt-oss-120b && cd gpt-oss-120b

Then create a file named config.yaml and paste the following:

config.yaml

# Public-artifact port of the Model API production config for openai/gpt-oss-120b
# (mapi/openai/gpt-oss-120b/config.yaml). Every bis_llm field, including the
# `gpuTRTImage` serving image and the `bis_llm.version` pin, is kept verbatim from
# production. The platform-default image that `engine_backend: trtllm` alone selects
# rejects `chat_processor: harmony`, so the production image is pinned. The checkpoint
# is public and ungated (Apache-2.0), so no HF auth secret. model_name follows the repo
# convention and resources use instance_type.
model_name: model:gpt-oss-120b preset:latency
model_metadata:
  repo_id: openai/gpt-oss-120b
  tags:
  - openai-compatible
  example_model_input:
    model: openai/gpt-oss-120b
    stream: true
    messages:
    - role: system
      content: You are a helpful assistant.
    - role: user
      content: Summarize the rules of chess in three sentences.
bis_llm:
  config:
    model_name: openai/gpt-oss-120b
    gpuTRTImage: baseten/bis-llm-trtllm:green-mp--ec11764bde-4884584f6-49e724f969
    engine_config:
      backend: pytorch
      moe_config:
        backend: TRTLLM
      max_seq_len: 131072
      sampler_type: TRTLLMSampler
      max_input_len: 131072
      max_batch_size: 64
      max_beam_width: 1
      max_num_tokens: 32768
      kv_cache_config:
        host_cache_size: 120000000000
        enable_block_reuse: true
        enable_partial_reuse: false
        event_buffer_max_size: 16384
        free_gpu_memory_fraction: 0.8
      stream_interval: 4
      enable_autotuner: true
      cuda_graph_config:
        batch_sizes:
        - 1
        - 2
        - 4
        - 8
        - 16
        - 24
        - 32
        - 40
        - 48
        - 56
        - 64
        enable_padding: true
        max_batch_size: 64
      trust_remote_code: true
      enable_chunked_prefill: true
      enable_iter_perf_stats: true
      guided_decoding_backend: xgrammar
      moe_expert_parallel_size: 2
      disable_overlap_scheduler: false
    chat_processor: harmony
    checkpoint_name: openai/gpt-oss-120b
    b10_routing_config:
      algo_selector: B10
      durable_kv_events: false
      enable_worker_tree: true
      rate_limit_enabled: true
      router_queue_policy: wspt
      router_cache_miss_weight: 0.02
      router_cache_miss_min_isl: 1024
      rate_limit_batch_multiplier: 2
      router_active_request_weight: 2
      router_prefill_token_discount: 0.3
    tensor_parallel_size: 4
    model_level_stop_words:
    - <|call|>
    tokenizer_limit_length: 131072
    additional_environment_variables:
      Worker:
        TRTLLM_ENABLE_PDL: '1'
        MIMALLOC_PURGE_DELAY: '0'
        PYTORCH_CUDA_ALLOC_CONF: expandable_segments:True
        NCCL_GRAPH_MIXING_SUPPORT: '0'
        BAD_TOKEN_ID_SEQ_CHECK_ENABLED: '1'
        TLLM_NUMA_AWARE_WORKER_AFFINITY: '1'
  version: 0.0.1-20261001221953-8f302211
environment_variables: {}
resources:
  instance_type: B200:4
weights:
# Public, ungated checkpoint, pinned to the revision `@main` resolved to.
- source: hf://openai/gpt-oss-120b@b5c939de8f754692c1647ca79fbf85e8c1e70f8a
  mount_location: /models/gpt-oss-120b

Deploy

Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:

baseten model push

uvx truss push

You should see output similar to:

Pushing model "gpt-oss-120b"...
Uploading model...
Uploaded model in 0s
✨ Model gpt-oss-120b was successfully pushed ✨

  Model:       gpt-oss-120b (abc1d2ef)
  Deployment:  xyz123
  Environment: production

🪵 View logs:
   deployment:   baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
   environment:  baseten model environment logs --model-id abc1d2ef --environment production  (once deployed)
   app:          https://app.baseten.co/models/abc1d2ef/logs/xyz123

🚀 Invoke your model:
   URL:  https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
   CLI:  baseten model predict --model-id abc1d2ef

Call the model

Your deployment serves an OpenAI-compatible API.Now call your deployment to run inference:

Python
cURL

main.py

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["BASETEN_API_KEY"],
    base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)

response = client.chat.completions.create(
    model="openai/gpt-oss-120b",
    messages=[
        {"role": "user", "content": "What is machine learning?"}
    ],
)

print(response.choices[0].message.content)

curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '{
    "model": "openai/gpt-oss-120b",
    "messages": [
      {"role": "user", "content": "What is machine learning?"}
    ]
  }'

This preset serves GPT-OSS 120B on B200:4 through Baseten Inference Stack with a TensorRT-LLM PyTorch backend, the same configuration that serves the baseten/gpt-oss-120b library listing, optimized for maximum throughput on Blackwell.

Hardware

B200 × 4

Context

128K

Write the config

Create and move into the project directory:

mkdir gpt-oss-120b-throughput && cd gpt-oss-120b-throughput

Then create a file named config.yaml and paste the following:

config.yaml

# Registry copy of the config serving the Baseten library listing
# baseten/gpt-oss-120b (BIS-LLM, TRT-LLM PyTorch backend behind Dynamo cache-aware routing). The bis_llm block,
# environment, and weights values are the deployed values from BIS Config Registry
# composer `gpt-oss-120b` (bcr_version 1.0.5) verbatim; model_name
# follows the repo convention and resources use instance_type.
model_name: "model:gpt-oss-120b preset:bis-throughput"
model_metadata:
  repo_id: openai/gpt-oss-120b
  tags:
    - openai-compatible
  example_model_input:
    messages:
      - role: user
        content: "What is the meaning of life?"
    stream: true
    model: openai/gpt-oss-120b
    max_tokens: 1024
bis_llm:
  config:
    additional_environment_variables:
      Worker:
        BAD_TOKEN_ID_SEQ_CHECK_ENABLED: "1"
        MIMALLOC_PURGE_DELAY: "0"
        NCCL_GRAPH_MIXING_SUPPORT: "0"
        PYTORCH_CUDA_ALLOC_CONF: "expandable_segments:True"
        TLLM_NUMA_AWARE_WORKER_AFFINITY: "1"
        TRTLLM_ENABLE_PDL: "1"
        TRTLLM_SERVER_DISABLE_GC: "1"
        TRTLLM_WORKER_DISABLE_GC: "1"
    b10_routing_config:
      algo_selector: B10
      durable_kv_events: false
      enable_worker_tree: true
      rate_limit_batch_multiplier: 2
      rate_limit_enabled: true
      router_active_request_weight: 2
      router_cache_miss_min_isl: 1024
      router_cache_miss_weight: 0.02
      router_prefill_token_discount: 0.3
      router_queue_policy: wspt
    chat_processor: harmony
    checkpoint_name: openai/gpt-oss-120b
    engine_config:
      backend: pytorch
      cuda_graph_config:
        batch_sizes:
          - 1
          - 2
          - 4
          - 8
          - 16
          - 24
          - 32
          - 40
          - 48
          - 56
          - 64
        enable_padding: true
        max_batch_size: 64
      disable_overlap_scheduler: false
      enable_autotuner: true
      enable_chunked_prefill: true
      enable_iter_perf_stats: true
      guided_decoding_backend: xgrammar
      kv_cache_config:
        enable_block_reuse: true
        enable_partial_reuse: false
        event_buffer_max_size: 16384
        free_gpu_memory_fraction: 0.9
        host_cache_size: 120000000000
      max_batch_size: 64
      max_beam_width: 1
      max_input_len: 131072
      max_num_tokens: 16384
      max_seq_len: 131072
      moe_config:
        backend: TRTLLM
      moe_expert_parallel_size: 4
      sampler_type: TRTLLMSampler
      stream_interval: 4
      trust_remote_code: true
    gpuTRTImage: "baseten/dynamo-cache-aware-routing:bis-4nf059--trtllm-dfe755294d-c409f2fff-b690c11391"
    model_level_stop_words:
      - <|call|>
    model_name: openai/gpt-oss-120b
    model_path: /models/gpt-oss-120b
    reasoning_effort_policy:
      allowed:
        - low
        - medium
        - high
      default: medium
    tensor_parallel_size: 4
    tokenizer_limit_length: 131072
  version: 0.0.1-20260804033840-1d68536d
environment_variables: {}
resources:
  instance_type: "B200:4"
weights:
  # Public, ungated checkpoint (Apache-2.0); the auth secret is dropped. Pinned to
  # the revision `@main` resolved to (unchanged since 2025-08-26).
  - mount_location: /models/gpt-oss-120b
    source: "hf://openai/gpt-oss-120b@b5c939de8f754692c1647ca79fbf85e8c1e70f8a"

Deploy

Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:

baseten model push

uvx truss push

You should see output similar to:

Pushing model "gpt-oss-120b-throughput"...
Uploading model...
Uploaded model in 0s
✨ Model gpt-oss-120b-throughput was successfully pushed ✨

  Model:       gpt-oss-120b-throughput (abc1d2ef)
  Deployment:  xyz123
  Environment: production

🪵 View logs:
   deployment:   baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
   environment:  baseten model environment logs --model-id abc1d2ef --environment production  (once deployed)
   app:          https://app.baseten.co/models/abc1d2ef/logs/xyz123

🚀 Invoke your model:
   URL:  https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
   CLI:  baseten model predict --model-id abc1d2ef

Call the model

Your deployment serves an OpenAI-compatible API.Now call your deployment to run inference:

Python
cURL

main.py

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["BASETEN_API_KEY"],
    base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)

response = client.chat.completions.create(
    model="openai/gpt-oss-120b",
    messages=[
        {"role": "user", "content": "What is machine learning?"}
    ],
)

print(response.choices[0].message.content)

curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '{
    "model": "openai/gpt-oss-120b",
    "messages": [
      {"role": "user", "content": "What is machine learning?"}
    ]
  }'

This preset serves GPT-OSS 120B on H100:4 for deployments that don’t have Blackwell capacity.

Hardware

H100 × 4

Engine

vLLM (0.28.0-cu129 build)

Context

16K

Concurrency

256

Write the config

Create and move into the project directory:

mkdir gpt-oss-120b-h100-throughput && cd gpt-oss-120b-h100-throughput

Then create a file named config.yaml and paste the following:

config.yaml

model_name: "model:gpt-oss-120b preset:h100-throughput"
model_metadata:
  description: >-
    GPT-OSS 120B on vLLM H100 × 4 throughput; weights from BDN, async scheduling and prefix caching.
  repo_id: openai/gpt-oss-120b
  tags:
    - openai-compatible
  example_model_input:
    messages:
      - role: system
        content: "You are a helpful assistant."
      - role: user
        content: "Write FizzBuzz in Python"
    stream: true
    model: "openai/gpt-oss-120b"
    max_tokens: 4096
    temperature: 0.5
base_image:
  image: vllm/vllm-openai:v0.28.0-cu129
build_commands:
  - mkdir -p /opt/tiktoken
  - curl -fsSL https://openaipublic.blob.core.windows.net/encodings/o200k_base.tiktoken -o /opt/tiktoken/o200k_base.tiktoken
  - curl -fsSL https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken -o /opt/tiktoken/cl100k_base.tiktoken
weights:
  - source: "hf://openai/gpt-oss-120b@b5c939de8f754692c1647ca79fbf85e8c1e70f8a"
    mount_location: "/app/checkpoint/model"
    ignore_patterns: ["original/*", "metal/model.bin"]
environment_variables:
  TIKTOKEN_ENCODINGS_BASE: "/opt/tiktoken"
  TIKTOKEN_RS_CACHE_DIR: "/opt/tiktoken"
  VLLM_LOGGING_LEVEL: WARNING
  VLLM_ENGINE_READY_TIMEOUT_S: "3600"
docker_server:
  start_command: >-
    sh -c 'GPU_COUNT=$(nvidia-smi --list-gpus | wc -l) && vllm serve /app/checkpoint/model
    --host 0.0.0.0
    --port 8000
    --served-model-name openai/gpt-oss-120b
    --tensor-parallel-size $GPU_COUNT
    --gpu-memory-utilization 0.90
    --max-model-len 16384
    --max-num-batched-tokens 16384
    --max-num-seqs 256
    --stream-interval 20
    --enable-chunked-prefill
    --enable-prefix-caching
    --async-scheduling
    --trust-remote-code
    --load-format runai_streamer'
  readiness_endpoint: /health
  liveness_endpoint: /health
  predict_endpoint: /v1/chat/completions
  server_port: 8000
resources:
  accelerator: H100:4
  use_gpu: true
runtime:
  predict_concurrency: 256
  health_checks:
    startup_threshold_seconds: 1800
    restart_threshold_seconds: 1200
    stop_traffic_threshold_seconds: 120

Flags

The start_command passes these flags to the engine. Each one controls a runtime or serving behavior:

Flag	Value	What it does
`--tensor-parallel-size`	`$GPU_COUNT`	Number of GPUs to shard the model across.
`--gpu-memory-utilization`	`0.90`	Fraction of GPU memory vLLM may use for weights and KV cache.
`--max-model-len`	`16384`	Maximum context length (tokens) the server accepts per request.
`--max-num-batched-tokens`	`16384`	Maximum total tokens processed per scheduler step.
`--max-num-seqs`	`256`	Maximum number of concurrent sequences in the batch.
`--stream-interval`	`20`	Tokens emitted per streaming chunk.
`--enable-chunked-prefill`	(no value)	Process long prompts in chunks so decode requests keep running.
`--enable-prefix-caching`	(no value)	Reuse KV cache across requests that share a prefix.
`--async-scheduling`	(no value)	Overlap scheduling with GPU execution to hide scheduler latency.
`--trust-remote-code`	(no value)	Execute model-specific Python from the checkpoint (required for many Qwen, Phi, and custom architectures).
`--load-format`	`runai_streamer`	Weight loading backend. runai_streamer: Stream weights from object storage without materializing to disk.

Deploy

Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:

baseten model push

uvx truss push

You should see output similar to:

Pushing model "gpt-oss-120b-h100-throughput"...
Uploading model...
Uploaded model in 0s
✨ Model gpt-oss-120b-h100-throughput was successfully pushed ✨

  Model:       gpt-oss-120b-h100-throughput (abc1d2ef)
  Deployment:  xyz123
  Environment: production

🪵 View logs:
   deployment:   baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
   environment:  baseten model environment logs --model-id abc1d2ef --environment production  (once deployed)
   app:          https://app.baseten.co/models/abc1d2ef/logs/xyz123

🚀 Invoke your model:
   URL:  https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
   CLI:  baseten model predict --model-id abc1d2ef

Call the model

Your deployment serves an OpenAI-compatible API.Now call your deployment to run inference:

Python
cURL

main.py

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["BASETEN_API_KEY"],
    base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)

response = client.chat.completions.create(
    model="openai/gpt-oss-120b",
    messages=[
        {"role": "user", "content": "What is machine learning?"}
    ],
)

print(response.choices[0].message.content)

curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '{
    "model": "openai/gpt-oss-120b",
    "messages": [
      {"role": "user", "content": "What is machine learning?"}
    ]
  }'

Examples

Models

Engines

Custom Docker servers

Custom Python models

Chains

Baseten for Model Labs

Setup

Hardware

Engine

Context

Concurrency

Write the config

Key parameters

Deploy

Call the model

Hardware

Context

Write the config

Deploy

Call the model

Hardware

Context

Write the config

Deploy

Call the model

Hardware

Engine

Context

Concurrency

Write the config

Flags

Deploy

Call the model

Next steps

Call your model

Autoscaling

​Setup

Hardware

Engine

Context

Concurrency

​Write the config

​Key parameters

​Deploy

​Call the model

Hardware

Context

​Write the config

​Deploy

​Call the model

Hardware

Context

​Write the config

​Deploy

​Call the model

Hardware

Engine

Context

Concurrency

​Write the config

​Flags

​Deploy

​Call the model

​Next steps

Call your model

Autoscaling

Setup

Write the config

Key parameters

Deploy

Call the model

Write the config

Deploy

Call the model

Write the config

Deploy

Call the model

Write the config

Flags

Deploy

Call the model

Next steps