To install without Homebrew, see the Baseten CLI install reference.Pick the model you want to deploy. Each tab is a self-contained recipe.
20B
120B
openai/gpt-oss-20b is a 20B-parameter dense model with up to 128K context.This preset serves GPT-OSS 20B on a single H100 using the Harmony response format, tuned for low time-to-first-token.
Hardware
H100
Engine
TRT-LLM v2
Context
128K
Concurrency
64
Write the config
Create and move into the project directory:
mkdir gpt-oss-20b-latency && cd gpt-oss-20b-latency
Then create a file named config.yaml and paste the following:
config.yaml
model_name: "model:gpt-oss-20b preset:latency"build_commands: - python -c 'from openai_harmony import load_harmony_encoding; load_harmony_encoding("HarmonyGptOss")'model_metadata: repo_id: openai/gpt-oss-20b example_model_input: { "model": "openai/gpt-oss-20b", "messages": [ { "role": "user", "content": "Given an array of integers nums and an integer target, return indices of the two numbers such that they add up to target. You may assume that each input would have exactly one solution, and you may not use the same element twice. You can return the answer in any order. class Solution: def twoSum(self, nums: List[int], target: int) -> List[int]:", }, ], "stream": true, "max_tokens": 4096, "temperature": 0.5, } tags: - openai-compatibleresources: accelerator: H100 cpu: "1" memory: 10Gi use_gpu: trueweights: - source: "hf://openai/gpt-oss-20b@main" mount_location: "/app/model_cache/trt_model"trt_llm: build: checkpoint_repository: repo: michaelfeil/empty-model revision: main source: HF inference_stack: v2 runtime: enable_chunked_prefill: true max_batch_size: 64 max_num_tokens: 8192 max_seq_len: 131072 patch_kwargs: model_path: /app/model_cache/trt_model chat_processor: harmony moe_expert_parallel_size: 1 backend: pytorch cuda_graph_config: enable_padding: true disable_overlap_scheduler: 1 enable_autotuner: 0 enable_iter_perf_stats: 0 enable_trtllm_sampler: 1 guided_decoding_backend: xgrammar kv_cache_config: enable_block_reuse: true free_gpu_memory_fraction: 0.8 event_buffer_max_size: 1024 max_beam_width: 1 max_input_len: 131072 model_level_stop_words: - "<|call|>" tokenizer_limit_length: 131072 trust_remote_code: 1 moe_config: backend: CUTLASS served_model_name: openai/gpt-oss-20b tensor_parallel_size: 1 version_overrides: v2_llm_version: null
Key parameters
Baseten Inference Stack (BIS) reads these fields from the trt_llm block. Each one shapes how the engine is built and served:
Parameter
Value
Tensor parallel size
1
Max sequence length
131072
Max batch size
64
Max batched tokens
8192
Chunked prefill
enabled
Inference stack
v2
Served model name
openai/gpt-oss-20b
Deploy
Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:
baseten model push
uvx truss push
You should see output similar to:
Pushing model "gpt-oss-20b-latency"...Uploading model...Uploaded model in 0sβ¨ Model gpt-oss-20b-latency was successfully pushed β¨ Model: gpt-oss-20b-latency (abc1d2ef) Deployment: xyz123 Environment: productionπͺ΅ View logs: deployment: baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123 environment: baseten model environment logs --model-id abc1d2ef --environment production (once deployed) app: https://app.baseten.co/models/abc1d2ef/logs/xyz123π Invoke your model: URL: https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict CLI: baseten model predict --model-id abc1d2ef
baseten model push prints your model ID (abc1d2ef in the example). The examples below use it wherever you see {model_id}, and read your API key from the BASETEN_API_KEY environment variable.
Call the model
Your deployment serves an OpenAI-compatible API.Now call your deployment to run inference:
openai/gpt-oss-120b is a 120B-parameter MoE model with up to 128K context.This variant ships in 3 presets tuned for different goals: Latency for lowest time-to-first-token, Throughput for highest tokens per second, and H100 Throughput for high throughput on H100 hardware. Pick the tab that matches your workload.
Latency
Throughput
H100 Throughput
Hardware
B200 Γ 4
Context
128K
Write the config
Create and move into the project directory:
mkdir gpt-oss-120b && cd gpt-oss-120b
Then create a file named config.yaml and paste the following:
config.yaml
# Public-artifact port of the Model API production config for openai/gpt-oss-120b# (mapi/openai/gpt-oss-120b/config.yaml). Every bis_llm field, including the# `gpuTRTImage` serving image and the `bis_llm.version` pin, is kept verbatim from# production. The platform-default image that `engine_backend: trtllm` alone selects# rejects `chat_processor: harmony`, so the production image is pinned. The checkpoint# is public and ungated (Apache-2.0), so no HF auth secret. model_name follows the repo# convention and resources use instance_type.model_name: model:gpt-oss-120b preset:latencymodel_metadata: repo_id: openai/gpt-oss-120b tags: - openai-compatible example_model_input: model: openai/gpt-oss-120b stream: true messages: - role: system content: You are a helpful assistant. - role: user content: Summarize the rules of chess in three sentences.bis_llm: config: model_name: openai/gpt-oss-120b gpuTRTImage: baseten/bis-llm-trtllm:green-mp--ec11764bde-4884584f6-49e724f969 engine_config: backend: pytorch moe_config: backend: TRTLLM max_seq_len: 131072 sampler_type: TRTLLMSampler max_input_len: 131072 max_batch_size: 64 max_beam_width: 1 max_num_tokens: 32768 kv_cache_config: host_cache_size: 120000000000 enable_block_reuse: true enable_partial_reuse: false event_buffer_max_size: 16384 free_gpu_memory_fraction: 0.8 stream_interval: 4 enable_autotuner: true cuda_graph_config: batch_sizes: - 1 - 2 - 4 - 8 - 16 - 24 - 32 - 40 - 48 - 56 - 64 enable_padding: true max_batch_size: 64 trust_remote_code: true enable_chunked_prefill: true enable_iter_perf_stats: true guided_decoding_backend: xgrammar moe_expert_parallel_size: 2 disable_overlap_scheduler: false chat_processor: harmony checkpoint_name: openai/gpt-oss-120b b10_routing_config: algo_selector: B10 durable_kv_events: false enable_worker_tree: true rate_limit_enabled: true router_queue_policy: wspt router_cache_miss_weight: 0.02 router_cache_miss_min_isl: 1024 rate_limit_batch_multiplier: 2 router_active_request_weight: 2 router_prefill_token_discount: 0.3 tensor_parallel_size: 4 model_level_stop_words: - <|call|> tokenizer_limit_length: 131072 additional_environment_variables: Worker: TRTLLM_ENABLE_PDL: '1' MIMALLOC_PURGE_DELAY: '0' PYTORCH_CUDA_ALLOC_CONF: expandable_segments:True NCCL_GRAPH_MIXING_SUPPORT: '0' BAD_TOKEN_ID_SEQ_CHECK_ENABLED: '1' TLLM_NUMA_AWARE_WORKER_AFFINITY: '1' version: 0.0.1-20261001221953-8f302211environment_variables: {}resources: instance_type: B200:4weights:# Public, ungated checkpoint, pinned to the revision `@main` resolved to.- source: hf://openai/gpt-oss-120b@b5c939de8f754692c1647ca79fbf85e8c1e70f8a mount_location: /models/gpt-oss-120b
Deploy
Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:
baseten model push
uvx truss push
You should see output similar to:
Pushing model "gpt-oss-120b"...Uploading model...Uploaded model in 0sβ¨ Model gpt-oss-120b was successfully pushed β¨ Model: gpt-oss-120b (abc1d2ef) Deployment: xyz123 Environment: productionπͺ΅ View logs: deployment: baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123 environment: baseten model environment logs --model-id abc1d2ef --environment production (once deployed) app: https://app.baseten.co/models/abc1d2ef/logs/xyz123π Invoke your model: URL: https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict CLI: baseten model predict --model-id abc1d2ef
baseten model push prints your model ID (abc1d2ef in the example). The examples below use it wherever you see {model_id}, and read your API key from the BASETEN_API_KEY environment variable.
Call the model
Your deployment serves an OpenAI-compatible API.Now call your deployment to run inference:
This preset serves GPT-OSS 120B on B200:4 through Baseten Inference Stack with a TensorRT-LLM PyTorch backend, the same configuration that serves the baseten/gpt-oss-120b library listing, optimized for maximum throughput on Blackwell.
Hardware
B200 Γ 4
Context
128K
Write the config
Create and move into the project directory:
mkdir gpt-oss-120b-throughput && cd gpt-oss-120b-throughput
Then create a file named config.yaml and paste the following:
config.yaml
# Registry copy of the config serving the Baseten library listing# baseten/gpt-oss-120b (BIS-LLM, TRT-LLM PyTorch backend behind Dynamo cache-aware routing). The bis_llm block,# environment, and weights values are the deployed values from BIS Config Registry# composer `gpt-oss-120b` (bcr_version 1.0.5) verbatim; model_name# follows the repo convention and resources use instance_type.model_name: "model:gpt-oss-120b preset:bis-throughput"model_metadata: repo_id: openai/gpt-oss-120b tags: - openai-compatible example_model_input: messages: - role: user content: "What is the meaning of life?" stream: true model: openai/gpt-oss-120b max_tokens: 1024bis_llm: config: additional_environment_variables: Worker: BAD_TOKEN_ID_SEQ_CHECK_ENABLED: "1" MIMALLOC_PURGE_DELAY: "0" NCCL_GRAPH_MIXING_SUPPORT: "0" PYTORCH_CUDA_ALLOC_CONF: "expandable_segments:True" TLLM_NUMA_AWARE_WORKER_AFFINITY: "1" TRTLLM_ENABLE_PDL: "1" TRTLLM_SERVER_DISABLE_GC: "1" TRTLLM_WORKER_DISABLE_GC: "1" b10_routing_config: algo_selector: B10 durable_kv_events: false enable_worker_tree: true rate_limit_batch_multiplier: 2 rate_limit_enabled: true router_active_request_weight: 2 router_cache_miss_min_isl: 1024 router_cache_miss_weight: 0.02 router_prefill_token_discount: 0.3 router_queue_policy: wspt chat_processor: harmony checkpoint_name: openai/gpt-oss-120b engine_config: backend: pytorch cuda_graph_config: batch_sizes: - 1 - 2 - 4 - 8 - 16 - 24 - 32 - 40 - 48 - 56 - 64 enable_padding: true max_batch_size: 64 disable_overlap_scheduler: false enable_autotuner: true enable_chunked_prefill: true enable_iter_perf_stats: true guided_decoding_backend: xgrammar kv_cache_config: enable_block_reuse: true enable_partial_reuse: false event_buffer_max_size: 16384 free_gpu_memory_fraction: 0.9 host_cache_size: 120000000000 max_batch_size: 64 max_beam_width: 1 max_input_len: 131072 max_num_tokens: 16384 max_seq_len: 131072 moe_config: backend: TRTLLM moe_expert_parallel_size: 4 sampler_type: TRTLLMSampler stream_interval: 4 trust_remote_code: true gpuTRTImage: "baseten/dynamo-cache-aware-routing:bis-4nf059--trtllm-dfe755294d-c409f2fff-b690c11391" model_level_stop_words: - <|call|> model_name: openai/gpt-oss-120b model_path: /models/gpt-oss-120b reasoning_effort_policy: allowed: - low - medium - high default: medium tensor_parallel_size: 4 tokenizer_limit_length: 131072 version: 0.0.1-20260804033840-1d68536denvironment_variables: {}resources: instance_type: "B200:4"weights: # Public, ungated checkpoint (Apache-2.0); the auth secret is dropped. Pinned to # the revision `@main` resolved to (unchanged since 2025-08-26). - mount_location: /models/gpt-oss-120b source: "hf://openai/gpt-oss-120b@b5c939de8f754692c1647ca79fbf85e8c1e70f8a"
Deploy
Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:
baseten model push
uvx truss push
You should see output similar to:
Pushing model "gpt-oss-120b-throughput"...Uploading model...Uploaded model in 0sβ¨ Model gpt-oss-120b-throughput was successfully pushed β¨ Model: gpt-oss-120b-throughput (abc1d2ef) Deployment: xyz123 Environment: productionπͺ΅ View logs: deployment: baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123 environment: baseten model environment logs --model-id abc1d2ef --environment production (once deployed) app: https://app.baseten.co/models/abc1d2ef/logs/xyz123π Invoke your model: URL: https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict CLI: baseten model predict --model-id abc1d2ef
baseten model push prints your model ID (abc1d2ef in the example). The examples below use it wherever you see {model_id}, and read your API key from the BASETEN_API_KEY environment variable.
Call the model
Your deployment serves an OpenAI-compatible API.Now call your deployment to run inference:
The start_command passes these flags to the engine. Each one controls a runtime or serving behavior:
Flag
Value
What it does
--tensor-parallel-size
$GPU_COUNT
Number of GPUs to shard the model across.
--gpu-memory-utilization
0.90
Fraction of GPU memory vLLM may use for weights and KV cache.
--max-model-len
16384
Maximum context length (tokens) the server accepts per request.
--max-num-batched-tokens
16384
Maximum total tokens processed per scheduler step.
--max-num-seqs
256
Maximum number of concurrent sequences in the batch.
--stream-interval
20
Tokens emitted per streaming chunk.
--enable-chunked-prefill
(no value)
Process long prompts in chunks so decode requests keep running.
--enable-prefix-caching
(no value)
Reuse KV cache across requests that share a prefix.
--async-scheduling
(no value)
Overlap scheduling with GPU execution to hide scheduler latency.
--trust-remote-code
(no value)
Execute model-specific Python from the checkpoint (required for many Qwen, Phi, and custom architectures).
--load-format
runai_streamer
Weight loading backend. runai_streamer: Stream weights from object storage without materializing to disk.
Deploy
Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:
baseten model push
uvx truss push
You should see output similar to:
Pushing model "gpt-oss-120b-h100-throughput"...Uploading model...Uploaded model in 0sβ¨ Model gpt-oss-120b-h100-throughput was successfully pushed β¨ Model: gpt-oss-120b-h100-throughput (abc1d2ef) Deployment: xyz123 Environment: productionπͺ΅ View logs: deployment: baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123 environment: baseten model environment logs --model-id abc1d2ef --environment production (once deployed) app: https://app.baseten.co/models/abc1d2ef/logs/xyz123π Invoke your model: URL: https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict CLI: baseten model predict --model-id abc1d2ef
baseten model push prints your model ID (abc1d2ef in the example). The examples below use it wherever you see {model_id}, and read your API key from the BASETEN_API_KEY environment variable.
Call the model
Your deployment serves an OpenAI-compatible API.Now call your deployment to run inference: