DeepSeek V4

Reasoning Tool calling Agentic Long context

Setup

Install the Baseten CLI and sign in, then install the OpenAI SDK.

Install and sign in to Baseten

macOS or Linux
Windows

Terminal

brew tap basetenlabs/baseten
brew trust --formula basetenlabs/baseten/baseten
brew install baseten

Download and extract the Windows archive for the latest Baseten CLI release:

Terminal

$release = Invoke-RestMethod https://api.github.com/repos/basetenlabs/baseten-cli/releases/latest
$asset = $release.assets | Where-Object { $_.name -like "*_windows_amd64.zip" }
Invoke-WebRequest $asset.browser_download_url -OutFile baseten.zip
Expand-Archive -Force baseten.zip .

Move baseten.exe to a directory on your PATH.

For other platforms or a specific version, see the Baseten CLI install reference.

baseten auth login

Install the OpenAI SDK

uv pip install openai

To install without Homebrew, see the Baseten CLI install reference. Pick the model you want to deploy. Each tab is a self-contained recipe.

Flash
Pro

deepseek-ai/DeepSeek-V4-Flash is a VERIFY-parameter MoE model (VERIFY active per token) with up to VERIFY context.This preset serves DeepSeek V4 Flash on B200:4 with FP8 KV cache, the deep_gemm_mega_moe backend, expert parallelism, and MTP speculative decoding, tuned for low time-to-first-token.

Hardware

B200 × 4

Write the config

Create and move into the project directory:

mkdir deepseek-v4-flash-latency && cd deepseek-v4-flash-latency

Then create a file named config.yaml and paste the following:

config.yaml

# Registry copy of the config serving the Baseten library listing baseten/deepseek-v4-flash
# (BIS-LLM). The bis_llm block and environment are the deployed values verbatim; model_name
# follows the repo convention, resources use instance_type, and the weights entry drops the
# workspace-specific auth secret because the checkpoint is public and ungated.
model_name: "model:deepseek-v4-flash preset:bis-latency"
model_metadata:
  repo_id: deepseek-ai/DeepSeek-V4-Flash-0731
  tags:
    - openai-compatible
  example_model_input:
    messages:
      - role: user
        content: "What is the meaning of life?"
    stream: true
    model: deepseek-ai/DeepSeek-V4-Flash-0731
    max_tokens: 32768
    temperature: 1
environment_variables: {}
resources:
  instance_type: B200:4
bis_llm:
  version: 0.0.1-20260727180920-c08268b2
  config:
    model_name: deepseek-ai/DeepSeek-V4-Flash-0731
    model_path: /models/deepseek-v4-flash
    model_path_for_tokenizer: /models/deepseek-v4-flash/tokenizer
    checkpoint_name: deepseek-ai/DeepSeek-V4-Flash-0731
    gpuTRTImage: baseten/dynamo-cache-aware-routing:trtllm-rc23-1.3.0rc23-f598f80c4-e3957193bc
    served_model_name: deepseek-ai/DeepSeek-V4-Flash-0731
    served_model_name_response: deepseek-ai/DeepSeek-V4-Flash-0731
    chat_processor: deepseek_v4
    reasoning_parser: deepseek_v4
    tool_call_parser: deepseek_v4
    arguments_as_json: true
    tensor_parallel_size: 4
    tokenizer_limit_length: 1048576
    tokenizer_max_new_tokens_limit: 32768
    default_thinking_enabled: true
    default_sampling_params:
      top_p: 1
      temperature: 1
    b10_routing_config:
      durable_kv_events: false
      enable_worker_tree: true
      router_track_residency: true
      router_overlap_score_weight: 8
    additional_environment_variables:
      Worker:
        TRTLLM_ENABLE_PDL: "1"
        BAD_TOKEN_ID_SEQ_LEN: "10"
        BAD_TOKEN_ID_SEQ_CHECK_ENABLED: "1"
    engine_config:
      backend: pytorch
      trust_remote_code: true
      max_seq_len: 1048576
      max_input_len: 1048576
      max_batch_size: 8
      max_beam_width: 1
      max_num_tokens: 8192
      enable_chunked_prefill: true
      enable_iter_perf_stats: true
      moe_expert_parallel_size: 4
      moe_config:
        backend: TRTLLM
        use_low_precision_moe_combine: true
      kv_cache_config:
        dtype: fp8
        host_cache_size: 100000000000
        tokens_per_block: 128
        enable_block_reuse: true
        enable_partial_reuse: false
        event_buffer_max_size: 65536
        free_gpu_memory_fraction: 0.9
      speculative_config:
        decoding_type: DSpark
        speculative_model_dir: /models/deepseek-v4-flash/worker
        max_draft_len: 5
        block_size: 5
        markov_rank: 256
        markov_head_type: vanilla
        mask_token_id: 128799
        target_layer_ids: [40, 41, 42]
        allow_advanced_sampling: true
weights:
  # Public, ungated checkpoint (MIT); no auth secret. The deployed listing referenced a
  # workspace-specific secret that does not exist in the registry deploy account.
  - source: hf://deepseek-ai/DeepSeek-V4-Flash-0731@7872f01b1d1fe23eabc4c98b48bffcef5a386062
    mount_location: /models/deepseek-v4-flash

The container loads DeepSeek V4 Flash weights to /models/deepseek-v4-flash and serves the OpenAI-compatible API on port 8000. FP8 KV cache and the deep_gemm_mega_moe backend keep memory bandwidth in check, and the MTP speculator runs two draft tokens per step to amortize sampling cost.

Deploy

Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:

baseten model push

uvx truss push

You should see output similar to:

Pushing model "deepseek-v4-flash-latency"...
Uploading model...
Uploaded model in 0s
✨ Model deepseek-v4-flash-latency was successfully pushed ✨

  Model:       deepseek-v4-flash-latency (abc1d2ef)
  Deployment:  xyz123
  Environment: production

🪵 View logs:
   deployment:   baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
   environment:  baseten model environment logs --model-id abc1d2ef --environment production  (once deployed)
   app:          https://app.baseten.co/models/abc1d2ef/logs/xyz123

🚀 Invoke your model:
   URL:  https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
   CLI:  baseten model predict --model-id abc1d2ef

baseten model push prints your model ID (abc1d2ef in the example). The examples below use it wherever you see {model_id}, and read your API key from the BASETEN_API_KEY environment variable.

Call the model

Your deployment serves an OpenAI-compatible API.Now call your deployment to run inference:

Python
cURL

main.py

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["BASETEN_API_KEY"],
    base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)

response = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V4-Flash-0731",
    messages=[
        {"role": "user", "content": "What is machine learning?"}
    ],
)

print(response.choices[0].message.content)

curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '{
    "model": "deepseek-ai/DeepSeek-V4-Flash-0731",
    "messages": [
      {"role": "user", "content": "What is machine learning?"}
    ]
  }'

deepseek-ai/DeepSeek-V4-Pro is a VERIFY-parameter MoE model (VERIFY active per token) with up to VERIFY context.This preset serves DeepSeek V4 Pro on B200:8 with FP8 KV cache, the deep_gemm_mega_moe backend, expert parallelism, and MTP speculative decoding, tuned for low time-to-first-token at full scale.

Hardware

B200 × 8

Write the config

Create and move into the project directory:

mkdir deepseek-v4-pro-latency && cd deepseek-v4-pro-latency

Then create a file named config.yaml and paste the following:

config.yaml

# Registry copy of the config serving the Baseten library listing
# baseten/deepseek-v4-pro (BIS-LLM, TRT-LLM PyTorch backend behind Dynamo cache-aware routing). The bis_llm block,
# environment, and weights values are the deployed values from BIS Config Registry
# composer `dsv4-pro-golden` (bcr_version 1.0.9) verbatim; model_name
# follows the repo convention and resources use instance_type.
model_name: "model:deepseek-v4-pro preset:bis-latency"
model_metadata:
  repo_id: deepseek-ai/DeepSeek-V4-Pro
  tags:
    - openai-compatible
  example_model_input:
    messages:
      - role: user
        content: "What is the meaning of life?"
    stream: true
    model: deepseek-ai/DeepSeek-V4-Pro
    max_tokens: 1024
bis_llm:
  config:
    additional_environment_variables:
      Worker:
        BAD_TOKEN_ID_SEQ_CHECK_ENABLED: "1"
        BAD_TOKEN_ID_SEQ_LEN: "10"
        TRTLLM_ENABLE_PDL: "1"
    arguments_as_json: true
    b10_autoscaling_config:
      additional_autoscaling_config:
        metrics:
          - name: in_flight_tokens
            target: 130000
    b10_routing_config:
      durable_kv_events: false
      enable_worker_tree: true
    chat_processor: deepseek_v4
    checkpoint_name: deepseek-ai/DeepSeek-V4-Pro
    default_sampling_params:
      temperature: 1
      top_p: 1
    default_thinking_enabled: true
    engine_config:
      backend: pytorch
      enable_chunked_prefill: true
      enable_iter_perf_stats: true
      kv_cache_config:
        dtype: fp8
        enable_block_reuse: true
        enable_partial_reuse: false
        enable_tp_mla_replicated_host_offload: true
        event_buffer_max_size: 65536
        free_gpu_memory_fraction: 0.9
        host_cache_size: 500000000000
        tokens_per_block: 128
      max_batch_size: 8
      max_beam_width: 1
      max_input_len: 1048576
      max_num_tokens: 16384
      max_seq_len: 1048576
      moe_config:
        backend: TRTLLM
        use_low_precision_moe_combine: true
      moe_expert_parallel_size: 4
      speculative_config:
        allow_advanced_sampling: true
        decoding_type: MTP
        max_draft_len: 3
      trust_remote_code: true
    gpuTRTImage: "baseten/dynamo-cache-aware-routing:trtllm-gpu-f83d539b2a-00b032f66-c62638826d"
    model_name: deepseek-ai/DeepSeek-V4-Pro
    model_path: /models/deepseek-v4-pro
    model_path_for_tokenizer: /models/deepseek-v4-pro/tokenizer
    reasoning_parser: deepseek_v4
    served_model_name: deepseek-ai/DeepSeek-V4-Pro
    served_model_name_response: deepseek-ai/DeepSeek-V4-Pro
    tensor_parallel_size: 8
    tokenizer_limit_length: 1048576
    tokenizer_max_new_tokens_limit: 32768
    tool_call_parser: deepseek_v4
  version: 0.0.1-20260601190849-691c46ce
environment_variables: {}
# The 865 GB checkpoint and TRT-LLM initialization outlasted the PR deployment
# startup window; allow the worker to finish before readiness is judged.
runtime:
  health_checks:
    startup_threshold_seconds: 2400
resources:
  instance_type: "B200:8"
weights:
  # Public, ungated checkpoint; the deployed listing's workspace-specific secret
  # (hf_access_b10_b10admin) is dropped.
  - mount_location: /models/deepseek-v4-pro
    source: "hf://deepseek-ai/DeepSeek-V4-Pro@5607980f3a4b8ea0371b9f11e1848ac41f14979e"

The container loads DeepSeek V4 Pro weights to /models/deepseek-v4-pro and serves the OpenAI-compatible API on port 8000. Tensor parallelism is set to 8 across the B200 fleet, FP8 KV cache and the deep_gemm_mega_moe backend hold memory bandwidth, and the MTP speculator runs two draft tokens per step.

Deploy

Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:

baseten model push

uvx truss push

You should see output similar to:

Pushing model "deepseek-v4-pro-latency"...
Uploading model...
Uploaded model in 0s
✨ Model deepseek-v4-pro-latency was successfully pushed ✨

  Model:       deepseek-v4-pro-latency (abc1d2ef)
  Deployment:  xyz123
  Environment: production

🪵 View logs:
   deployment:   baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
   environment:  baseten model environment logs --model-id abc1d2ef --environment production  (once deployed)
   app:          https://app.baseten.co/models/abc1d2ef/logs/xyz123

🚀 Invoke your model:
   URL:  https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
   CLI:  baseten model predict --model-id abc1d2ef

Call the model

Your deployment serves an OpenAI-compatible API.Now call your deployment to run inference:

Python
cURL

main.py

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["BASETEN_API_KEY"],
    base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)

response = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V4-Pro",
    messages=[
        {"role": "user", "content": "What is machine learning?"}
    ],
)

print(response.choices[0].message.content)

curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '{
    "model": "deepseek-ai/DeepSeek-V4-Pro",
    "messages": [
      {"role": "user", "content": "What is machine learning?"}
    ]
  }'

Examples

Models

Engines

Custom Docker servers

Custom Python models

Chains

Baseten for Model Labs

Setup

Hardware

Write the config

Deploy

Call the model

Hardware

Write the config

Deploy

Call the model

Next steps

Call your model

Autoscaling

​Setup

Hardware

​Write the config

​Deploy

​Call the model

Hardware

​Write the config

​Deploy

​Call the model

​Next steps

Call your model

Autoscaling

Setup

Write the config

Deploy

Call the model

Write the config

Deploy

Call the model

Next steps