> ## Documentation Index
> Fetch the complete documentation index at: https://docs.baseten.co/llms.txt
> Use this file to discover all available pages before exploring further.

# DeepSeek V4

> DeepSeek V4 recipes: 2 variants (Flash, Pro), MoE architecture.

[Reasoning](https://docs.baseten.co/examples/models/capabilities/reasoning)
[Tool calling](https://docs.baseten.co/examples/models/capabilities/tool-calling)
[Agentic](https://docs.baseten.co/examples/models/capabilities/agentic)
[Long context](https://docs.baseten.co/examples/models/capabilities/long-context)

## Setup

Install the Baseten CLI and sign in, then install the OpenAI SDK.

**Install and sign in to Baseten**

**macOS or Linux**

```bash Terminal theme={"system"}
brew tap basetenlabs/baseten
brew trust --formula basetenlabs/baseten/baseten
brew install baseten
```

**Windows**

Download and extract the Windows archive for the latest Baseten CLI release:

```powershell Terminal theme={"system"}
$release = Invoke-RestMethod https://api.github.com/repos/basetenlabs/baseten-cli/releases/latest
$asset = $release.assets | Where-Object { $_.name -like "*_windows_amd64.zip" }
Invoke-WebRequest $asset.browser_download_url -OutFile baseten.zip
Expand-Archive -Force baseten.zip .
```

Move `baseten.exe` to a directory on your `PATH`.

For other platforms or a specific version, see the [Baseten CLI install reference](https://docs.baseten.co/reference/cli/baseten/overview.md#install).

```sh theme={"system"}
baseten auth login
```

**Install the OpenAI SDK**

```sh theme={"system"}
uv pip install openai
```

To install without Homebrew, see the [Baseten CLI install reference](https://docs.baseten.co/reference/cli/baseten/overview.md#install).

Pick the model you want to deploy. Each tab is a self-contained recipe.

**Flash**

[deepseek-ai/DeepSeek-V4-Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash) is a VERIFY-parameter MoE model (VERIFY active per token) with up to VERIFY context.

This preset serves DeepSeek V4 Flash on B200:4 with FP8 KV cache, the `deep_gemm_mega_moe` backend, expert parallelism, and MTP speculative decoding, tuned for low time-to-first-token.

> **Hardware**
>
> B200 × 4

## Write the config

Create and move into the project directory:

```sh theme={"system"}
mkdir deepseek-v4-flash-latency && cd deepseek-v4-flash-latency
```

Then create a file named `config.yaml` and paste the following:

```yaml config.yaml theme={"system"}
# Registry copy of the config serving the Baseten library listing baseten/deepseek-v4-flash
# (BIS-LLM). The bis_llm block and environment are the deployed values verbatim; model_name
# follows the repo convention, resources use instance_type, and the weights entry drops the
# workspace-specific auth secret because the checkpoint is public and ungated.
model_name: "model:deepseek-v4-flash preset:bis-latency"
model_metadata:
  repo_id: deepseek-ai/DeepSeek-V4-Flash-0731
  tags:
    - openai-compatible
  example_model_input:
    messages:
      - role: user
        content: "What is the meaning of life?"
    stream: true
    model: deepseek-ai/DeepSeek-V4-Flash-0731
    max_tokens: 32768
    temperature: 1
environment_variables: {}
resources:
  instance_type: B200:4
bis_llm:
  version: 0.0.1-20260727180920-c08268b2
  config:
    model_name: deepseek-ai/DeepSeek-V4-Flash-0731
    model_path: /models/deepseek-v4-flash
    model_path_for_tokenizer: /models/deepseek-v4-flash/tokenizer
    checkpoint_name: deepseek-ai/DeepSeek-V4-Flash-0731
    gpuTRTImage: baseten/dynamo-cache-aware-routing:trtllm-rc23-1.3.0rc23-f598f80c4-e3957193bc
    served_model_name: deepseek-ai/DeepSeek-V4-Flash-0731
    served_model_name_response: deepseek-ai/DeepSeek-V4-Flash-0731
    chat_processor: deepseek_v4
    reasoning_parser: deepseek_v4
    tool_call_parser: deepseek_v4
    arguments_as_json: true
    tensor_parallel_size: 4
    tokenizer_limit_length: 1048576
    tokenizer_max_new_tokens_limit: 32768
    default_thinking_enabled: true
    default_sampling_params:
      top_p: 1
      temperature: 1
    b10_routing_config:
      durable_kv_events: false
      enable_worker_tree: true
      router_track_residency: true
      router_overlap_score_weight: 8
    additional_environment_variables:
      Worker:
        TRTLLM_ENABLE_PDL: "1"
        BAD_TOKEN_ID_SEQ_LEN: "10"
        BAD_TOKEN_ID_SEQ_CHECK_ENABLED: "1"
    engine_config:
      backend: pytorch
      trust_remote_code: true
      max_seq_len: 1048576
      max_input_len: 1048576
      max_batch_size: 8
      max_beam_width: 1
      max_num_tokens: 8192
      enable_chunked_prefill: true
      enable_iter_perf_stats: true
      moe_expert_parallel_size: 4
      moe_config:
        backend: TRTLLM
        use_low_precision_moe_combine: true
      kv_cache_config:
        dtype: fp8
        host_cache_size: 100000000000
        tokens_per_block: 128
        enable_block_reuse: true
        enable_partial_reuse: false
        event_buffer_max_size: 65536
        free_gpu_memory_fraction: 0.9
      speculative_config:
        decoding_type: DSpark
        speculative_model_dir: /models/deepseek-v4-flash/worker
        max_draft_len: 5
        block_size: 5
        markov_rank: 256
        markov_head_type: vanilla
        mask_token_id: 128799
        target_layer_ids: [40, 41, 42]
        allow_advanced_sampling: true
weights:
  # Public, ungated checkpoint (MIT); no auth secret. The deployed listing referenced a
  # workspace-specific secret that does not exist in the registry deploy account.
  - source: hf://deepseek-ai/DeepSeek-V4-Flash-0731@7872f01b1d1fe23eabc4c98b48bffcef5a386062
    mount_location: /models/deepseek-v4-flash
```

The container loads DeepSeek V4 Flash weights to `/models/deepseek-v4-flash` and serves the OpenAI-compatible API on port 8000. FP8 KV cache and the `deep_gemm_mega_moe` backend keep memory bandwidth in check, and the MTP speculator runs two draft tokens per step to amortize sampling cost.

## Deploy

Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:

```sh Baseten CLI theme={"system"}
baseten model push
```

```sh Truss CLI theme={"system"}
uvx truss push
```

You should see output similar to:

```output theme={"system"}
Pushing model "deepseek-v4-flash-latency"...
Uploading model...
Uploaded model in 0s
✨ Model deepseek-v4-flash-latency was successfully pushed ✨

  Model:       deepseek-v4-flash-latency (abc1d2ef)
  Deployment:  xyz123
  Environment: production

🪵 View logs:
   deployment:   baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
   environment:  baseten model environment logs --model-id abc1d2ef --environment production  (once deployed)
   app:          https://app.baseten.co/models/abc1d2ef/logs/xyz123

🚀 Invoke your model:
   URL:  https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
   CLI:  baseten model predict --model-id abc1d2ef
```

`baseten model push` prints your **model ID** (`abc1d2ef` in the example). The examples below use it wherever you see `{model_id}`, and read your API key from the `BASETEN_API_KEY` environment variable.

## Call the model

Your deployment serves an OpenAI-compatible API.

Now call your deployment to run inference:

**Python**

```python main.py theme={"system"}
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["BASETEN_API_KEY"],
    base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)

response = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V4-Flash-0731",
    messages=[
        {"role": "user", "content": "What is machine learning?"}
    ],
)

print(response.choices[0].message.content)
```

**cURL**

```sh theme={"system"}
curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '{
    "model": "deepseek-ai/DeepSeek-V4-Flash-0731",
    "messages": [
      {"role": "user", "content": "What is machine learning?"}
    ]
  }'
```

**Pro**

[deepseek-ai/DeepSeek-V4-Pro](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro) is a VERIFY-parameter MoE model (VERIFY active per token) with up to VERIFY context.

This preset serves DeepSeek V4 Pro on B200:8 with FP8 KV cache, the `deep_gemm_mega_moe` backend, expert parallelism, and MTP speculative decoding, tuned for low time-to-first-token at full scale.

> **Hardware**
>
> B200 × 8

## Write the config

Create and move into the project directory:

```sh theme={"system"}
mkdir deepseek-v4-pro-latency && cd deepseek-v4-pro-latency
```

Then create a file named `config.yaml` and paste the following:

```yaml config.yaml theme={"system"}
# Registry copy of the config serving the Baseten library listing
# baseten/deepseek-v4-pro (BIS-LLM, TRT-LLM PyTorch backend behind Dynamo cache-aware routing). The bis_llm block,
# environment, and weights values are the deployed values from BIS Config Registry
# composer `dsv4-pro-golden` (bcr_version 1.0.9) verbatim; model_name
# follows the repo convention and resources use instance_type.
model_name: "model:deepseek-v4-pro preset:bis-latency"
model_metadata:
  repo_id: deepseek-ai/DeepSeek-V4-Pro
  tags:
    - openai-compatible
  example_model_input:
    messages:
      - role: user
        content: "What is the meaning of life?"
    stream: true
    model: deepseek-ai/DeepSeek-V4-Pro
    max_tokens: 1024
bis_llm:
  config:
    additional_environment_variables:
      Worker:
        BAD_TOKEN_ID_SEQ_CHECK_ENABLED: "1"
        BAD_TOKEN_ID_SEQ_LEN: "10"
        TRTLLM_ENABLE_PDL: "1"
    arguments_as_json: true
    b10_autoscaling_config:
      additional_autoscaling_config:
        metrics:
          - name: in_flight_tokens
            target: 130000
    b10_routing_config:
      durable_kv_events: false
      enable_worker_tree: true
    chat_processor: deepseek_v4
    checkpoint_name: deepseek-ai/DeepSeek-V4-Pro
    default_sampling_params:
      temperature: 1
      top_p: 1
    default_thinking_enabled: true
    engine_config:
      backend: pytorch
      enable_chunked_prefill: true
      enable_iter_perf_stats: true
      kv_cache_config:
        dtype: fp8
        enable_block_reuse: true
        enable_partial_reuse: false
        enable_tp_mla_replicated_host_offload: true
        event_buffer_max_size: 65536
        free_gpu_memory_fraction: 0.9
        host_cache_size: 500000000000
        tokens_per_block: 128
      max_batch_size: 8
      max_beam_width: 1
      max_input_len: 1048576
      max_num_tokens: 16384
      max_seq_len: 1048576
      moe_config:
        backend: TRTLLM
        use_low_precision_moe_combine: true
      moe_expert_parallel_size: 4
      speculative_config:
        allow_advanced_sampling: true
        decoding_type: MTP
        max_draft_len: 3
      trust_remote_code: true
    gpuTRTImage: "baseten/dynamo-cache-aware-routing:trtllm-gpu-f83d539b2a-00b032f66-c62638826d"
    model_name: deepseek-ai/DeepSeek-V4-Pro
    model_path: /models/deepseek-v4-pro
    model_path_for_tokenizer: /models/deepseek-v4-pro/tokenizer
    reasoning_parser: deepseek_v4
    served_model_name: deepseek-ai/DeepSeek-V4-Pro
    served_model_name_response: deepseek-ai/DeepSeek-V4-Pro
    tensor_parallel_size: 8
    tokenizer_limit_length: 1048576
    tokenizer_max_new_tokens_limit: 32768
    tool_call_parser: deepseek_v4
  version: 0.0.1-20260601190849-691c46ce
environment_variables: {}
# The 865 GB checkpoint and TRT-LLM initialization outlasted the PR deployment
# startup window; allow the worker to finish before readiness is judged.
runtime:
  health_checks:
    startup_threshold_seconds: 2400
resources:
  instance_type: "B200:8"
weights:
  # Public, ungated checkpoint; the deployed listing's workspace-specific secret
  # (hf_access_b10_b10admin) is dropped.
  - mount_location: /models/deepseek-v4-pro
    source: "hf://deepseek-ai/DeepSeek-V4-Pro@5607980f3a4b8ea0371b9f11e1848ac41f14979e"
```

The container loads DeepSeek V4 Pro weights to `/models/deepseek-v4-pro` and serves the OpenAI-compatible API on port 8000. Tensor parallelism is set to 8 across the B200 fleet, FP8 KV cache and the `deep_gemm_mega_moe` backend hold memory bandwidth, and the MTP speculator runs two draft tokens per step.

## Deploy

Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:

```sh Baseten CLI theme={"system"}
baseten model push
```

```sh Truss CLI theme={"system"}
uvx truss push
```

You should see output similar to:

```output theme={"system"}
Pushing model "deepseek-v4-pro-latency"...
Uploading model...
Uploaded model in 0s
✨ Model deepseek-v4-pro-latency was successfully pushed ✨

  Model:       deepseek-v4-pro-latency (abc1d2ef)
  Deployment:  xyz123
  Environment: production

🪵 View logs:
   deployment:   baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
   environment:  baseten model environment logs --model-id abc1d2ef --environment production  (once deployed)
   app:          https://app.baseten.co/models/abc1d2ef/logs/xyz123

🚀 Invoke your model:
   URL:  https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
   CLI:  baseten model predict --model-id abc1d2ef
```

`baseten model push` prints your **model ID** (`abc1d2ef` in the example). The examples below use it wherever you see `{model_id}`, and read your API key from the `BASETEN_API_KEY` environment variable.

## Call the model

Your deployment serves an OpenAI-compatible API.

Now call your deployment to run inference:

**Python**

```python main.py theme={"system"}
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["BASETEN_API_KEY"],
    base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)

response = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V4-Pro",
    messages=[
        {"role": "user", "content": "What is machine learning?"}
    ],
)

print(response.choices[0].message.content)
```

**cURL**

```sh theme={"system"}
curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '{
    "model": "deepseek-ai/DeepSeek-V4-Pro",
    "messages": [
      {"role": "user", "content": "What is machine learning?"}
    ]
  }'
```

## Next steps

> **[Call your model](https://docs.baseten.co/inference/calling-your-model.md)**
>
> Endpoint anatomy, authentication, and sync versus async inference

> **[Autoscaling](https://docs.baseten.co/deployment/autoscaling/overview.md)**
>
> Scale replicas with traffic, including scale to zero