> ## Documentation Index
> Fetch the complete documentation index at: https://docs.baseten.co/llms.txt
> Use this file to discover all available pages before exploring further.

# Mellum2

> JetBrains' Mellum2 open MoE code model (12B total, 2.5B active) with a 131k-token context window and tool calling.

<div className="capability-pills">
  <a href="/examples/models/capabilities/tool-calling" className="capability-pill">Tool calling</a>
  <a href="/examples/models/capabilities/long-context" className="capability-pill">Long context</a>
</div>

## Setup

Sign in to Baseten with Truss, then install the OpenAI SDK.

<Columns cols={2}>
  <Column>
    **Sign in to Baseten**

    ```sh theme={"system"}
    uvx truss login --browser
    ```
  </Column>

  <Column>
    **Install the OpenAI SDK**

    ```sh theme={"system"}
    uv pip install openai
    ```
  </Column>
</Columns>

[JetBrains/Mellum2-12B-A2.5B-Instruct](https://huggingface.co/JetBrains/Mellum2-12B-A2.5B-Instruct) is a 12B-parameter MoE model (2.5B active per token) with up to 128K context.

This preset serves Mellum2 12B A2.5B Instruct on a single H100 through vLLM, optimized for low-latency code generation.

<CardGroup cols={4}>
  <Card title="Hardware" icon="microchip">H100</Card>
  <Card title="Engine" icon="server">vLLM 0.23.0</Card>
  <Card title="Context" icon="ruler-horizontal">128K</Card>
  <Card title="Concurrency" icon="layer-group">128</Card>
</CardGroup>

## Write the config

Create and move into the project directory:

```sh theme={"system"}
mkdir mellum2-12b-a2.5b-instruct-latency && cd mellum2-12b-a2.5b-instruct-latency
```

Then create a file named `config.yaml` and paste the following:

```yaml config.yaml theme={"system"}
model_name: model:mellum2-12b-a2.5b-instruct preset:latency

model_metadata:
  example_model_input:
    model: "JetBrains/Mellum2-12B-A2.5B-Instruct"
    messages:
      - role: user
        content: "Write a Python function to reverse a string."
    stream: true
    max_tokens: 4096
    temperature: 0.6
    top_p: 0.95
  tags:
    - openai-compatible
    - code
    - moe

base_image:
  image: vllm/vllm-openai:v0.23.0

weights:
  - source: "hf://JetBrains/Mellum2-12B-A2.5B-Instruct@main"
    mount_location: "/app/checkpoint/model"
    auth_secret_name: "hf_access_token"

resources:
  accelerator: H100
  use_gpu: true

runtime:
  predict_concurrency: 128
  health_checks:
    startup_threshold_seconds: 1800
    restart_threshold_seconds: 600
    stop_traffic_threshold_seconds: 120

environment_variables:
  HF_HUB_ENABLE_HF_TRANSFER: "1"
  VLLM_LOGGING_LEVEL: WARNING
  VLLM_ENGINE_READY_TIMEOUT_S: "3600"

secrets:
  hf_access_token: null

docker_server:
  # No --reasoning-parser: Instruct answers directly without <think> blocks.
  # Remove --enable-auto-tool-choice / --tool-call-parser if you don't need tool use.
  start_command: >-
    sh -c "GPU_COUNT=$(nvidia-smi --list-gpus | wc -l) && vllm serve /app/checkpoint/model
    --tensor-parallel-size $GPU_COUNT
    --served-model-name JetBrains/Mellum2-12B-A2.5B-Instruct
    --host 0.0.0.0
    --port 8000
    --max-model-len auto
    --enable-prefix-caching
    --enable-auto-tool-choice
    --tool-call-parser hermes
    --trust-remote-code
    --load-format runai_streamer"
  readiness_endpoint: /health
  liveness_endpoint: /health
  predict_endpoint: /v1/chat/completions
  server_port: 8000
```

This config runs the official `vllm/vllm-openai:v0.23.0` image, the release that adds `MellumForCausalLM` support, and streams weights from `JetBrains/Mellum2-12B-A2.5B-Instruct` with the Run:ai streamer. The Hermes tool-call parser enables OpenAI-compatible function calling, and a concurrency ceiling of 128 keeps the deployment throughput-friendly for coding assistants.

## Flags

The `start_command` passes these flags to the engine. Each one controls a runtime or serving behavior:

| Flag                        | Value            | What it does                                                                                                    |
| --------------------------- | ---------------- | --------------------------------------------------------------------------------------------------------------- |
| `--tensor-parallel-size`    | `$GPU_COUNT`     | Number of GPUs to shard the model across.                                                                       |
| `--max-model-len`           | `auto`           | Maximum context length (tokens) the server accepts per request.                                                 |
| `--enable-prefix-caching`   | (no value)       | Reuse KV cache across requests that share a prefix.                                                             |
| `--enable-auto-tool-choice` | (no value)       | Let the model choose when to call tools without requiring `tool_choice: "required"`.                            |
| `--tool-call-parser`        | `hermes`         | Server-side parser that emits structured `tool_calls` on the response. **hermes:** Hermes-style function calls. |
| `--trust-remote-code`       | (no value)       | Execute model-specific Python from the checkpoint (required for many Qwen, Phi, and custom architectures).      |
| `--load-format`             | `runai_streamer` | Weight loading backend. **runai\_streamer:** Stream weights from object storage without materializing to disk.  |

## Deploy

Push the config to Baseten:

```sh theme={"system"}
uvx truss push
```

You should see output similar to:

```output theme={"system"}
✨ Model mellum2-12b-a2.5b-instruct-latency was successfully pushed ✨

   Model ID:      abc1d2ef
   Deployment ID: xyz123
   Endpoint:      model-abc1d2ef.api.baseten.co
   Logs:          https://app.baseten.co/models/abc1d2ef/logs/xyz123
```

Your **model ID** is printed in the `truss push` output (`abcd1234` in the example). Use it wherever you see `{model_id}` in the next section.

## Call the model

Your deployment serves an OpenAI-compatible API. Replace `{model_id}` with your model ID and make sure `BASETEN_API_KEY` is set.

Now call your deployment to run inference:

<Tabs>
  <Tab title="Python">
    ```python main.py theme={"system"}
    import os
    from openai import OpenAI

    client = OpenAI(
        api_key=os.environ["BASETEN_API_KEY"],
        base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
    )

    response = client.chat.completions.create(
        model="JetBrains/Mellum2-12B-A2.5B-Instruct",
        messages=[
            {"role": "user", "content": "What is machine learning?"}
        ],
    )

    print(response.choices[0].message.content)
    ```
  </Tab>

  <Tab title="cURL">
    ```sh theme={"system"}
    curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
      -H "Content-Type: application/json" \
      -H "Authorization: Bearer $BASETEN_API_KEY" \
      -d '{
        "model": "JetBrains/Mellum2-12B-A2.5B-Instruct",
        "messages": [
          {"role": "user", "content": "What is machine learning?"}
        ]
      }'
    ```
  </Tab>
</Tabs>

To let the model call tools, pass a `tools` array. The server returns structured `tool_calls` on the response:

```python theme={"system"}
tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "parameters": {
            "type": "object",
            "properties": {"location": {"type": "string"}},
            "required": ["location"],
        },
    },
}]

response = client.chat.completions.create(
    model="JetBrains/Mellum2-12B-A2.5B-Instruct",
    messages=[
        {"role": "user", "content": "What's the weather in Paris?"}
    ],
    tools=tools,
)
print(response.choices[0].message.tool_calls)
```