Laguna

Agentic Reasoning Tool calling Long context

Setup

Sign in to Baseten

uvx truss login --browser

Install the OpenAI SDK

uv pip install openai

Pick the model you want to deploy. Each tab is a self-contained recipe.

M.1
S 2.1

poolside/Laguna-M.1-FP8 is a MoE model with up to 256K context.This preset serves Laguna M.1 on H100:4 with FP8 weights, optimized for low time-to-first-token on interactive reasoning and coding workloads.

Hardware

H100 × 4

Engine

vLLM 0.21.0

Context

256K

Concurrency

Write the config

Create and move into the project directory:

mkdir laguna-m.1-latency && cd laguna-m.1-latency

Then create a file named config.yaml and paste the following:

config.yaml

model_name: "model:laguna-m.1 preset:latency"

model_metadata:
  description: >-
    Laguna M.1 FP8 MoE reasoning model from Poolside, served with vLLM (H100 TP=4),
    OpenAI-compatible chat with tool calling and extended reasoning support.
    Latency-optimized: low max-num-seqs to minimize head-of-line blocking from long thinking traces.
  repo_id: poolside/Laguna-M.1-FP8
  trust_remote_code: true
  tags:
    - openai-compatible
    - vllm
    - moe
    - reasoning
    - agentic-coding
    - fp8
  example_model_input:
    model: poolside/laguna-m.1
    messages:
      - role: user
        content: "Write a Python retry wrapper with exponential backoff."
    stream: true
    temperature: 1.0
    top_k: 20

# ---------------------------------------------------------------------------
# Base image — vLLM with Laguna support (requires vLLM >= 0.21.0)
# ---------------------------------------------------------------------------
base_image:
  image: vllm/vllm-openai:v0.21.0
  python_executable_path: /usr/bin/python3

# ---------------------------------------------------------------------------
# Weights — FP8 quantized checkpoint (~225 GB, fits in 4× H100 / 320 GB)
# Quantization is detected automatically from the checkpoint's
# quantization_config — no extra vLLM flags needed.
# ---------------------------------------------------------------------------
weights:
  - source: "hf://poolside/Laguna-M.1-FP8"
    mount_location: "/models/laguna-m1"

environment_variables:
  VLLM_LOGGING_LEVEL: WARNING
  VLLM_ENGINE_READY_TIMEOUT_S: "3600"

# ---------------------------------------------------------------------------
# Docker server — vLLM OpenAI-compatible endpoint
# ---------------------------------------------------------------------------
docker_server:
  start_command: >
    vllm serve /models/laguna-m1
    --served-model-name poolside/laguna-m.1
    --host 0.0.0.0
    --port 8000
    --tool-call-parser poolside_v1
    --reasoning-parser poolside_v1
    --enable-auto-tool-choice
    --default-chat-template-kwargs '{"enable_thinking": true}'
    --tensor-parallel-size 4
    --max-model-len 262144
    --max-num-seqs 64
    --gpu-memory-utilization 0.95
    --trust-remote-code
  readiness_endpoint: /health
  liveness_endpoint: /health
  predict_endpoint: /v1/chat/completions
  server_port: 8000

# ---------------------------------------------------------------------------
# Resources
# FP8 ~225 GB → 4× H100 (320 GB total VRAM) with comfortable headroom
# ---------------------------------------------------------------------------
resources:
  accelerator: H100:4
  cpu: "8"
  memory: 32Gi
  use_gpu: true

# ---------------------------------------------------------------------------
# Runtime
# ---------------------------------------------------------------------------
runtime:
  predict_concurrency: 64
  health_checks:
    restart_check_delay_seconds: 1800
    restart_threshold_seconds: 600
    stop_traffic_threshold_seconds: 180

Flags

The start_command passes these flags to the engine. Each one controls a runtime or serving behavior:

Flag	Value	What it does
`--tool-call-parser`	`poolside_v1`	Server-side parser that emits structured `tool_calls` on the response.
`--reasoning-parser`	`poolside_v1`	Server-side parser that separates reasoning output into `reasoning_content`.
`--enable-auto-tool-choice`	(no value)	Let the model choose when to call tools without requiring `tool_choice: "required"`.
`--default-chat-template-kwargs`	`{"enable_thinking": true}`	Default keyword arguments applied to the chat template, used to set behaviors like enabling reasoning by default.
`--tensor-parallel-size`	`4`	Number of GPUs to shard the model across.
`--max-model-len`	`262144`	Maximum context length (tokens) the server accepts per request.
`--max-num-seqs`	`64`	Maximum number of concurrent sequences in the batch.
`--gpu-memory-utilization`	`0.95`	Fraction of GPU memory vLLM may use for weights and KV cache.
`--trust-remote-code`	(no value)	Execute model-specific Python from the checkpoint (required for many Qwen, Phi, and custom architectures).

Deploy

Push the config to Baseten:

uvx truss push

You should see output similar to:

✨ Model laguna-m.1-latency was successfully pushed ✨

   Model ID:      abc1d2ef
   Deployment ID: xyz123
   Endpoint:      model-abc1d2ef.api.baseten.co
   Logs:          https://app.baseten.co/models/abc1d2ef/logs/xyz123

truss push prints your model ID (abc1d2ef in the example). The examples below use it wherever you see {model_id}, and read your API key from the BASETEN_API_KEY environment variable.

Call the model

Your deployment serves an OpenAI-compatible API.Now call your deployment to run inference:

Python
cURL

main.py

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["BASETEN_API_KEY"],
    base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)

response = client.chat.completions.create(
    model="poolside/laguna-m.1",
    messages=[
        {"role": "user", "content": "What is machine learning?"}
    ],
)

print(response.choices[0].message.content)

curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '{
    "model": "poolside/laguna-m.1",
    "messages": [
      {"role": "user", "content": "What is machine learning?"}
    ]
  }'

The server parses the model’s chain of thought into a separate reasoning_content field on the response. Read it alongside the final answer:

response = client.chat.completions.create(
    model="poolside/laguna-m.1",
    messages=[
        {"role": "user", "content": "How many r's in strawberry?"}
    ],
)
print(response.choices[0].message.reasoning_content)  # chain of thought
print(response.choices[0].message.content)            # final answer

To let the model call tools, pass a tools array. The server returns structured tool_calls on the response:

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "parameters": {
            "type": "object",
            "properties": {"location": {"type": "string"}},
            "required": ["location"],
        },
    },
}]

response = client.chat.completions.create(
    model="poolside/laguna-m.1",
    messages=[
        {"role": "user", "content": "What's the weather in Paris?"}
    ],
    tools=tools,
)
print(response.choices[0].message.tool_calls)

poolside/Laguna-S-2.1-FP8 is a MoE model with up to 256K context.This preset serves Laguna S 2.1 on H100:8 with FP8 weights through the Baseten Inference Stack’s TensorRT-LLM engine, optimized for low latency on agentic coding and reasoning workloads.

Hardware

H100 × 8

Context

256K

Write the config

Create and move into the project directory:

mkdir laguna-s.2.1-latency && cd laguna-s.2.1-latency

Then create a file named config.yaml and paste the following:

config.yaml

model_name: "model:laguna-s.2.1 preset:latency"

model_metadata:
  description: >-
    Poolside Laguna S 2.1 FP8 agentic coding model served with TensorRT-LLM
    on eight H100 GPUs, with OpenAI-compatible chat, tool calling, reasoning,
    and a 262K-token context window.
  repo_id: poolside/Laguna-S-2.1-FP8
  trust_remote_code: true
  tags:
    - openai-compatible
    - trt-llm
    - moe
    - reasoning
    - agentic-coding
    - fp8
  example_model_input:
    model: poolside/Laguna-S-2.1-FP8
    messages:
      - role: user
        content: "Write a Python retry wrapper with exponential backoff."
    stream: true
    max_tokens: 32768
    temperature: 1.0

environment_variables: {}

resources:
  accelerator: H100:8
  use_gpu: true

bis_llm:
  version: 0.0.1-20260601190849-691c46ce
  config:
    additional_environment_variables:
      Worker:
        BAD_TOKEN_ID_SEQ_CHECK_ENABLED: "1"
        LD_LIBRARY_PATH: /usr/local/mpi/lib:/src/.venv-3.12/lib/python3.12/site-packages/tensorrt_llm/libs:/src/.venv-3.12/lib/python3.12/site-packages/tensorrt_libs:/usr/local/cuda/targets/x86_64-linux/lib:/usr/local/lib/python3.12/dist-packages/torch/lib:/usr/local/lib/python3.12/dist-packages/torch_tensorrt/lib:/usr/local/cuda/compat/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
        PATH: /src/.venv-3.12/bin:/usr/local/lib/python3.12/dist-packages/torch_tensorrt/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/mpi/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/ucx/bin:/opt/amazon/efa/bin:/opt/tensorrt/bin
        PYTHONPATH: /src/.venv-3.12/lib/python3.12/site-packages:/workspace/trtllm
        PYTORCH_CUDA_ALLOC_CONF: expandable_segments:True
        TRTLLM_ENABLE_PDL: "0"
    arguments_as_json: true
    b10_autoscaling_config:
      additional_autoscaling_config:
        metrics:
          - name: in_flight_tokens
            target: 130000
      max_replica: 1
      min_replica: 1
    b10_routing_config:
      algo_selector: B10
      router_active_replicas: 1
      router_cache_miss_min_isl: 15000
      router_cache_miss_weight: 0.03
      router_decode_token_discount: 1.5
      router_disable_snapshots_in_primary: true
      router_overlap_score_weight: 6.5
      router_prefill_token_discount: 0.15
      router_queue_threshold: 2
      router_replica_sync: true
      router_snapshot_threshold: 200000
      router_temperature: 0.015
    checkpoint_name: poolside/Laguna-S-2.1-FP8
    default_sampling_params:
      max_tokens: 32768
    default_sampling_params_thinking:
      max_tokens: 32768
    default_thinking_enabled: true
    engine_config:
      backend: pytorch
      cuda_graph_config:
        batch_sizes:
          - 1
          - 2
          - 4
          - 8
        enable_padding: true
      enable_chunked_prefill: true
      guided_decoding_backend: xgrammar
      kv_cache_config:
        dtype: fp8
        enable_block_reuse: true
        enable_partial_reuse: false
        event_buffer_max_size: 16384
        free_gpu_memory_fraction: 0.95
        max_attention_window:
          - 262144
          - 512
          - 512
          - 512
      max_batch_size: 8
      max_input_len: 262144
      max_num_tokens: 16384
      max_seq_len: 262144
      moe_config:
        backend: CUTLASS
      trust_remote_code: true
    gpuTRTImage: baseten/dynamo-cache-aware-routing:maxtokclamp-20260715--trtllm-laguna-dflash-laguna-pr15666-dflash-ac9980ff16-3ee6c64a9-969251ab0@sha256:921b680c6a7c8b343ddd88ceb4099b304bcc4069d1ebd6b0f98f83651b4928c8
    model_level_stop_words:
      - </assistant>
    model_name: poolside/Laguna-S-2.1-FP8
    model_path: /models/laguna
    model_path_for_tokenizer: /models/laguna
    reasoning_parser: laguna
    served_model_name: poolside/Laguna-S-2.1-FP8
    served_model_name_response: poolside/Laguna-S-2.1-FP8
    tensor_parallel_size: 8
    tokenizer_limit_length: 262144
    tokenizer_max_new_tokens_limit: 32768
    tool_call_parser: laguna

weights:
  # Pinned to the SpinQuant-rotated FP8 checkpoint the gpuTRTImage was built
  # for. Later upstream revisions (17cacdc6 "spinquantless FP8", 9e0b8ba6
  # "RC2 1M release config") changed the quantization format and crash this
  # engine at weight load (MoE w3_w1 shape mismatch), looping until the
  # deploy times out.
  - source: hf://poolside/Laguna-S-2.1-FP8@610e62523e5ae1261d6b4d3ae7974479041d6027
    mount_location: /models/laguna
    auth_secret_name: hf_access_token

This deployment serves poolside’s Laguna S 2.1 from an FP8 checkpoint on eight H100 GPUs with tensor parallelism across all eight, running the Baseten Inference Stack’s TensorRT-LLM engine with an FP8 KV cache and a 262K-token context window. The server exposes an OpenAI-compatible chat completions API with reasoning and tool calling enabled by the model’s laguna parsers.

Deploy

Push the config to Baseten:

uvx truss push

You should see output similar to:

✨ Model laguna-s.2.1-latency was successfully pushed ✨

   Model ID:      abc1d2ef
   Deployment ID: xyz123
   Endpoint:      model-abc1d2ef.api.baseten.co
   Logs:          https://app.baseten.co/models/abc1d2ef/logs/xyz123

truss push prints your model ID (abc1d2ef in the example). The examples below use it wherever you see {model_id}, and read your API key from the BASETEN_API_KEY environment variable.

Call the model

Your deployment serves an OpenAI-compatible API.Now call your deployment to run inference:

Python
cURL

main.py

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["BASETEN_API_KEY"],
    base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)

response = client.chat.completions.create(
    model="poolside/Laguna-S-2.1-FP8",
    messages=[
        {"role": "user", "content": "What is machine learning?"}
    ],
)

print(response.choices[0].message.content)

curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '{
    "model": "poolside/Laguna-S-2.1-FP8",
    "messages": [
      {"role": "user", "content": "What is machine learning?"}
    ]
  }'

Examples

Models

Engines

Custom Docker servers

Custom Python models

Chains

Setup

Hardware

Engine

Context

Concurrency

Write the config

Flags

Deploy

Call the model

Hardware

Context

Write the config

Deploy

Call the model

Next steps

Call your model

Autoscaling

​Setup

Hardware

Engine

Context

Concurrency

​Write the config

​Flags

​Deploy

​Call the model

Hardware

Context

​Write the config

​Deploy

​Call the model

​Next steps

Call your model

Autoscaling

Setup

Write the config

Flags

Deploy

Call the model

Write the config

Deploy

Call the model

Next steps