To install without Homebrew, see the Baseten CLI install reference.Pick the model you want to deploy. Each tab is a self-contained recipe.
M.1
S 2.1
poolside/Laguna-M.1-FP8 is a MoE model with up to 256K context.This preset serves Laguna M.1 on H100:4 with FP8 weights, optimized for low time-to-first-token on interactive reasoning and coding workloads.
Hardware
H100 × 4
Engine
vLLM 0.21.0
Context
256K
Concurrency
64
Write the config
Create and move into the project directory:
mkdir laguna-m.1-latency && cd laguna-m.1-latency
Then create a file named config.yaml and paste the following:
config.yaml
model_name: "model:laguna-m.1 preset:latency"model_metadata: description: >- Laguna M.1 FP8 MoE reasoning model from Poolside, served with vLLM (H100 TP=4), OpenAI-compatible chat with tool calling and extended reasoning support. Latency-optimized: low max-num-seqs to minimize head-of-line blocking from long thinking traces. repo_id: poolside/Laguna-M.1-FP8 trust_remote_code: true tags: - openai-compatible - vllm - moe - reasoning - agentic-coding - fp8 example_model_input: model: poolside/laguna-m.1 messages: - role: user content: "Write a Python retry wrapper with exponential backoff." stream: true temperature: 1.0 top_k: 20# ---------------------------------------------------------------------------# Base image — vLLM with Laguna support (requires vLLM >= 0.21.0)# ---------------------------------------------------------------------------base_image: image: vllm/vllm-openai:v0.21.0 python_executable_path: /usr/bin/python3# ---------------------------------------------------------------------------# Weights — FP8 quantized checkpoint (~225 GB, fits in 4× H100 / 320 GB)# Quantization is detected automatically from the checkpoint's# quantization_config — no extra vLLM flags needed.# ---------------------------------------------------------------------------weights: - source: "hf://poolside/Laguna-M.1-FP8@3732a656e6f0bf09f90250518d8de9c8e5c54f12" mount_location: "/models/laguna-m1"requirements: - runai-model-streamer==0.16.1environment_variables: VLLM_LOGGING_LEVEL: WARNING VLLM_ENGINE_READY_TIMEOUT_S: "3600"# ---------------------------------------------------------------------------# Docker server — vLLM OpenAI-compatible endpoint# ---------------------------------------------------------------------------docker_server: start_command: >- vllm serve /models/laguna-m1 --served-model-name poolside/laguna-m.1 --host 0.0.0.0 --port 8000 --tool-call-parser poolside_v1 --reasoning-parser poolside_v1 --enable-auto-tool-choice --default-chat-template-kwargs '{"enable_thinking": true}' --tensor-parallel-size 4 --max-model-len 262144 --max-num-seqs 64 --gpu-memory-utilization 0.95 --trust-remote-code --load-format runai_streamer readiness_endpoint: /health liveness_endpoint: /health predict_endpoint: /v1/chat/completions server_port: 8000# ---------------------------------------------------------------------------# Resources# FP8 ~225 GB → 4× H100 (320 GB total VRAM) with comfortable headroom# ---------------------------------------------------------------------------resources: accelerator: H100:4 cpu: "8" memory: 32Gi use_gpu: true# ---------------------------------------------------------------------------# Runtime# ---------------------------------------------------------------------------runtime: predict_concurrency: 64 health_checks: startup_threshold_seconds: 1800 restart_threshold_seconds: 600 stop_traffic_threshold_seconds: 180
Flags
The start_command passes these flags to the engine. Each one controls a runtime or serving behavior:
Flag
Value
What it does
--tool-call-parser
poolside_v1
Server-side parser that emits structured tool_calls on the response.
--reasoning-parser
poolside_v1
Server-side parser that separates reasoning output into reasoning_content.
--enable-auto-tool-choice
(no value)
Let the model choose when to call tools without requiring tool_choice: "required".
--default-chat-template-kwargs
{"enable_thinking": true}
Default keyword arguments applied to the chat template, used to set behaviors like enabling reasoning by default.
--tensor-parallel-size
4
Number of GPUs to shard the model across.
--max-model-len
262144
Maximum context length (tokens) the server accepts per request.
--max-num-seqs
64
Maximum number of concurrent sequences in the batch.
--gpu-memory-utilization
0.95
Fraction of GPU memory vLLM may use for weights and KV cache.
--trust-remote-code
(no value)
Execute model-specific Python from the checkpoint (required for many Qwen, Phi, and custom architectures).
--load-format
runai_streamer
Weight loading backend. runai_streamer: Stream weights from object storage without materializing to disk.
Deploy
Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:
baseten model push
uvx truss push
You should see output similar to:
Pushing model "laguna-m.1-latency"...Uploading model...Uploaded model in 0s✨ Model laguna-m.1-latency was successfully pushed ✨ Model: laguna-m.1-latency (abc1d2ef) Deployment: xyz123 Environment: production🪵 View logs: deployment: baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123 environment: baseten model environment logs --model-id abc1d2ef --environment production (once deployed) app: https://app.baseten.co/models/abc1d2ef/logs/xyz123🚀 Invoke your model: URL: https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict CLI: baseten model predict --model-id abc1d2ef
baseten model push prints your model ID (abc1d2ef in the example). The examples below use it wherever you see {model_id}, and read your API key from the BASETEN_API_KEY environment variable.
Call the model
Your deployment serves an OpenAI-compatible API.Now call your deployment to run inference:
poolside/Laguna-S-2.1-FP8 is a MoE model with up to 256K context.This preset serves Laguna S 2.1 on H100:8 with FP8 weights through the Baseten Inference Stack’s TensorRT-LLM engine, optimized for low latency on agentic coding and reasoning workloads.
Hardware
H100 × 8
Context
256K
Write the config
Create and move into the project directory:
mkdir laguna-s.2.1-latency && cd laguna-s.2.1-latency
Then create a file named config.yaml and paste the following:
This deployment serves poolside’s Laguna S 2.1 from an FP8 checkpoint on eight H100 GPUs with tensor parallelism across all eight, running the Baseten Inference Stack’s TensorRT-LLM engine with an FP8 KV cache and a 262K-token context window. The server exposes an OpenAI-compatible chat completions API with reasoning and tool calling enabled by the model’s laguna parsers.
Deploy
Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:
baseten model push
uvx truss push
You should see output similar to:
Pushing model "laguna-s.2.1-latency"...Uploading model...Uploaded model in 0s✨ Model laguna-s.2.1-latency was successfully pushed ✨ Model: laguna-s.2.1-latency (abc1d2ef) Deployment: xyz123 Environment: production🪵 View logs: deployment: baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123 environment: baseten model environment logs --model-id abc1d2ef --environment production (once deployed) app: https://app.baseten.co/models/abc1d2ef/logs/xyz123🚀 Invoke your model: URL: https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict CLI: baseten model predict --model-id abc1d2ef
baseten model push prints your model ID (abc1d2ef in the example). The examples below use it wherever you see {model_id}, and read your API key from the BASETEN_API_KEY environment variable.
Call the model
Your deployment serves an OpenAI-compatible API.Now call your deployment to run inference: