> ## Documentation Index > Fetch the complete documentation index at: https://docs.baseten.co/llms.txt > Use this file to discover all available pages before exploring further. # Laguna > Laguna recipes: 2 variants (M.1, S 2.1), MoE architecture.
Agentic Reasoning Tool calling Long context
## Setup Install the Baseten CLI and sign in, then install the OpenAI SDK. **Install and sign in to Baseten** ```bash Terminal theme={"system"} brew tap basetenlabs/baseten brew trust --formula basetenlabs/baseten/baseten brew install baseten ``` Download and extract the Windows archive for the latest Baseten CLI release: ```powershell Terminal theme={"system"} $release = Invoke-RestMethod https://api.github.com/repos/basetenlabs/baseten-cli/releases/latest $asset = $release.assets | Where-Object { $_.name -like "*_windows_amd64.zip" } Invoke-WebRequest $asset.browser_download_url -OutFile baseten.zip Expand-Archive -Force baseten.zip . ``` Move `baseten.exe` to a directory on your `PATH`. For other platforms or a specific version, see the [Baseten CLI install reference](/reference/cli/baseten/overview#install). ```sh theme={"system"} baseten auth login ``` **Install the OpenAI SDK** ```sh theme={"system"} uv pip install openai ``` To install without Homebrew, see the [Baseten CLI install reference](/reference/cli/baseten/overview#install). Pick the model you want to deploy. Each tab is a self-contained recipe. [poolside/Laguna-M.1-FP8](https://huggingface.co/poolside/Laguna-M.1-FP8) is a MoE model with up to 256K context. This preset serves Laguna M.1 on H100:4 with FP8 weights, optimized for low time-to-first-token on interactive reasoning and coding workloads. H100 × 4 vLLM 0.21.0 256K 64 ## Write the config Create and move into the project directory: ```sh theme={"system"} mkdir laguna-m.1-latency && cd laguna-m.1-latency ``` Then create a file named `config.yaml` and paste the following: ```yaml config.yaml theme={"system"} model_name: "model:laguna-m.1 preset:latency" model_metadata: description: >- Laguna M.1 FP8 MoE reasoning model from Poolside, served with vLLM (H100 TP=4), OpenAI-compatible chat with tool calling and extended reasoning support. Latency-optimized: low max-num-seqs to minimize head-of-line blocking from long thinking traces. repo_id: poolside/Laguna-M.1-FP8 trust_remote_code: true tags: - openai-compatible - vllm - moe - reasoning - agentic-coding - fp8 example_model_input: model: poolside/laguna-m.1 messages: - role: user content: "Write a Python retry wrapper with exponential backoff." stream: true temperature: 1.0 top_k: 20 # --------------------------------------------------------------------------- # Base image — vLLM with Laguna support (requires vLLM >= 0.21.0) # --------------------------------------------------------------------------- base_image: image: vllm/vllm-openai:v0.21.0 python_executable_path: /usr/bin/python3 # --------------------------------------------------------------------------- # Weights — FP8 quantized checkpoint (~225 GB, fits in 4× H100 / 320 GB) # Quantization is detected automatically from the checkpoint's # quantization_config — no extra vLLM flags needed. # --------------------------------------------------------------------------- weights: - source: "hf://poolside/Laguna-M.1-FP8@3732a656e6f0bf09f90250518d8de9c8e5c54f12" mount_location: "/models/laguna-m1" requirements: - runai-model-streamer==0.16.1 environment_variables: VLLM_LOGGING_LEVEL: WARNING VLLM_ENGINE_READY_TIMEOUT_S: "3600" # --------------------------------------------------------------------------- # Docker server — vLLM OpenAI-compatible endpoint # --------------------------------------------------------------------------- docker_server: start_command: >- vllm serve /models/laguna-m1 --served-model-name poolside/laguna-m.1 --host 0.0.0.0 --port 8000 --tool-call-parser poolside_v1 --reasoning-parser poolside_v1 --enable-auto-tool-choice --default-chat-template-kwargs '{"enable_thinking": true}' --tensor-parallel-size 4 --max-model-len 262144 --max-num-seqs 64 --gpu-memory-utilization 0.95 --trust-remote-code --load-format runai_streamer readiness_endpoint: /health liveness_endpoint: /health predict_endpoint: /v1/chat/completions server_port: 8000 # --------------------------------------------------------------------------- # Resources # FP8 ~225 GB → 4× H100 (320 GB total VRAM) with comfortable headroom # --------------------------------------------------------------------------- resources: accelerator: H100:4 cpu: "8" memory: 32Gi use_gpu: true # --------------------------------------------------------------------------- # Runtime # --------------------------------------------------------------------------- runtime: predict_concurrency: 64 health_checks: startup_threshold_seconds: 1800 restart_threshold_seconds: 600 stop_traffic_threshold_seconds: 180 ``` ## Flags The `start_command` passes these flags to the engine. Each one controls a runtime or serving behavior: | Flag | Value | What it does | | - | - | - | | `--tool-call-parser` | `poolside_v1` | Server-side parser that emits structured `tool_calls` on the response. | | `--reasoning-parser` | `poolside_v1` | Server-side parser that separates reasoning output into `reasoning_content`. | | `--enable-auto-tool-choice` | (no value) | Let the model choose when to call tools without requiring `tool_choice: "required"`. | | `--default-chat-template-kwargs` | `{"enable_thinking": true}` | Default keyword arguments applied to the chat template, used to set behaviors like enabling reasoning by default. | | `--tensor-parallel-size` | `4` | Number of GPUs to shard the model across. | | `--max-model-len` | `262144` | Maximum context length (tokens) the server accepts per request. | | `--max-num-seqs` | `64` | Maximum number of concurrent sequences in the batch. | | `--gpu-memory-utilization` | `0.95` | Fraction of GPU memory vLLM may use for weights and KV cache. | | `--trust-remote-code` | (no value) | Execute model-specific Python from the checkpoint (required for many Qwen, Phi, and custom architectures). | | `--load-format` | `runai_streamer` | Weight loading backend. **runai\_streamer:** Stream weights from object storage without materializing to disk. | ## Deploy Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it: ```sh Baseten CLI theme={"system"} baseten model push ``` ```sh Truss CLI theme={"system"} uvx truss push ``` You should see output similar to: ```output theme={"system"} Pushing model "laguna-m.1-latency"... Uploading model... Uploaded model in 0s ✨ Model laguna-m.1-latency was successfully pushed ✨ Model: laguna-m.1-latency (abc1d2ef) Deployment: xyz123 Environment: production 🪵 View logs: deployment: baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123 environment: baseten model environment logs --model-id abc1d2ef --environment production (once deployed) app: https://app.baseten.co/models/abc1d2ef/logs/xyz123 🚀 Invoke your model: URL: https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict CLI: baseten model predict --model-id abc1d2ef ``` `baseten model push` prints your **model ID** (`abc1d2ef` in the example). The examples below use it wherever you see `{model_id}`, and read your API key from the `BASETEN_API_KEY` environment variable. ## Call the model Your deployment serves an OpenAI-compatible API. Now call your deployment to run inference: ```python main.py theme={"system"} import os from openai import OpenAI client = OpenAI( api_key=os.environ["BASETEN_API_KEY"], base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1", ) response = client.chat.completions.create( model="poolside/laguna-m.1", messages=[ {"role": "user", "content": "What is machine learning?"} ], ) print(response.choices[0].message.content) ``` ```sh theme={"system"} curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $BASETEN_API_KEY" \ -d '{ "model": "poolside/laguna-m.1", "messages": [ {"role": "user", "content": "What is machine learning?"} ] }' ``` The server parses the model's chain of thought into a separate `reasoning_content` field on the response. Read it alongside the final answer: ```python theme={"system"} response = client.chat.completions.create( model="poolside/laguna-m.1", messages=[ {"role": "user", "content": "How many r's in strawberry?"} ], ) print(response.choices[0].message.reasoning_content) # chain of thought print(response.choices[0].message.content) # final answer ``` To let the model call tools, pass a `tools` array. The server returns structured `tool_calls` on the response: ```python theme={"system"} tools = [{ "type": "function", "function": { "name": "get_weather", "parameters": { "type": "object", "properties": {"location": {"type": "string"}}, "required": ["location"], }, }, }] response = client.chat.completions.create( model="poolside/laguna-m.1", messages=[ {"role": "user", "content": "What's the weather in Paris?"} ], tools=tools, ) print(response.choices[0].message.tool_calls) ``` [poolside/Laguna-S-2.1-FP8](https://huggingface.co/poolside/Laguna-S-2.1-FP8) is a MoE model with up to 256K context. This preset serves Laguna S 2.1 on H100:8 with FP8 weights through the Baseten Inference Stack's TensorRT-LLM engine, optimized for low latency on agentic coding and reasoning workloads. H100 × 8 256K ## Write the config Create and move into the project directory: ```sh theme={"system"} mkdir laguna-s.2.1-latency && cd laguna-s.2.1-latency ``` Then create a file named `config.yaml` and paste the following: ```yaml config.yaml theme={"system"} model_name: "model:laguna-s.2.1 preset:latency" model_metadata: description: >- Poolside Laguna S 2.1 FP8 agentic coding model served with TensorRT-LLM on eight H100 GPUs, with OpenAI-compatible chat, tool calling, reasoning, and a 262K-token context window. repo_id: poolside/Laguna-S-2.1-FP8 trust_remote_code: true tags: - openai-compatible - trt-llm - moe - reasoning - agentic-coding - fp8 example_model_input: model: poolside/Laguna-S-2.1-FP8 messages: - role: user content: "Write a Python retry wrapper with exponential backoff." stream: true max_tokens: 32768 temperature: 1.0 environment_variables: {} resources: accelerator: H100:8 use_gpu: true bis_llm: version: 0.0.1-20260601190849-691c46ce config: additional_environment_variables: Worker: BAD_TOKEN_ID_SEQ_CHECK_ENABLED: "1" LD_LIBRARY_PATH: /usr/local/mpi/lib:/src/.venv-3.12/lib/python3.12/site-packages/tensorrt_llm/libs:/src/.venv-3.12/lib/python3.12/site-packages/tensorrt_libs:/usr/local/cuda/targets/x86_64-linux/lib:/usr/local/lib/python3.12/dist-packages/torch/lib:/usr/local/lib/python3.12/dist-packages/torch_tensorrt/lib:/usr/local/cuda/compat/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64 PATH: /src/.venv-3.12/bin:/usr/local/lib/python3.12/dist-packages/torch_tensorrt/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/mpi/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/ucx/bin:/opt/amazon/efa/bin:/opt/tensorrt/bin PYTHONPATH: /src/.venv-3.12/lib/python3.12/site-packages:/workspace/trtllm PYTORCH_CUDA_ALLOC_CONF: expandable_segments:True TRTLLM_ENABLE_PDL: "0" arguments_as_json: true b10_autoscaling_config: additional_autoscaling_config: metrics: - name: in_flight_tokens target: 130000 max_replica: 1 min_replica: 1 b10_routing_config: algo_selector: B10 router_active_replicas: 1 router_cache_miss_min_isl: 15000 router_cache_miss_weight: 0.03 router_decode_token_discount: 1.5 router_disable_snapshots_in_primary: true router_overlap_score_weight: 6.5 router_prefill_token_discount: 0.15 router_queue_threshold: 2 router_replica_sync: true router_snapshot_threshold: 200000 router_temperature: 0.015 checkpoint_name: poolside/Laguna-S-2.1-FP8 default_sampling_params: max_tokens: 32768 default_sampling_params_thinking: max_tokens: 32768 default_thinking_enabled: true engine_config: backend: pytorch cuda_graph_config: batch_sizes: - 1 - 2 - 4 - 8 enable_padding: true enable_chunked_prefill: true guided_decoding_backend: xgrammar kv_cache_config: dtype: fp8 enable_block_reuse: true enable_partial_reuse: false event_buffer_max_size: 16384 free_gpu_memory_fraction: 0.95 max_attention_window: - 262144 - 512 - 512 - 512 max_batch_size: 8 max_input_len: 262144 max_num_tokens: 16384 max_seq_len: 262144 moe_config: backend: CUTLASS trust_remote_code: true gpuTRTImage: baseten/dynamo-cache-aware-routing:maxtokclamp-20260715--trtllm-laguna-dflash-laguna-pr15666-dflash-ac9980ff16-3ee6c64a9-969251ab0@sha256:921b680c6a7c8b343ddd88ceb4099b304bcc4069d1ebd6b0f98f83651b4928c8 model_level_stop_words: - model_name: poolside/Laguna-S-2.1-FP8 model_path: /models/laguna model_path_for_tokenizer: /models/laguna reasoning_parser: laguna served_model_name: poolside/Laguna-S-2.1-FP8 served_model_name_response: poolside/Laguna-S-2.1-FP8 tensor_parallel_size: 8 tokenizer_limit_length: 262144 tokenizer_max_new_tokens_limit: 32768 tool_call_parser: laguna weights: # Pinned to the SpinQuant-rotated FP8 checkpoint the gpuTRTImage was built # for. Later upstream revisions (17cacdc6 "spinquantless FP8", 9e0b8ba6 # "RC2 1M release config") changed the quantization format and crash this # engine at weight load (MoE w3_w1 shape mismatch), looping until the # deploy times out. - source: hf://poolside/Laguna-S-2.1-FP8@610e62523e5ae1261d6b4d3ae7974479041d6027 mount_location: /models/laguna auth_secret_name: hf_access_token ``` This deployment serves poolside's Laguna S 2.1 from an FP8 checkpoint on eight H100 GPUs with tensor parallelism across all eight, running the Baseten Inference Stack's TensorRT-LLM engine with an FP8 KV cache and a 262K-token context window. The server exposes an OpenAI-compatible chat completions API with reasoning and tool calling enabled by the model's `laguna` parsers. ## Deploy Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it: ```sh Baseten CLI theme={"system"} baseten model push ``` ```sh Truss CLI theme={"system"} uvx truss push ``` You should see output similar to: ```output theme={"system"} Pushing model "laguna-s.2.1-latency"... Uploading model... Uploaded model in 0s ✨ Model laguna-s.2.1-latency was successfully pushed ✨ Model: laguna-s.2.1-latency (abc1d2ef) Deployment: xyz123 Environment: production 🪵 View logs: deployment: baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123 environment: baseten model environment logs --model-id abc1d2ef --environment production (once deployed) app: https://app.baseten.co/models/abc1d2ef/logs/xyz123 🚀 Invoke your model: URL: https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict CLI: baseten model predict --model-id abc1d2ef ``` `baseten model push` prints your **model ID** (`abc1d2ef` in the example). The examples below use it wherever you see `{model_id}`, and read your API key from the `BASETEN_API_KEY` environment variable. ## Call the model Your deployment serves an OpenAI-compatible API. Now call your deployment to run inference: ```python main.py theme={"system"} import os from openai import OpenAI client = OpenAI( api_key=os.environ["BASETEN_API_KEY"], base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1", ) response = client.chat.completions.create( model="poolside/Laguna-S-2.1-FP8", messages=[ {"role": "user", "content": "What is machine learning?"} ], ) print(response.choices[0].message.content) ``` ```sh theme={"system"} curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $BASETEN_API_KEY" \ -d '{ "model": "poolside/Laguna-S-2.1-FP8", "messages": [ {"role": "user", "content": "What is machine learning?"} ] }' ``` ## Next steps Endpoint anatomy, authentication, and sync versus async inference Scale replicas with traffic, including scale to zero