> ## Documentation Index > Fetch the complete documentation index at: https://docs.baseten.co/llms.txt > Use this file to discover all available pages before exploring further. # Nemotron 3 > Nemotron 3 recipes: 2 variants (Nano 30B A3B, Super 120B A12B), Hybrid MoE and MoE architectures.
Reasoning Tool calling Long context Agentic
## Setup Install the Baseten CLI and sign in, then install the OpenAI SDK. **Install and sign in to Baseten** ```bash Terminal theme={"system"} brew tap basetenlabs/baseten brew trust --formula basetenlabs/baseten/baseten brew install baseten ``` Download and extract the Windows archive for the latest Baseten CLI release: ```powershell Terminal theme={"system"} $release = Invoke-RestMethod https://api.github.com/repos/basetenlabs/baseten-cli/releases/latest $asset = $release.assets | Where-Object { $_.name -like "*_windows_amd64.zip" } Invoke-WebRequest $asset.browser_download_url -OutFile baseten.zip Expand-Archive -Force baseten.zip . ``` Move `baseten.exe` to a directory on your `PATH`. For other platforms or a specific version, see the [Baseten CLI install reference](/reference/cli/baseten/overview#install). ```sh theme={"system"} baseten auth login ``` **Install the OpenAI SDK** ```sh theme={"system"} uv pip install openai ``` To install without Homebrew, see the [Baseten CLI install reference](/reference/cli/baseten/overview#install). Pick the model you want to deploy. Each tab is a self-contained recipe. [nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8) is a 30B-parameter hybrid MoE model (3.5B active per token) with up to 256K context. This preset serves NVIDIA Nemotron 3 Nano 30B A3B on one H100 with FP8 weights and KV cache, optimized for low-latency reasoning at the 32K context window sized for the NVIDIA cascaded voice pipeline. H100 vLLM (0.22.0-cu129 build) 32K 8 ## Write the config Create and move into the project directory: ```sh theme={"system"} mkdir nemotron-3-nano-latency && cd nemotron-3-nano-latency ``` Then create a file named `config.yaml` and paste the following: ```yaml config.yaml theme={"system"} model_name: "model:nemotron-3-nano preset:latency" model_metadata: description: >- NVIDIA Nemotron 3 Nano 30B A3B (text-only) reasoning model served via vLLM on a single H100. A Mamba2-Transformer hybrid MoE — 23 Mamba-2 layers, 23 MoE layers and 6 GQA attention layers, 128 routed experts plus 1 shared expert, 6 active per token, 3.5B active of 30B total parameters. FP8 checkpoint (weights and KV cache), with tool calling and a toggleable reasoning trace. repo_id: nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 example_model_input: model: nvidia/nemotron-3-nano messages: - role: user content: "What is the capital of France?" stream: true max_tokens: 512 temperature: 0.2 extra_body: chat_template_kwargs: enable_thinking: false tags: - openai-compatible base_image: # Matches the sibling NVIDIA vLLM entries. Needs --reasoning-parser-plugin, # present since v0.20.0. image: vllm/vllm-openai:v0.22.0-cu129 weights: # Public, ungated upstream repo — not a baseten-admin mirror. The Xet 403 that # this repo is known for is specific to the model_cache: path; weights: pulls it # fine (33 GB in 34s on first deploy). - source: "hf://nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8@f8dc1c0afee92f44417695b4f5ddca9afc95ea58" mount_location: "/app/checkpoint/model" docker_server: # The reasoning parser is nano_v3, shipped inside the HF repo and loaded as a # plugin — NOT the built-in nemotron_v3 that the omni entry and the NIM use. # weights: mounts the whole repo, so the plugin is already on disk. # # --kv-cache-dtype fp8: the checkpoint quantized the KV cache to FP8 with the # weights (attention and the Mamba layers feeding it stay BF16). # --mamba-ssm-cache-dtype float32: keeps the hybrid Mamba SSM state cache in # fp32 to avoid accuracy drift. Beyond the model card; matches omni. # --max-model-len 32768 / --max-num-seqs 8: latency preset, sized for the # cascaded voice pipeline. A throughput preset would raise both. # --tensor-parallel-size 1: single-GPU preset, so it needs no `sh -c` wrapper # at all. The multi-GPU entries derive TP from nvidia-smi and must use # SINGLE quotes (see #283) to defer expansion into the container; invoking # vllm directly avoids that whole class of quoting bug. Keep it that way. start_command: >- vllm serve /app/checkpoint/model --tensor-parallel-size 1 --host 0.0.0.0 --port 8000 --served-model-name nvidia/nemotron-3-nano --trust-remote-code --max-model-len 32768 --max-num-seqs 8 --gpu-memory-utilization 0.9 --kv-cache-dtype fp8 --mamba-ssm-cache-dtype float32 --reasoning-parser-plugin /app/checkpoint/model/nano_v3_reasoning_parser.py --reasoning-parser nano_v3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --enable-prefix-caching --load-format runai_streamer readiness_endpoint: /health liveness_endpoint: /health predict_endpoint: /v1/chat/completions server_port: 8000 environment_variables: VLLM_LOGGING_LEVEL: WARNING VLLM_ENGINE_READY_TIMEOUT_S: "3600" # Model card: FlashInfer FP8 MoE kernels for this checkpoint. VLLM_USE_FLASHINFER_MOE_FP8: "1" resources: accelerator: H100 use_gpu: true runtime: predict_concurrency: 8 health_checks: startup_threshold_seconds: 1800 restart_threshold_seconds: 1200 stop_traffic_threshold_seconds: 120 ``` This preset deploys the FP8 checkpoint of Nemotron 3 Nano 30B A3B through vLLM on a single H100 with the model's own nano\_v3 reasoning parser loaded as a plugin. The KV cache is FP8 to match the checkpoint, and the Mamba SSM state cache stays in FP32 to avoid accuracy drift. The deployment exposes an OpenAI-compatible chat completions endpoint with reasoning and tool calling at a 32K context window. ## Flags The `start_command` passes these flags to the engine. Each one controls a runtime or serving behavior: | Flag | Value | What it does | | - | - | - | | `--tensor-parallel-size` | `1` | Number of GPUs to shard the model across. | | `--trust-remote-code` | (no value) | Execute model-specific Python from the checkpoint (required for many Qwen, Phi, and custom architectures). | | `--max-model-len` | `32768` | Maximum context length (tokens) the server accepts per request. | | `--max-num-seqs` | `8` | Maximum number of concurrent sequences in the batch. | | `--gpu-memory-utilization` | `0.9` | Fraction of GPU memory vLLM may use for weights and KV cache. | | `--kv-cache-dtype` | `fp8` | KV cache numeric precision. **fp8:** \~2× KV cache density with negligible quality impact on most models. | | `--mamba-ssm-cache-dtype` | `float32` | Precision for the Mamba state-space model cache on hybrid attention and SSM architectures. | | `--reasoning-parser-plugin` | `/app/checkpoint/model/nano_v3_reasoning_parser.py` | Path to a custom reasoning parser loaded as a vLLM plugin, used when the parser is not built into vLLM. | | `--reasoning-parser` | `nano_v3` | Server-side parser that separates reasoning output into `reasoning_content`. **nano\_v3:** Nemotron 3 Nano thinking format (shipped inside the HF repo as a plugin). | | `--enable-auto-tool-choice` | (no value) | Let the model choose when to call tools without requiring `tool_choice: "required"`. | | `--tool-call-parser` | `qwen3_coder` | Server-side parser that emits structured `tool_calls` on the response. **qwen3\_coder:** Qwen3-Coder tool format. | | `--enable-prefix-caching` | (no value) | Reuse KV cache across requests that share a prefix. | | `--load-format` | `runai_streamer` | Weight loading backend. **runai\_streamer:** Stream weights from object storage without materializing to disk. | ## Deploy Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it: ```sh Baseten CLI theme={"system"} baseten model push ``` ```sh Truss CLI theme={"system"} uvx truss push ``` You should see output similar to: ```output theme={"system"} Pushing model "nemotron-3-nano-latency"... Uploading model... Uploaded model in 0s ✨ Model nemotron-3-nano-latency was successfully pushed ✨ Model: nemotron-3-nano-latency (abc1d2ef) Deployment: xyz123 Environment: production 🪵 View logs: deployment: baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123 environment: baseten model environment logs --model-id abc1d2ef --environment production (once deployed) app: https://app.baseten.co/models/abc1d2ef/logs/xyz123 🚀 Invoke your model: URL: https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict CLI: baseten model predict --model-id abc1d2ef ``` `baseten model push` prints your **model ID** (`abc1d2ef` in the example). The examples below use it wherever you see `{model_id}`, and read your API key from the `BASETEN_API_KEY` environment variable. ## Call the model Your deployment serves an OpenAI-compatible API. Now call your deployment to run inference: ```python main.py theme={"system"} import os from openai import OpenAI client = OpenAI( api_key=os.environ["BASETEN_API_KEY"], base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1", ) response = client.chat.completions.create( model="nvidia/nemotron-3-nano", messages=[ {"role": "user", "content": "What is machine learning?"} ], ) print(response.choices[0].message.content) ``` ```sh theme={"system"} curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $BASETEN_API_KEY" \ -d '{ "model": "nvidia/nemotron-3-nano", "messages": [ {"role": "user", "content": "What is machine learning?"} ] }' ``` The server parses the model's chain of thought into a separate `reasoning_content` field on the response. Read it alongside the final answer: ```python theme={"system"} response = client.chat.completions.create( model="nvidia/nemotron-3-nano", messages=[ {"role": "user", "content": "How many r's in strawberry?"} ], ) print(response.choices[0].message.reasoning_content) # chain of thought print(response.choices[0].message.content) # final answer ``` To let the model call tools, pass a `tools` array. The server returns structured `tool_calls` on the response: ```python theme={"system"} tools = [{ "type": "function", "function": { "name": "get_weather", "parameters": { "type": "object", "properties": {"location": {"type": "string"}}, "required": ["location"], }, }, }] response = client.chat.completions.create( model="nvidia/nemotron-3-nano", messages=[ {"role": "user", "content": "What's the weather in Paris?"} ], tools=tools, ) print(response.choices[0].message.tool_calls) ``` [nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4) is a 120B-parameter MoE model with 12B active per token. This preset serves Nemotron 3 Super 120B A12B on B200:4 through [Baseten Inference Stack](/engines/bis-llm/overview) (TensorRT-LLM) with NVFP4 weights, expert parallelism, and MTP speculative decoding. It targets high-throughput reasoning. B200 × 4 TRT-LLM v2 128K 32 ## Write the config Create and move into the project directory: ```sh theme={"system"} mkdir nemotron-3-super-120b-a12b-throughput && cd nemotron-3-super-120b-a12b-throughput ``` Then create a file named `config.yaml` and paste the following: ```yaml config.yaml theme={"system"} model_name: model:nemotron-3-super-120b-a12b preset:throughput model_metadata: example_model_input: model: "nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4" max_tokens: 512 messages: - role: user content: Tell me everything you know about optimized inference. stream: true temperature: 0.5 tags: - openai-compatible resources: accelerator: B200:4 cpu: "1" memory: 10Gi use_gpu: true environment_variables: PYTORCH_CUDA_ALLOC_CONF: "expandable_segments:True" TRTLLM_ENABLE_PDL: "1" BAD_TOKEN_ID_SEQ_CHECK_ENABLED: "1" ENABLE_B10_LOOKAHEAD: "0" secrets: hf_access_token: null trt_llm: inference_stack: v2 build: checkpoint_repository: repo: nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 revision: main source: HF runtime_secret_name: hf_access_token runtime: enable_chunked_prefill: true max_batch_size: 32 max_num_tokens: 16384 max_seq_len: 131072 tensor_parallel_size: 4 served_model_name: nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 patch_kwargs: reasoning_parser: nemotron3 tool_call_parser: qwen3_coder tokenizer_limit_length: 131072 arguments_as_json: true engine_config: backend: pytorch enable_chunked_prefill: true enable_iter_perf_stats: true max_batch_size: 32 max_beam_width: 1 max_input_len: 131072 max_num_tokens: 16384 max_seq_len: 131072 trust_remote_code: true moe_expert_parallel_size: 4 cuda_graph_config: enable_padding: true max_batch_size: 32 kv_cache_config: dtype: fp8 enable_block_reuse: false free_gpu_memory_fraction: 0.8 mamba_ssm_cache_dtype: float32 moe_config: backend: TRTLLM speculative_config: decoding_type: MTP num_nextn_predict_layers: 3 allow_advanced_sampling: true ``` This config tells Baseten to compile a TensorRT-LLM engine for Nemotron 3 Super 120B A12B on four B200 GPUs with NVFP4-quantized weights, wiring the Qwen3-coder tool-call parser and Nemotron 3 reasoning parser into the engine config. Expert parallelism across all four GPUs, MTP speculative decoding with three draft tokens, and chunked prefill combine to push high reasoning throughput, serving up to a 128K context window. ## Key parameters [Baseten Inference Stack](/engines/bis-llm/overview) (BIS) reads these fields from the `trt_llm` block. Each one shapes how the engine is built and served: | Parameter | Value | | - | - | | Tensor parallel size | `4` | | Max sequence length | `131072` | | Max batch size | `32` | | Max batched tokens | `16384` | | Chunked prefill | `enabled` | | Inference stack | `v2` | | Served model name | `nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4` | ## Deploy Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it: ```sh Baseten CLI theme={"system"} baseten model push ``` ```sh Truss CLI theme={"system"} uvx truss push ``` You should see output similar to: ```output theme={"system"} Pushing model "nemotron-3-super-120b-a12b-throughput"... Uploading model... Uploaded model in 0s ✨ Model nemotron-3-super-120b-a12b-throughput was successfully pushed ✨ Model: nemotron-3-super-120b-a12b-throughput (abc1d2ef) Deployment: xyz123 Environment: production 🪵 View logs: deployment: baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123 environment: baseten model environment logs --model-id abc1d2ef --environment production (once deployed) app: https://app.baseten.co/models/abc1d2ef/logs/xyz123 🚀 Invoke your model: URL: https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict CLI: baseten model predict --model-id abc1d2ef ``` `baseten model push` prints your **model ID** (`abc1d2ef` in the example). The examples below use it wherever you see `{model_id}`, and read your API key from the `BASETEN_API_KEY` environment variable. ## Call the model Your deployment serves an OpenAI-compatible API. Now call your deployment to run inference: ```python main.py theme={"system"} import os from openai import OpenAI client = OpenAI( api_key=os.environ["BASETEN_API_KEY"], base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1", ) response = client.chat.completions.create( model="nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4", messages=[ {"role": "user", "content": "What is machine learning?"} ], ) print(response.choices[0].message.content) ``` ```sh theme={"system"} curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $BASETEN_API_KEY" \ -d '{ "model": "nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4", "messages": [ {"role": "user", "content": "What is machine learning?"} ] }' ``` ## Next steps Endpoint anatomy, authentication, and sync versus async inference Scale replicas with traffic, including scale to zero