> ## Documentation Index > Fetch the complete documentation index at: https://docs.baseten.co/llms.txt > Use this file to discover all available pages before exploring further. # Nemotron 3.5 > NVIDIA's Nemotron 3.5 Lightning is a hybrid Mamba-2, MoE, and attention reasoning model with 30B total and 3B active parameters.
Reasoning Tool calling Long context
NVIDIA's Nemotron 3.5 Lightning is a hybrid Mamba-2, MoE, and attention reasoning model with 30B total and 3B active parameters. This preset serves the NVFP4 checkpoint on one H100 with vLLM and DFlash speculative decoding, targeting high throughput at the model's native 1M-token context. ## Setup Sign in to Baseten with Truss, then install the OpenAI SDK. **Sign in to Baseten** ```sh theme={"system"} uvx truss login --browser ``` **Install the OpenAI SDK** ```sh theme={"system"} uv pip install openai ``` This preset serves Nemotron 3.5 Lightning 30B-A3B on one H100 under vLLM with the NVFP4 checkpoint and DFlash speculative decoding, optimized for total token throughput at the model's native 1M-token context window. H100 vLLM (0.27.0-x86\_64 build) 1M 512 ## Write the config Create and move into the project directory: ```sh theme={"system"} mkdir nemotron-3.5-lightning-throughput && cd nemotron-3.5-lightning-throughput ``` Then create a file named `config.yaml` and paste the following: ```yaml config.yaml theme={"system"} model_name: "model:nemotron-3.5-lightning preset:throughput" model_metadata: description: >- NVIDIA Nemotron 3.5 Lightning 30B-A3B (NVFP4) on a single H100 with vLLM v0.27.0 and DFlash speculative decoding (K=3), tuned for the checkpoint's native 1,048,576-token context window. The verifier and drafter are pinned public NVIDIA Hugging Face repositories mounted through BDN. repo_id: nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 example_model_input: model: nvidia/nemotron-3.5-lightning-nvfp4 messages: - role: user content: "Reply with exactly: deployment healthy" max_tokens: 32 temperature: 0 stream: false tags: - openai-compatible - vllm - dflash - nemotron - nvfp4 - h100 - throughput - 1m-context - public-huggingface base_image: image: vllm/vllm-openai:v0.27.0-x86_64 weights: - source: "hf://nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4@0dcd680e5585c791728c83342b311d0a0026dbeb" mount_location: /models/nemotron-3.5-lightning-nvfp4 auth_secret_name: hf_access_token allow_patterns: - "*.safetensors" - "*.json" - "*.py" - "*.jinja" - "*.txt" - "*.md" - source: "hf://nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DFlash@7fc1f1ff4b82b917efbd0710df0872c2bb89caa5" mount_location: /models/nemotron-3.5-lightning-nvfp4-dflash auth_secret_name: hf_access_token allow_patterns: - "*.safetensors" - "*.json" - "*.md" secrets: hf_access_token: null docker_server: start_command: >- vllm serve /models/nemotron-3.5-lightning-nvfp4 --served-model-name nvidia/nemotron-3.5-lightning-nvfp4 --host 0.0.0.0 --port 8000 --trust-remote-code --max-num-seqs 512 --max-model-len 1048576 --max-num-batched-tokens 32768 --enable-prefix-caching --mamba-cache-mode align --quantization modelopt_fp4 --speculative-config '{"method":"dflash","model":"/models/nemotron-3.5-lightning-nvfp4-dflash","num_speculative_tokens":3}' --moe-backend humming --linear-backend humming --mamba-backend flashinfer --mamba-ssm-cache-dtype float16 --enable-mamba-cache-stochastic-rounding --mamba-cache-philox-rounds 5 readiness_endpoint: /health liveness_endpoint: /health predict_endpoint: /v1/chat/completions server_port: 8000 resources: accelerator: H100 use_gpu: true runtime: predict_concurrency: 512 streaming_read_timeout: 3600 health_checks: startup_threshold_seconds: 1800 restart_check_delay_seconds: 300 restart_threshold_seconds: 900 stop_traffic_threshold_seconds: 120 environment_variables: VLLM_USE_RUST_FRONTEND: "1" VLLM_WORKER_MULTIPROC_METHOD: spawn VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS: "0" VLLM_CACHE_ROOT: /tmp/vllmcache TORCHINDUCTOR_CACHE_DIR: /tmp/inductor HF_HUB_OFFLINE: "1" ``` This config mounts NVIDIA's public NVFP4 verifier and DFlash drafter checkpoints through BDN, then launches vLLM with Humming MoE and quantized linear kernels, FlashInfer Mamba, and FP16 Mamba state with stochastic rounding. DFlash speculative decoding with three draft tokens combines with aligned prefix caching and a 512-sequence batch ceiling to push throughput on the model's full 1,048,576-token context. ## Flags The `start_command` passes these flags to the engine. Each one controls a runtime or serving behavior: | Flag | Value | What it does | | ------------------------------------------ | ------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `--trust-remote-code` | (no value) | Execute model-specific Python from the checkpoint (required for many Qwen, Phi, and custom architectures). | | `--max-num-seqs` | `512` | Maximum number of concurrent sequences in the batch. | | `--max-model-len` | `1048576` | Maximum context length (tokens) the server accepts per request. | | `--max-num-batched-tokens` | `32768` | Maximum total tokens processed per scheduler step. | | `--enable-prefix-caching` | (no value) | Reuse KV cache across requests that share a prefix. | | `--mamba-cache-mode` | `align` | Layout strategy for the Mamba state-space model cache on hybrid architectures. **align:** Align cache blocks across attention and Mamba layers for shared prefix reuse. | | `--quantization` | `modelopt_fp4` | Weight quantization method loaded at runtime. **modelopt\_fp4:** NVIDIA Model Optimizer NVFP4 (4-bit float) weights with W4A16 dequantization on Hopper. | | `--speculative-config` | `{"method":"dflash","model":"/models/nemotron-3.5-lightning-nvfp4-dflash","num_speculative_tokens":3}` | Speculative decoding configuration as a JSON object. The dotted form (`--speculative-config.method`, `--speculative-config.num_speculative_tokens`, ...) sets the same fields one at a time. | | `--moe-backend` | `humming` | MoE expert dispatch kernel. Engine-specific values select between routing implementations tuned for different hardware or model layouts. | | `--linear-backend` | `humming` | Kernel vLLM uses for linear (non-MoE) layers. **humming:** NVIDIA Humming quantized linear kernels for NVFP4 inference. | | `--mamba-backend` | `flashinfer` | Kernel vLLM uses for Mamba-2 state-space layers. **flashinfer:** FlashInfer Mamba kernels, optimized for hybrid attention-SSM architectures. | | `--mamba-ssm-cache-dtype` | `float16` | Precision for the Mamba state-space model cache on hybrid attention and SSM architectures. | | `--enable-mamba-cache-stochastic-rounding` | (no value) | Use stochastic rounding for FP16 Mamba state updates, preventing bias accumulation in the recurrent cache. | | `--mamba-cache-philox-rounds` | `5` | Number of Philox counter rounds per stochastic rounding step in the Mamba state cache. | ## Deploy Push the config to Baseten: ```sh theme={"system"} uvx truss push ``` You should see output similar to: ```output theme={"system"} ✨ Model nemotron-3.5-lightning-throughput was successfully pushed ✨ Model ID: abc1d2ef Deployment ID: xyz123 Endpoint: model-abc1d2ef.api.baseten.co Logs: https://app.baseten.co/models/abc1d2ef/logs/xyz123 ``` `truss push` prints your **model ID** (`abc1d2ef` in the example). The examples below use it wherever you see `{model_id}`, and read your API key from the `BASETEN_API_KEY` environment variable. ## Call the model Your deployment serves an OpenAI-compatible API. Now call your deployment to run inference: ```python main.py theme={"system"} import os from openai import OpenAI client = OpenAI( api_key=os.environ["BASETEN_API_KEY"], base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1", ) response = client.chat.completions.create( model="nvidia/nemotron-3.5-lightning-nvfp4", messages=[ {"role": "user", "content": "What is machine learning?"} ], ) print(response.choices[0].message.content) ``` ```sh theme={"system"} curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $BASETEN_API_KEY" \ -d '{ "model": "nvidia/nemotron-3.5-lightning-nvfp4", "messages": [ {"role": "user", "content": "What is machine learning?"} ] }' ``` ## Next steps Endpoint anatomy, authentication, and sync versus async inference Scale replicas with traffic, including scale to zero