> ## Documentation Index
> Fetch the complete documentation index at: https://docs.baseten.co/llms.txt
> Use this file to discover all available pages before exploring further.

# Nemotron 3 Embed

> Nemotron 3 Embed recipes: 2 variants (1B, 8B), Dense architecture.

[Embeddings](https://docs.baseten.co/examples/models/capabilities/embedding)

## Setup

Install the Baseten CLI and sign in, then install the OpenAI SDK.

**Install and sign in to Baseten**

> [tab] macOS or Linux
>
> ```bash Terminal theme={"system"}
> brew tap basetenlabs/baseten
> brew trust --formula basetenlabs/baseten/baseten
> brew install baseten
> ```

> [tab] Windows
>
> Download and extract the Windows archive for the latest Baseten CLI release:
>
> ```powershell Terminal theme={"system"}
> $release = Invoke-RestMethod https://api.github.com/repos/basetenlabs/baseten-cli/releases/latest
> $asset = $release.assets | Where-Object { $_.name -like "*_windows_amd64.zip" }
> Invoke-WebRequest $asset.browser_download_url -OutFile baseten.zip
> Expand-Archive -Force baseten.zip .
> ```
>
> Move `baseten.exe` to a directory on your `PATH`.

For other platforms or a specific version, see the [Baseten CLI install reference](https://docs.baseten.co/reference/cli/baseten/overview.md#install).

```sh theme={"system"}
baseten auth login
```

**Install the OpenAI SDK**

```sh theme={"system"}
uv pip install openai
```

To install without Homebrew, see the [Baseten CLI install reference](https://docs.baseten.co/reference/cli/baseten/overview.md#install).

Pick the model you want to deploy. Each tab is a self-contained recipe.

> [tab] 1B
>
> [nvidia/Nemotron-3-Embed-1B-BF16](https://huggingface.co/nvidia/Nemotron-3-Embed-1B-BF16) is a 1B-parameter dense model with up to 32K context.
>
> This preset serves Nemotron 3 Embed 1B on a single H100 with BF16 weights through vLLM's OpenAI-compatible server, and pre-mounts the weights to local disk for fast cold starts.
>
> > [card] Hardware
> >
> > H100
>
> > [card] Engine
> >
> > vLLM 0.25.0
>
> > [card] Context
> >
> > 32K
>
> > [card] Concurrency
> >
> > 64
>
> ## Write the config
>
> Create and move into the project directory:
>
> ```sh theme={"system"}
> mkdir nemotron-3-embed-1b-throughput && cd nemotron-3-embed-1b-throughput
> ```
>
> Then create a file named `config.yaml` and paste the following:
>
> ```yaml config.yaml theme={"system"}
> # yaml-language-server: $schema=https://raw.githubusercontent.com/basetenlabs/truss/main/truss/config.schema.json
>
> model_name: model:Nemotron-3-Embed-1B preset:bf16
>
> model_metadata:
>   example_model_input:
>     model: nvidia/Nemotron-3-Embed-1B-BF16
>     input:
>       - "query: What is the capital of France?"
>       - "passage: Paris is the capital of France."
>     encoding_format: float
>     dimensions: 2048
>     truncate_prompt_tokens: 32768
>     truncation_side: right
>   repo_id: nvidia/Nemotron-3-Embed-1B-BF16
>   tags:
>     - openai-compatible
>     - embedding
>     - vllm
>
> base_image:
>   image: vllm/vllm-openai:v0.25.0
>
> docker_server:
>   start_command: >-
>     vllm serve /models/nemotron-1b
>     --host 0.0.0.0
>     --port 8000
>     --served-model-name nvidia/Nemotron-3-Embed-1B-BF16
>     --runner pooling
>     --hf-overrides.is_matryoshka true
>     --max-model-len 32768
>     --max-num-batched-tokens 131072
>     --max-num-seqs 64
>     --enable-chunked-prefill
>     --gpu-memory-utilization 0.95
>
>   readiness_endpoint: /health
>   liveness_endpoint: /health
>   predict_endpoint: /v1/embeddings
>   server_port: 8000
>
> weights:
>   - source: "hf://nvidia/Nemotron-3-Embed-1B-BF16@c5e9806ae078a32aedc0829038410c7e94f5a748"
>     mount_location: "/models/nemotron-1b"
>
> resources:
>   accelerator: H100
>   use_gpu: true
>
> runtime:
>   predict_concurrency: 64
>   health_checks:
>     restart_threshold_seconds: 600
>     stop_traffic_threshold_seconds: 240
>
> environment_variables:
>   VLLM_LOGGING_LEVEL: INFO
>
> secrets: []
> system_packages: []
> requirements: []
> ```
>
> This config tells Baseten to serve `nvidia/Nemotron-3-Embed-1B-BF16` on a single H100 with the stock `vllm/vllm-openai:v0.25.0` image running the pooling runner. The `weights:` block pins the checkpoint to a commit SHA, mirrors it to the Baseten Delivery Network at deploy time, and pre-mounts it at `/models/nemotron-1b`, so vLLM loads from local disk and never calls Hugging Face at runtime. The deployment exposes an OpenAI-compatible `/v1/embeddings` endpoint with Matryoshka support, so requests can shrink the native 2048-dimensional embeddings with the `dimensions` parameter.
>
> ## Flags
>
> The `start_command` passes these flags to the engine. Each one controls a runtime or serving behavior:
>
> | Flag | Value | What it does |
> | - | - | - |
> | `--runner` | `pooling` | Which runner type vLLM uses for the model. **pooling:** Serve the model as a pooling model (embeddings or reranking) instead of text generation. |
> | `--hf-overrides.is_matryoshka` | `true` | Marks the model's embeddings as Matryoshka, so the server accepts a `dimensions` parameter and truncates output vectors to the requested size. |
> | `--max-model-len` | `32768` | Maximum context length (tokens) the server accepts per request. |
> | `--max-num-batched-tokens` | `131072` | Maximum total tokens processed per scheduler step. |
> | `--max-num-seqs` | `64` | Maximum number of concurrent sequences in the batch. |
> | `--enable-chunked-prefill` | (no value) | Process long prompts in chunks so decode requests keep running. |
> | `--gpu-memory-utilization` | `0.95` | Fraction of GPU memory vLLM may use for weights and KV cache. |
>
> ## Deploy
>
> Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:
>
> ```sh Baseten CLI theme={"system"}
> baseten model push
> ```
>
> ```sh Truss CLI theme={"system"}
> uvx truss push
> ```
>
> You should see output similar to:
>
> ```output theme={"system"}
> Pushing model "nemotron-3-embed-1b-throughput"...
> Uploading model...
> Uploaded model in 0s
> ✨ Model nemotron-3-embed-1b-throughput was successfully pushed ✨
>
>   Model:       nemotron-3-embed-1b-throughput (abc1d2ef)
>   Deployment:  xyz123
>   Environment: production
>
> 🪵 View logs:
>    deployment:   baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
>    environment:  baseten model environment logs --model-id abc1d2ef --environment production  (once deployed)
>    app:          https://app.baseten.co/models/abc1d2ef/logs/xyz123
>
> 🚀 Invoke your model:
>    URL:  https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
>    CLI:  baseten model predict --model-id abc1d2ef
> ```
>
> `baseten model push` prints your **model ID** (`abc1d2ef` in the example). The examples below use it wherever you see `{model_id}`, and read your API key from the `BASETEN_API_KEY` environment variable.
>
> ## Call the model
>
> Your deployment serves an OpenAI-compatible embeddings API at `/v1/embeddings`.
>
> Now call your deployment to generate embeddings:
>
> > [tab] Python
> >
> > ```python main.py theme={"system"}
> > import os
> > from openai import OpenAI
> >
> > client = OpenAI(
> >     api_key=os.environ["BASETEN_API_KEY"],
> >     base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
> > )
> >
> > response = client.embeddings.create(
> >     model="nvidia/Nemotron-3-Embed-1B-BF16",
> >     input=[
> >         "query: What is the capital of France?",
> >         "passage: Paris is the capital of France.",
> >     ],
> > )
> >
> > for item in response.data:
> >     print(len(item.embedding), item.embedding[:4])
> > ```
>
> > [tab] cURL
> >
> > ```sh theme={"system"}
> > curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/embeddings \
> >   -H "Content-Type: application/json" \
> >   -H "Authorization: Bearer $BASETEN_API_KEY" \
> >   -d '{
> >     "model": "nvidia/Nemotron-3-Embed-1B-BF16",
> >     "input": [
> >       "query: What is the capital of France?",
> >       "passage: Paris is the capital of France."
> >     ]
> >   }'
> > ```
>
> Nemotron 3 Embed is trained with retrieval prompts, and the `/v1/embeddings`
> endpoint doesn't add them for you. Prefix each input yourself: `query: `
> for search queries and `passage: ` for the documents they match against,
> as shown above.
>
> Embeddings are 2048-dimensional by default. For smaller vectors, pass the
> OpenAI `dimensions` parameter with any size up to 2048; because the config
> marks the model as Matryoshka (`--hf-overrides.is_matryoshka true`), the
> server truncates and re-normalizes the vector to the requested size.
>
> ```python theme={"system"}
> response = client.embeddings.create(
>     model="nvidia/Nemotron-3-Embed-1B-BF16",
>     input=["query: What is the capital of France?"],
>     dimensions=512,
> )
> print(len(response.data[0].embedding))  # 512
> ```
>
> For higher throughput, use the [Baseten Performance Client](https://www.baseten.co/blog/your-client-code-matters-10x-higher-embedding-throughput-with-python-and-rust/), which batches and pipelines requests automatically.

> [tab] 8B
>
> [nvidia/Nemotron-3-Embed-8B-BF16](https://huggingface.co/nvidia/Nemotron-3-Embed-8B-BF16) is an 8B-parameter dense model with up to 32K context.
>
> This preset serves Nemotron 3 Embed 8B on a single H100 with BF16 weights through vLLM's OpenAI-compatible server, and pre-mounts the weights to local disk for fast cold starts.
>
> > [card] Hardware
> >
> > H100
>
> > [card] Engine
> >
> > vLLM 0.25.0
>
> > [card] Context
> >
> > 32K
>
> > [card] Concurrency
> >
> > 32
>
> ## Write the config
>
> Create and move into the project directory:
>
> ```sh theme={"system"}
> mkdir nemotron-3-embed-8b-throughput && cd nemotron-3-embed-8b-throughput
> ```
>
> Then create a file named `config.yaml` and paste the following:
>
> ```yaml config.yaml theme={"system"}
> # yaml-language-server: $schema=https://raw.githubusercontent.com/basetenlabs/truss/main/truss/config.schema.json
> model_name: model:Nemotron-3-Embed-8B preset:bf16
>
> model_metadata:
>   example_model_input:
>     model: nvidia/Nemotron-3-Embed-8B-BF16
>     input:
>       - "query: What is the capital of France?"
>       - "passage: Paris is the capital of France."
>     encoding_format: float
>     dimensions: 4096
>     truncate_prompt_tokens: 32768
>     truncation_side: right
>   repo_id: nvidia/Nemotron-3-Embed-8B-BF16
>   tags:
>     - openai-compatible
>     - embedding
>     - vllm
>
> base_image:
>   image: vllm/vllm-openai:v0.25.0
>
> docker_server:
>   start_command: >-
>     vllm serve /models/nemotron-8b
>     --host 0.0.0.0
>     --port 8000
>     --served-model-name nvidia/Nemotron-3-Embed-8B-BF16
>     --runner pooling
>     --hf-overrides.is_matryoshka true
>     --max-model-len 32768
>     --max-num-batched-tokens 65536
>     --max-num-seqs 32
>     --enable-chunked-prefill
>     --gpu-memory-utilization 0.95
>
>   readiness_endpoint: /health
>   liveness_endpoint: /health
>   predict_endpoint: /v1/embeddings
>   server_port: 8000
>
> weights:
>   - source: "hf://nvidia/Nemotron-3-Embed-8B-BF16@c663c3bd4f1f7792928dfd91db0b136d57631fca"
>     mount_location: "/models/nemotron-8b"
>
> resources:
>   accelerator: H100
>   use_gpu: true
>
> runtime:
>   predict_concurrency: 32
>   health_checks:
>     restart_threshold_seconds: 600
>     stop_traffic_threshold_seconds: 240
>
> environment_variables:
>   VLLM_LOGGING_LEVEL: INFO
>
> secrets: []
> system_packages: []
> requirements: []
> ```
>
> This config tells Baseten to serve `nvidia/Nemotron-3-Embed-8B-BF16` on a single H100 with the stock `vllm/vllm-openai:v0.25.0` image running the pooling runner. The `weights:` block pins the checkpoint to a commit SHA, mirrors it to the Baseten Delivery Network at deploy time, and pre-mounts it at `/models/nemotron-8b`, so vLLM loads from local disk and never calls Hugging Face at runtime. The deployment exposes an OpenAI-compatible `/v1/embeddings` endpoint with Matryoshka support, so requests can shrink the native 4096-dimensional embeddings with the `dimensions` parameter.
>
> ## Flags
>
> The `start_command` passes these flags to the engine. Each one controls a runtime or serving behavior:
>
> | Flag | Value | What it does |
> | - | - | - |
> | `--runner` | `pooling` | Which runner type vLLM uses for the model. **pooling:** Serve the model as a pooling model (embeddings or reranking) instead of text generation. |
> | `--hf-overrides.is_matryoshka` | `true` | Marks the model's embeddings as Matryoshka, so the server accepts a `dimensions` parameter and truncates output vectors to the requested size. |
> | `--max-model-len` | `32768` | Maximum context length (tokens) the server accepts per request. |
> | `--max-num-batched-tokens` | `65536` | Maximum total tokens processed per scheduler step. |
> | `--max-num-seqs` | `32` | Maximum number of concurrent sequences in the batch. |
> | `--enable-chunked-prefill` | (no value) | Process long prompts in chunks so decode requests keep running. |
> | `--gpu-memory-utilization` | `0.95` | Fraction of GPU memory vLLM may use for weights and KV cache. |
>
> ## Deploy
>
> Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:
>
> ```sh Baseten CLI theme={"system"}
> baseten model push
> ```
>
> ```sh Truss CLI theme={"system"}
> uvx truss push
> ```
>
> You should see output similar to:
>
> ```output theme={"system"}
> Pushing model "nemotron-3-embed-8b-throughput"...
> Uploading model...
> Uploaded model in 0s
> ✨ Model nemotron-3-embed-8b-throughput was successfully pushed ✨
>
>   Model:       nemotron-3-embed-8b-throughput (abc1d2ef)
>   Deployment:  xyz123
>   Environment: production
>
> 🪵 View logs:
>    deployment:   baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
>    environment:  baseten model environment logs --model-id abc1d2ef --environment production  (once deployed)
>    app:          https://app.baseten.co/models/abc1d2ef/logs/xyz123
>
> 🚀 Invoke your model:
>    URL:  https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
>    CLI:  baseten model predict --model-id abc1d2ef
> ```
>
> `baseten model push` prints your **model ID** (`abc1d2ef` in the example). The examples below use it wherever you see `{model_id}`, and read your API key from the `BASETEN_API_KEY` environment variable.
>
> ## Call the model
>
> Your deployment serves an OpenAI-compatible embeddings API at `/v1/embeddings`.
>
> Now call your deployment to generate embeddings:
>
> > [tab] Python
> >
> > ```python main.py theme={"system"}
> > import os
> > from openai import OpenAI
> >
> > client = OpenAI(
> >     api_key=os.environ["BASETEN_API_KEY"],
> >     base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
> > )
> >
> > response = client.embeddings.create(
> >     model="nvidia/Nemotron-3-Embed-8B-BF16",
> >     input=[
> >         "query: What is the capital of France?",
> >         "passage: Paris is the capital of France.",
> >     ],
> > )
> >
> > for item in response.data:
> >     print(len(item.embedding), item.embedding[:4])
> > ```
>
> > [tab] cURL
> >
> > ```sh theme={"system"}
> > curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/embeddings \
> >   -H "Content-Type: application/json" \
> >   -H "Authorization: Bearer $BASETEN_API_KEY" \
> >   -d '{
> >     "model": "nvidia/Nemotron-3-Embed-8B-BF16",
> >     "input": [
> >       "query: What is the capital of France?",
> >       "passage: Paris is the capital of France."
> >     ]
> >   }'
> > ```
>
> Nemotron 3 Embed is trained with retrieval prompts, and the `/v1/embeddings`
> endpoint doesn't add them for you. Prefix each input yourself: `query: `
> for search queries and `passage: ` for the documents they match against,
> as shown above.
>
> Embeddings are 4096-dimensional by default. For smaller vectors, pass the
> OpenAI `dimensions` parameter with any size up to 4096; because the config
> marks the model as Matryoshka (`--hf-overrides.is_matryoshka true`), the
> server truncates and re-normalizes the vector to the requested size.
>
> ```python theme={"system"}
> response = client.embeddings.create(
>     model="nvidia/Nemotron-3-Embed-8B-BF16",
>     input=["query: What is the capital of France?"],
>     dimensions=512,
> )
> print(len(response.data[0].embedding))  # 512
> ```
>
> For higher throughput, use the [Baseten Performance Client](https://www.baseten.co/blog/your-client-code-matters-10x-higher-embedding-throughput-with-python-and-rust/), which batches and pipelines requests automatically.

## Next steps

> [card] [Call your model](https://docs.baseten.co/inference/calling-your-model.md)
>
> Endpoint anatomy, authentication, and sync versus async inference

> [card] [Autoscaling](https://docs.baseten.co/deployment/autoscaling/overview.md)
>
> Scale replicas with traffic, including scale to zero
