Nemotron 3 Embed

Setup

Sign in to Baseten

uvx truss login --browser

Install the OpenAI SDK

uv pip install openai

Pick the model you want to deploy. Each tab is a self-contained recipe.

nvidia/Nemotron-3-Embed-1B-BF16 is a 1B-parameter dense model with up to 32K context.This preset serves Nemotron 3 Embed 1B on a single H100 with BF16 weights through vLLM’s OpenAI-compatible server, and pre-mounts the weights to local disk for fast cold starts.

Hardware

H100

Engine

vLLM 0.24.0

Context

32K

Concurrency

Write the config

Create and move into the project directory:

mkdir nemotron-3-embed-1b-bf16 && cd nemotron-3-embed-1b-bf16

Then create a file named config.yaml and paste the following:

config.yaml

# yaml-language-server: $schema=https://raw.githubusercontent.com/basetenlabs/truss/main/truss/config.schema.json
model_name: model:Nemotron-3-Embed-1B preset:bf16

model_metadata:
  example_model_input:
    model: nvidia/Nemotron-3-Embed-1B-BF16
    input:
      - "query: What is the capital of France?"
      - "passage: Paris is the capital of France."
    encoding_format: float
  repo_id: nvidia/Nemotron-3-Embed-1B-BF16
  tags:
    - openai-compatible
    - embedding
    - vllm

base_image:
  image: vllm/vllm-openai:v0.24.0
docker_server:
  start_command: >-
    vllm serve /models/nemotron-1b
    --host 0.0.0.0
    --port 8000
    --served-model-name nvidia/Nemotron-3-Embed-1B-BF16
    --max-model-len 32768
    --hf-overrides '{"is_matryoshka":true,"matryoshka_dimensions":[512,1024,2048]}'
  readiness_endpoint: /health
  liveness_endpoint: /health
  predict_endpoint: /v1/embeddings
  server_port: 8000

# Weights are mirrored to the BDN at deploy time and pre-mounted at mount_location
# before the container starts, so vLLM loads from local disk and never calls HF at
# runtime (no cold-start 429s). @<sha> pins the exact commit for reproducible pulls.
weights:
  - source: "hf://nvidia/Nemotron-3-Embed-1B-BF16@c5e9806ae078a32aedc0829038410c7e94f5a748"
    mount_location: "/models/nemotron-1b"

resources:
  accelerator: H100
  use_gpu: true

runtime:
  predict_concurrency: 32
  health_checks:
    restart_threshold_seconds: 600
    stop_traffic_threshold_seconds: 240

environment_variables:
  VLLM_LOGGING_LEVEL: INFO

secrets: {}
system_packages: []
requirements: []

This config tells Baseten to serve nvidia/Nemotron-3-Embed-1B-BF16 on a single H100 with the stock vllm/vllm-openai:v0.24.0 image. The weights: block pins the checkpoint to a commit SHA, mirrors it to the Baseten Delivery Network at deploy time, and pre-mounts it at /models/nemotron-1b, so vLLM loads from local disk and never calls Hugging Face at runtime. The deployment exposes an OpenAI-compatible /v1/embeddings endpoint with Matryoshka output dimensions of 512, 1024, and 2048.

Flags

The start_command passes these flags to the engine. Each one controls a runtime or serving behavior:

Flag	Value	What it does
`--max-model-len`	`32768`	Maximum context length (tokens) the server accepts per request.
`--hf-overrides`	`{"is_matryoshka":true,"matryoshka_dimensions":[512,1024,2048]}`	Overrides fields in the model’s Hugging Face config as a JSON object. The dotted form (`--hf-overrides.<field>`) sets the same fields one at a time.

Deploy

Push the config to Baseten:

uvx truss push

You should see output similar to:

✨ Model nemotron-3-embed-1b-bf16 was successfully pushed ✨

   Model ID:      abc1d2ef
   Deployment ID: xyz123
   Endpoint:      model-abc1d2ef.api.baseten.co
   Logs:          https://app.baseten.co/models/abc1d2ef/logs/xyz123

truss push prints your model ID (abc1d2ef in the example). The examples below use it wherever you see {model_id}, and read your API key from the BASETEN_API_KEY environment variable.

Call the model

Your deployment serves an OpenAI-compatible embeddings API at /v1/embeddings.Now call your deployment to generate embeddings:

Python
cURL

main.py

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["BASETEN_API_KEY"],
    base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)

response = client.embeddings.create(
    model="nvidia/Nemotron-3-Embed-1B-BF16",
    input=[
        "query: What is the capital of France?",
        "passage: Paris is the capital of France.",
    ],
)

for item in response.data:
    print(len(item.embedding), item.embedding[:4])

curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/embeddings \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '{
    "model": "nvidia/Nemotron-3-Embed-1B-BF16",
    "input": [
      "query: What is the capital of France?",
      "passage: Paris is the capital of France."
    ]
  }'

Nemotron 3 Embed is trained with retrieval prompts, and the /v1/embeddings endpoint doesn’t add them for you. Prefix each input yourself: query: for search queries and passage: for the documents they match against, as shown above.Embeddings are 2048-dimensional by default. For smaller vectors, pass the OpenAI dimensions parameter with one of the values from the config’s --hf-overrides flag (512, 1024, or 2048); the server slices and re-normalizes the vector, and rejects values outside that list.

response = client.embeddings.create(
    model="nvidia/Nemotron-3-Embed-1B-BF16",
    input=["query: What is the capital of France?"],
    dimensions=512,
)
print(len(response.data[0].embedding))  # 512

For higher throughput, use the Baseten Performance Client, which batches and pipelines requests automatically.

nvidia/Nemotron-3-Embed-8B-BF16 is an 8B-parameter dense model with up to 32K context.This preset serves Nemotron 3 Embed 8B on a single H100 with BF16 weights through vLLM’s OpenAI-compatible server, and pre-mounts the weights to local disk for fast cold starts.

Hardware

H100

Engine

vLLM 0.24.0

Context

32K

Concurrency

Write the config

Create and move into the project directory:

mkdir nemotron-3-embed-8b-bf16 && cd nemotron-3-embed-8b-bf16

Then create a file named config.yaml and paste the following:

config.yaml

# yaml-language-server: $schema=https://raw.githubusercontent.com/basetenlabs/truss/main/truss/config.schema.json
model_name: model:Nemotron-3-Embed-8B preset:bf16

model_metadata:
  example_model_input:
    model: nvidia/Nemotron-3-Embed-8B-BF16
    input:
      - "query: What is the capital of France?"
      - "passage: Paris is the capital of France."
    encoding_format: float
  repo_id: nvidia/Nemotron-3-Embed-8B-BF16
  tags:
    - openai-compatible
    - embedding
    - vllm

base_image:
  image: vllm/vllm-openai:v0.24.0
docker_server:
  start_command: >-
    vllm serve /models/nemotron-8b
    --host 0.0.0.0
    --port 8000
    --served-model-name nvidia/Nemotron-3-Embed-8B-BF16
    --max-model-len 32768
    --hf-overrides '{"is_matryoshka":true,"matryoshka_dimensions":[512,1024,2048,4096]}'
  readiness_endpoint: /health
  liveness_endpoint: /health
  predict_endpoint: /v1/embeddings
  server_port: 8000

# Weights are mirrored to the BDN at deploy time and pre-mounted at mount_location
# before the container starts, so vLLM loads from local disk and never calls HF at
# runtime (no cold-start 429s). @<sha> pins the exact commit for reproducible pulls.
weights:
  - source: "hf://nvidia/Nemotron-3-Embed-8B-BF16@c663c3bd4f1f7792928dfd91db0b136d57631fca"
    mount_location: "/models/nemotron-8b"

resources:
  accelerator: H100
  use_gpu: true

runtime:
  predict_concurrency: 32
  health_checks:
    restart_threshold_seconds: 600
    stop_traffic_threshold_seconds: 240

environment_variables:
  VLLM_LOGGING_LEVEL: INFO

secrets: {}
system_packages: []
requirements: []

This config tells Baseten to serve nvidia/Nemotron-3-Embed-8B-BF16 on a single H100 with the stock vllm/vllm-openai:v0.24.0 image. The weights: block pins the checkpoint to a commit SHA, mirrors it to the Baseten Delivery Network at deploy time, and pre-mounts it at /models/nemotron-8b, so vLLM loads from local disk and never calls Hugging Face at runtime. The deployment exposes an OpenAI-compatible /v1/embeddings endpoint with Matryoshka output dimensions of 512, 1024, and 2048 alongside the native 4096 embedding width.

Flags

The start_command passes these flags to the engine. Each one controls a runtime or serving behavior:

Flag	Value	What it does
`--max-model-len`	`32768`	Maximum context length (tokens) the server accepts per request.
`--hf-overrides`	`{"is_matryoshka":true,"matryoshka_dimensions":[512,1024,2048,4096]}`	Overrides fields in the model’s Hugging Face config as a JSON object. The dotted form (`--hf-overrides.<field>`) sets the same fields one at a time.

Deploy

Push the config to Baseten:

uvx truss push

You should see output similar to:

✨ Model nemotron-3-embed-8b-bf16 was successfully pushed ✨

   Model ID:      abc1d2ef
   Deployment ID: xyz123
   Endpoint:      model-abc1d2ef.api.baseten.co
   Logs:          https://app.baseten.co/models/abc1d2ef/logs/xyz123

truss push prints your model ID (abc1d2ef in the example). The examples below use it wherever you see {model_id}, and read your API key from the BASETEN_API_KEY environment variable.

Call the model

Your deployment serves an OpenAI-compatible embeddings API at /v1/embeddings.Now call your deployment to generate embeddings:

Python
cURL

main.py

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["BASETEN_API_KEY"],
    base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)

response = client.embeddings.create(
    model="nvidia/Nemotron-3-Embed-8B-BF16",
    input=[
        "query: What is the capital of France?",
        "passage: Paris is the capital of France.",
    ],
)

for item in response.data:
    print(len(item.embedding), item.embedding[:4])

curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/embeddings \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '{
    "model": "nvidia/Nemotron-3-Embed-8B-BF16",
    "input": [
      "query: What is the capital of France?",
      "passage: Paris is the capital of France."
    ]
  }'

Nemotron 3 Embed is trained with retrieval prompts, and the /v1/embeddings endpoint doesn’t add them for you. Prefix each input yourself: query: for search queries and passage: for the documents they match against, as shown above.Embeddings are 4096-dimensional by default. For smaller vectors, pass the OpenAI dimensions parameter with one of the values from the config’s --hf-overrides flag (512, 1024, 2048, or 4096); the server slices and re-normalizes the vector, and rejects values outside that list.

response = client.embeddings.create(
    model="nvidia/Nemotron-3-Embed-8B-BF16",
    input=["query: What is the capital of France?"],
    dimensions=512,
)
print(len(response.data[0].embedding))  # 512

For higher throughput, use the Baseten Performance Client, which batches and pipelines requests automatically.

Nemotron 3 Embed

Setup

Hardware

Engine

Context

Concurrency

Write the config

Flags

Deploy

Call the model

Hardware

Engine

Context

Concurrency

Write the config

Flags

Deploy

Call the model

Next steps

Call your model

Autoscaling

​Setup

Hardware

Engine

Context

Concurrency

​Write the config

​Flags

​Deploy

​Call the model

Hardware

Engine

Context

Concurrency

​Write the config

​Flags

​Deploy

​Call the model

​Next steps

Call your model

Autoscaling

Setup

Write the config

Flags

Deploy

Call the model

Write the config

Flags

Deploy

Call the model

Next steps