> ## Documentation Index > Fetch the complete documentation index at: https://docs.baseten.co/llms.txt > Use this file to discover all available pages before exploring further. # Nemotron 3 Embed > Nemotron 3 Embed recipes: 2 variants (1B, 8B), Dense architecture.
Embeddings
## Setup Sign in to Baseten with Truss, then install the OpenAI SDK. **Sign in to Baseten** ```sh theme={"system"} uvx truss login --browser ``` **Install the OpenAI SDK** ```sh theme={"system"} uv pip install openai ``` Pick the model you want to deploy. Each tab is a self-contained recipe. [nvidia/Nemotron-3-Embed-1B-BF16](https://huggingface.co/nvidia/Nemotron-3-Embed-1B-BF16) is a 1B-parameter dense model with up to 32K context. This preset serves Nemotron 3 Embed 1B on a single H100 with BF16 weights through vLLM's OpenAI-compatible server, and pre-mounts the weights to local disk for fast cold starts. H100 vLLM 0.24.0 32K 32 ## Write the config Create and move into the project directory: ```sh theme={"system"} mkdir nemotron-3-embed-1b-bf16 && cd nemotron-3-embed-1b-bf16 ``` Then create a file named `config.yaml` and paste the following: ```yaml config.yaml theme={"system"} # yaml-language-server: $schema=https://raw.githubusercontent.com/basetenlabs/truss/main/truss/config.schema.json model_name: model:Nemotron-3-Embed-1B preset:bf16 model_metadata: example_model_input: model: nvidia/Nemotron-3-Embed-1B-BF16 input: - "query: What is the capital of France?" - "passage: Paris is the capital of France." encoding_format: float repo_id: nvidia/Nemotron-3-Embed-1B-BF16 tags: - openai-compatible - embedding - vllm base_image: image: vllm/vllm-openai:v0.24.0 docker_server: start_command: >- vllm serve /models/nemotron-1b --host 0.0.0.0 --port 8000 --served-model-name nvidia/Nemotron-3-Embed-1B-BF16 --max-model-len 32768 --hf-overrides '{"is_matryoshka":true,"matryoshka_dimensions":[512,1024,2048]}' readiness_endpoint: /health liveness_endpoint: /health predict_endpoint: /v1/embeddings server_port: 8000 # Weights are mirrored to the BDN at deploy time and pre-mounted at mount_location # before the container starts, so vLLM loads from local disk and never calls HF at # runtime (no cold-start 429s). @ pins the exact commit for reproducible pulls. weights: - source: "hf://nvidia/Nemotron-3-Embed-1B-BF16@c5e9806ae078a32aedc0829038410c7e94f5a748" mount_location: "/models/nemotron-1b" resources: accelerator: H100 use_gpu: true runtime: predict_concurrency: 32 health_checks: restart_threshold_seconds: 600 stop_traffic_threshold_seconds: 240 environment_variables: VLLM_LOGGING_LEVEL: INFO secrets: {} system_packages: [] requirements: [] ``` This config tells Baseten to serve `nvidia/Nemotron-3-Embed-1B-BF16` on a single H100 with the stock `vllm/vllm-openai:v0.24.0` image. The `weights:` block pins the checkpoint to a commit SHA, mirrors it to the Baseten Delivery Network at deploy time, and pre-mounts it at `/models/nemotron-1b`, so vLLM loads from local disk and never calls Hugging Face at runtime. The deployment exposes an OpenAI-compatible `/v1/embeddings` endpoint with Matryoshka output dimensions of 512, 1024, and 2048. ## Flags The `start_command` passes these flags to the engine. Each one controls a runtime or serving behavior: | Flag | Value | What it does | | ----------------- | ---------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | | `--max-model-len` | `32768` | Maximum context length (tokens) the server accepts per request. | | `--hf-overrides` | `{"is_matryoshka":true,"matryoshka_dimensions":[512,1024,2048]}` | Overrides fields in the model's Hugging Face config as a JSON object. The dotted form (`--hf-overrides.`) sets the same fields one at a time. | ## Deploy Push the config to Baseten: ```sh theme={"system"} uvx truss push ``` You should see output similar to: ```output theme={"system"} ✨ Model nemotron-3-embed-1b-bf16 was successfully pushed ✨ Model ID: abc1d2ef Deployment ID: xyz123 Endpoint: model-abc1d2ef.api.baseten.co Logs: https://app.baseten.co/models/abc1d2ef/logs/xyz123 ``` `truss push` prints your **model ID** (`abc1d2ef` in the example). The examples below use it wherever you see `{model_id}`, and read your API key from the `BASETEN_API_KEY` environment variable. ## Call the model Your deployment serves an OpenAI-compatible embeddings API at `/v1/embeddings`. Now call your deployment to generate embeddings: ```python main.py theme={"system"} import os from openai import OpenAI client = OpenAI( api_key=os.environ["BASETEN_API_KEY"], base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1", ) response = client.embeddings.create( model="nvidia/Nemotron-3-Embed-1B-BF16", input=[ "query: What is the capital of France?", "passage: Paris is the capital of France.", ], ) for item in response.data: print(len(item.embedding), item.embedding[:4]) ``` ```sh theme={"system"} curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/embeddings \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $BASETEN_API_KEY" \ -d '{ "model": "nvidia/Nemotron-3-Embed-1B-BF16", "input": [ "query: What is the capital of France?", "passage: Paris is the capital of France." ] }' ``` Nemotron 3 Embed is trained with retrieval prompts, and the `/v1/embeddings` endpoint doesn't add them for you. Prefix each input yourself: `query: ` for search queries and `passage: ` for the documents they match against, as shown above. Embeddings are 2048-dimensional by default. For smaller vectors, pass the OpenAI `dimensions` parameter with one of the values from the config's `--hf-overrides` flag (512, 1024, or 2048); the server slices and re-normalizes the vector, and rejects values outside that list. ```python theme={"system"} response = client.embeddings.create( model="nvidia/Nemotron-3-Embed-1B-BF16", input=["query: What is the capital of France?"], dimensions=512, ) print(len(response.data[0].embedding)) # 512 ``` For higher throughput, use the [Baseten Performance Client](https://www.baseten.co/blog/your-client-code-matters-10x-higher-embedding-throughput-with-python-and-rust/), which batches and pipelines requests automatically. [nvidia/Nemotron-3-Embed-8B-BF16](https://huggingface.co/nvidia/Nemotron-3-Embed-8B-BF16) is an 8B-parameter dense model with up to 32K context. This preset serves Nemotron 3 Embed 8B on a single H100 with BF16 weights through vLLM's OpenAI-compatible server, and pre-mounts the weights to local disk for fast cold starts. H100 vLLM 0.24.0 32K 32 ## Write the config Create and move into the project directory: ```sh theme={"system"} mkdir nemotron-3-embed-8b-bf16 && cd nemotron-3-embed-8b-bf16 ``` Then create a file named `config.yaml` and paste the following: ```yaml config.yaml theme={"system"} # yaml-language-server: $schema=https://raw.githubusercontent.com/basetenlabs/truss/main/truss/config.schema.json model_name: model:Nemotron-3-Embed-8B preset:bf16 model_metadata: example_model_input: model: nvidia/Nemotron-3-Embed-8B-BF16 input: - "query: What is the capital of France?" - "passage: Paris is the capital of France." encoding_format: float repo_id: nvidia/Nemotron-3-Embed-8B-BF16 tags: - openai-compatible - embedding - vllm base_image: image: vllm/vllm-openai:v0.24.0 docker_server: start_command: >- vllm serve /models/nemotron-8b --host 0.0.0.0 --port 8000 --served-model-name nvidia/Nemotron-3-Embed-8B-BF16 --max-model-len 32768 --hf-overrides '{"is_matryoshka":true,"matryoshka_dimensions":[512,1024,2048,4096]}' readiness_endpoint: /health liveness_endpoint: /health predict_endpoint: /v1/embeddings server_port: 8000 # Weights are mirrored to the BDN at deploy time and pre-mounted at mount_location # before the container starts, so vLLM loads from local disk and never calls HF at # runtime (no cold-start 429s). @ pins the exact commit for reproducible pulls. weights: - source: "hf://nvidia/Nemotron-3-Embed-8B-BF16@c663c3bd4f1f7792928dfd91db0b136d57631fca" mount_location: "/models/nemotron-8b" resources: accelerator: H100 use_gpu: true runtime: predict_concurrency: 32 health_checks: restart_threshold_seconds: 600 stop_traffic_threshold_seconds: 240 environment_variables: VLLM_LOGGING_LEVEL: INFO secrets: {} system_packages: [] requirements: [] ``` This config tells Baseten to serve `nvidia/Nemotron-3-Embed-8B-BF16` on a single H100 with the stock `vllm/vllm-openai:v0.24.0` image. The `weights:` block pins the checkpoint to a commit SHA, mirrors it to the Baseten Delivery Network at deploy time, and pre-mounts it at `/models/nemotron-8b`, so vLLM loads from local disk and never calls Hugging Face at runtime. The deployment exposes an OpenAI-compatible `/v1/embeddings` endpoint with Matryoshka output dimensions of 512, 1024, and 2048 alongside the native 4096 embedding width. ## Flags The `start_command` passes these flags to the engine. Each one controls a runtime or serving behavior: | Flag | Value | What it does | | ----------------- | --------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | | `--max-model-len` | `32768` | Maximum context length (tokens) the server accepts per request. | | `--hf-overrides` | `{"is_matryoshka":true,"matryoshka_dimensions":[512,1024,2048,4096]}` | Overrides fields in the model's Hugging Face config as a JSON object. The dotted form (`--hf-overrides.`) sets the same fields one at a time. | ## Deploy Push the config to Baseten: ```sh theme={"system"} uvx truss push ``` You should see output similar to: ```output theme={"system"} ✨ Model nemotron-3-embed-8b-bf16 was successfully pushed ✨ Model ID: abc1d2ef Deployment ID: xyz123 Endpoint: model-abc1d2ef.api.baseten.co Logs: https://app.baseten.co/models/abc1d2ef/logs/xyz123 ``` `truss push` prints your **model ID** (`abc1d2ef` in the example). The examples below use it wherever you see `{model_id}`, and read your API key from the `BASETEN_API_KEY` environment variable. ## Call the model Your deployment serves an OpenAI-compatible embeddings API at `/v1/embeddings`. Now call your deployment to generate embeddings: ```python main.py theme={"system"} import os from openai import OpenAI client = OpenAI( api_key=os.environ["BASETEN_API_KEY"], base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1", ) response = client.embeddings.create( model="nvidia/Nemotron-3-Embed-8B-BF16", input=[ "query: What is the capital of France?", "passage: Paris is the capital of France.", ], ) for item in response.data: print(len(item.embedding), item.embedding[:4]) ``` ```sh theme={"system"} curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/embeddings \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $BASETEN_API_KEY" \ -d '{ "model": "nvidia/Nemotron-3-Embed-8B-BF16", "input": [ "query: What is the capital of France?", "passage: Paris is the capital of France." ] }' ``` Nemotron 3 Embed is trained with retrieval prompts, and the `/v1/embeddings` endpoint doesn't add them for you. Prefix each input yourself: `query: ` for search queries and `passage: ` for the documents they match against, as shown above. Embeddings are 4096-dimensional by default. For smaller vectors, pass the OpenAI `dimensions` parameter with one of the values from the config's `--hf-overrides` flag (512, 1024, 2048, or 4096); the server slices and re-normalizes the vector, and rejects values outside that list. ```python theme={"system"} response = client.embeddings.create( model="nvidia/Nemotron-3-Embed-8B-BF16", input=["query: What is the capital of France?"], dimensions=512, ) print(len(response.data[0].embedding)) # 512 ``` For higher throughput, use the [Baseten Performance Client](https://www.baseten.co/blog/your-client-code-matters-10x-higher-embedding-throughput-with-python-and-rust/), which batches and pipelines requests automatically. ## Next steps Endpoint anatomy, authentication, and sync versus async inference Scale replicas with traffic, including scale to zero