> ## Documentation Index > Fetch the complete documentation index at: https://docs.baseten.co/llms.txt > Use this file to discover all available pages before exploring further. # Nemotron 3 Embed > Nemotron 3 Embed recipes: 2 variants (1B, 8B), Dense architecture.
Embeddings
## Setup Sign in to Baseten with Truss, then install the OpenAI SDK. **Sign in to Baseten** ```sh theme={"system"} uvx truss login --browser ``` **Install the OpenAI SDK** ```sh theme={"system"} uv pip install openai ``` Pick the model you want to deploy. Each tab is a self-contained recipe. [nvidia/Nemotron-3-Embed-1B-BF16](https://huggingface.co/nvidia/Nemotron-3-Embed-1B-BF16) is a 1B-parameter dense model with up to 32K context. This preset serves Nemotron 3 Embed 1B on a single H100 with BF16 weights through vLLM's OpenAI-compatible server, and pre-mounts the weights to local disk for fast cold starts. H100 vLLM 0.25.0 32K 16 ## Write the config Create and move into the project directory: ```sh theme={"system"} mkdir nemotron-3-embed-1b-bf16 && cd nemotron-3-embed-1b-bf16 ``` Then create a file named `config.yaml` and paste the following: ```yaml config.yaml theme={"system"} # yaml-language-server: $schema=https://raw.githubusercontent.com/basetenlabs/truss/main/truss/config.schema.json model_name: model:Nemotron-3-Embed-1B preset:bf16 model_metadata: example_model_input: model: nvidia/Nemotron-3-Embed-1B-BF16 input: - "query: What is the capital of France?" - "passage: Paris is the capital of France." encoding_format: float dimensions: 2048 truncate_prompt_tokens: 32768 truncation_side: right repo_id: nvidia/Nemotron-3-Embed-1B-BF16 tags: - openai-compatible - embedding - vllm base_image: image: vllm/vllm-openai:v0.25.0 docker_server: start_command: >- vllm serve /models/nemotron-1b --host 0.0.0.0 --port 8000 --served-model-name nvidia/Nemotron-3-Embed-1B-BF16 --runner pooling --hf-overrides.is_matryoshka true --max-model-len 32768 --max-num-batched-tokens 131072 --max-num-seqs 64 --enable-chunked-prefill --gpu-memory-utilization 0.95 readiness_endpoint: /health liveness_endpoint: /health predict_endpoint: /v1/embeddings server_port: 8000 weights: - source: "hf://nvidia/Nemotron-3-Embed-1B-BF16@c5e9806ae078a32aedc0829038410c7e94f5a748" mount_location: "/models/nemotron-1b" resources: accelerator: H100 use_gpu: true runtime: predict_concurrency: 16 health_checks: restart_threshold_seconds: 600 stop_traffic_threshold_seconds: 240 environment_variables: VLLM_LOGGING_LEVEL: INFO secrets: {} system_packages: [] requirements: [] ``` This config tells Baseten to serve `nvidia/Nemotron-3-Embed-1B-BF16` on a single H100 with the stock `vllm/vllm-openai:v0.25.0` image running the pooling runner. The `weights:` block pins the checkpoint to a commit SHA, mirrors it to the Baseten Delivery Network at deploy time, and pre-mounts it at `/models/nemotron-1b`, so vLLM loads from local disk and never calls Hugging Face at runtime. The deployment exposes an OpenAI-compatible `/v1/embeddings` endpoint with Matryoshka support, so requests can shrink the native 2048-dimensional embeddings with the `dimensions` parameter. ## Flags The `start_command` passes these flags to the engine. Each one controls a runtime or serving behavior: | Flag | Value | What it does | | ------------------------------ | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | | `--runner` | `pooling` | Which runner type vLLM uses for the model. **pooling:** Serve the model as a pooling model (embeddings or reranking) instead of text generation. | | `--hf-overrides.is_matryoshka` | `true` | Marks the model's embeddings as Matryoshka, so the server accepts a `dimensions` parameter and truncates output vectors to the requested size. | | `--max-model-len` | `32768` | Maximum context length (tokens) the server accepts per request. | | `--max-num-batched-tokens` | `131072` | Maximum total tokens processed per scheduler step. | | `--max-num-seqs` | `64` | Maximum number of concurrent sequences in the batch. | | `--enable-chunked-prefill` | (no value) | Process long prompts in chunks so decode requests keep running. | | `--gpu-memory-utilization` | `0.95` | Fraction of GPU memory vLLM may use for weights and KV cache. | ## Deploy Push the config to Baseten: ```sh theme={"system"} uvx truss push ``` You should see output similar to: ```output theme={"system"} ✨ Model nemotron-3-embed-1b-bf16 was successfully pushed ✨ Model ID: abc1d2ef Deployment ID: xyz123 Endpoint: model-abc1d2ef.api.baseten.co Logs: https://app.baseten.co/models/abc1d2ef/logs/xyz123 ``` `truss push` prints your **model ID** (`abc1d2ef` in the example). The examples below use it wherever you see `{model_id}`, and read your API key from the `BASETEN_API_KEY` environment variable. ## Call the model Your deployment serves an OpenAI-compatible embeddings API at `/v1/embeddings`. Now call your deployment to generate embeddings: ```python main.py theme={"system"} import os from openai import OpenAI client = OpenAI( api_key=os.environ["BASETEN_API_KEY"], base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1", ) response = client.embeddings.create( model="nvidia/Nemotron-3-Embed-1B-BF16", input=[ "query: What is the capital of France?", "passage: Paris is the capital of France.", ], ) for item in response.data: print(len(item.embedding), item.embedding[:4]) ``` ```sh theme={"system"} curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/embeddings \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $BASETEN_API_KEY" \ -d '{ "model": "nvidia/Nemotron-3-Embed-1B-BF16", "input": [ "query: What is the capital of France?", "passage: Paris is the capital of France." ] }' ``` Nemotron 3 Embed is trained with retrieval prompts, and the `/v1/embeddings` endpoint doesn't add them for you. Prefix each input yourself: `query: ` for search queries and `passage: ` for the documents they match against, as shown above. Embeddings are 2048-dimensional by default. For smaller vectors, pass the OpenAI `dimensions` parameter with any size up to 2048; because the config marks the model as Matryoshka (`--hf-overrides.is_matryoshka true`), the server truncates and re-normalizes the vector to the requested size. ```python theme={"system"} response = client.embeddings.create( model="nvidia/Nemotron-3-Embed-1B-BF16", input=["query: What is the capital of France?"], dimensions=512, ) print(len(response.data[0].embedding)) # 512 ``` For higher throughput, use the [Baseten Performance Client](https://www.baseten.co/blog/your-client-code-matters-10x-higher-embedding-throughput-with-python-and-rust/), which batches and pipelines requests automatically. [nvidia/Nemotron-3-Embed-8B-BF16](https://huggingface.co/nvidia/Nemotron-3-Embed-8B-BF16) is an 8B-parameter dense model with up to 32K context. This preset serves Nemotron 3 Embed 8B on a single H100 with BF16 weights through vLLM's OpenAI-compatible server, and pre-mounts the weights to local disk for fast cold starts. H100 vLLM 0.25.0 32K 4 ## Write the config Create and move into the project directory: ```sh theme={"system"} mkdir nemotron-3-embed-8b-bf16 && cd nemotron-3-embed-8b-bf16 ``` Then create a file named `config.yaml` and paste the following: ```yaml config.yaml theme={"system"} # yaml-language-server: $schema=https://raw.githubusercontent.com/basetenlabs/truss/main/truss/config.schema.json model_name: model:Nemotron-3-Embed-8B preset:bf16 model_metadata: example_model_input: model: nvidia/Nemotron-3-Embed-8B-BF16 input: - "query: What is the capital of France?" - "passage: Paris is the capital of France." encoding_format: float dimensions: 4096 truncate_prompt_tokens: 32768 truncation_side: right repo_id: nvidia/Nemotron-3-Embed-8B-BF16 tags: - openai-compatible - embedding - vllm base_image: image: vllm/vllm-openai:v0.25.0 docker_server: start_command: >- vllm serve /models/nemotron-8b --host 0.0.0.0 --port 8000 --served-model-name nvidia/Nemotron-3-Embed-8B-BF16 --runner pooling --hf-overrides.is_matryoshka true --max-model-len 32768 --max-num-batched-tokens 65536 --max-num-seqs 32 --enable-chunked-prefill --gpu-memory-utilization 0.95 readiness_endpoint: /health liveness_endpoint: /health predict_endpoint: /v1/embeddings server_port: 8000 weights: - source: "hf://nvidia/Nemotron-3-Embed-8B-BF16@c663c3bd4f1f7792928dfd91db0b136d57631fca" mount_location: "/models/nemotron-8b" resources: accelerator: H100 use_gpu: true runtime: predict_concurrency: 4 health_checks: restart_threshold_seconds: 600 stop_traffic_threshold_seconds: 240 environment_variables: VLLM_LOGGING_LEVEL: INFO secrets: {} system_packages: [] requirements: [] ``` This config tells Baseten to serve `nvidia/Nemotron-3-Embed-8B-BF16` on a single H100 with the stock `vllm/vllm-openai:v0.25.0` image running the pooling runner. The `weights:` block pins the checkpoint to a commit SHA, mirrors it to the Baseten Delivery Network at deploy time, and pre-mounts it at `/models/nemotron-8b`, so vLLM loads from local disk and never calls Hugging Face at runtime. The deployment exposes an OpenAI-compatible `/v1/embeddings` endpoint with Matryoshka support, so requests can shrink the native 4096-dimensional embeddings with the `dimensions` parameter. ## Flags The `start_command` passes these flags to the engine. Each one controls a runtime or serving behavior: | Flag | Value | What it does | | ------------------------------ | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | | `--runner` | `pooling` | Which runner type vLLM uses for the model. **pooling:** Serve the model as a pooling model (embeddings or reranking) instead of text generation. | | `--hf-overrides.is_matryoshka` | `true` | Marks the model's embeddings as Matryoshka, so the server accepts a `dimensions` parameter and truncates output vectors to the requested size. | | `--max-model-len` | `32768` | Maximum context length (tokens) the server accepts per request. | | `--max-num-batched-tokens` | `65536` | Maximum total tokens processed per scheduler step. | | `--max-num-seqs` | `32` | Maximum number of concurrent sequences in the batch. | | `--enable-chunked-prefill` | (no value) | Process long prompts in chunks so decode requests keep running. | | `--gpu-memory-utilization` | `0.95` | Fraction of GPU memory vLLM may use for weights and KV cache. | ## Deploy Push the config to Baseten: ```sh theme={"system"} uvx truss push ``` You should see output similar to: ```output theme={"system"} ✨ Model nemotron-3-embed-8b-bf16 was successfully pushed ✨ Model ID: abc1d2ef Deployment ID: xyz123 Endpoint: model-abc1d2ef.api.baseten.co Logs: https://app.baseten.co/models/abc1d2ef/logs/xyz123 ``` `truss push` prints your **model ID** (`abc1d2ef` in the example). The examples below use it wherever you see `{model_id}`, and read your API key from the `BASETEN_API_KEY` environment variable. ## Call the model Your deployment serves an OpenAI-compatible embeddings API at `/v1/embeddings`. Now call your deployment to generate embeddings: ```python main.py theme={"system"} import os from openai import OpenAI client = OpenAI( api_key=os.environ["BASETEN_API_KEY"], base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1", ) response = client.embeddings.create( model="nvidia/Nemotron-3-Embed-8B-BF16", input=[ "query: What is the capital of France?", "passage: Paris is the capital of France.", ], ) for item in response.data: print(len(item.embedding), item.embedding[:4]) ``` ```sh theme={"system"} curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/embeddings \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $BASETEN_API_KEY" \ -d '{ "model": "nvidia/Nemotron-3-Embed-8B-BF16", "input": [ "query: What is the capital of France?", "passage: Paris is the capital of France." ] }' ``` Nemotron 3 Embed is trained with retrieval prompts, and the `/v1/embeddings` endpoint doesn't add them for you. Prefix each input yourself: `query: ` for search queries and `passage: ` for the documents they match against, as shown above. Embeddings are 4096-dimensional by default. For smaller vectors, pass the OpenAI `dimensions` parameter with any size up to 4096; because the config marks the model as Matryoshka (`--hf-overrides.is_matryoshka true`), the server truncates and re-normalizes the vector to the requested size. ```python theme={"system"} response = client.embeddings.create( model="nvidia/Nemotron-3-Embed-8B-BF16", input=["query: What is the capital of France?"], dimensions=512, ) print(len(response.data[0].embedding)) # 512 ``` For higher throughput, use the [Baseten Performance Client](https://www.baseten.co/blog/your-client-code-matters-10x-higher-embedding-throughput-with-python-and-rust/), which batches and pipelines requests automatically. ## Next steps Endpoint anatomy, authentication, and sync versus async inference Scale replicas with traffic, including scale to zero