> ## Documentation Index > Fetch the complete documentation index at: https://docs.baseten.co/llms.txt > Use this file to discover all available pages before exploring further. # Qwen3 Embedding > Qwen3 Embedding recipes: 3 variants (0.6B, 4B, 8B), Dense architecture.
Embeddings
## Setup Install the Baseten CLI and sign in, then install the OpenAI SDK. **Install and sign in to Baseten** ```bash Terminal theme={"system"} brew tap basetenlabs/baseten brew trust --formula basetenlabs/baseten/baseten brew install baseten ``` Download and extract the Windows archive for the latest Baseten CLI release: ```powershell Terminal theme={"system"} $release = Invoke-RestMethod https://api.github.com/repos/basetenlabs/baseten-cli/releases/latest $asset = $release.assets | Where-Object { $_.name -like "*_windows_amd64.zip" } Invoke-WebRequest $asset.browser_download_url -OutFile baseten.zip Expand-Archive -Force baseten.zip . ``` Move `baseten.exe` to a directory on your `PATH`. For other platforms or a specific version, see the [Baseten CLI install reference](/reference/cli/baseten/overview#install). ```sh theme={"system"} baseten auth login ``` **Install the OpenAI SDK** ```sh theme={"system"} uv pip install openai ``` To install without Homebrew, see the [Baseten CLI install reference](/reference/cli/baseten/overview#install). Pick the model you want to deploy. Each tab is a self-contained recipe. [Qwen/Qwen3-Embedding-0.6B](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B) is a 0.6B-parameter dense model. This preset serves Qwen3 Embedding 0.6B on a single L4 through [Baseten Embeddings Inference](/engines/bei/overview) (BEI) with FP8 weights, optimized for embedding throughput on low-cost hardware. L4:4x16 TRT-LLM ## Write the config Create and move into the project directory: ```sh theme={"system"} mkdir qwen3-embedding-0.6b-throughput && cd qwen3-embedding-0.6b-throughput ``` Then create a file named `config.yaml` and paste the following: ```yaml config.yaml theme={"system"} model_metadata: example_model_input: input: - Baseten is a fast inference provider - Embeddings let you do semantic search. model: qwen3-embedding-0.6b model_name: "model:qwen3-embedding-0.6b preset:throughput" python_version: py311 resources: instance_type: L4:4x16 trt_llm: build: base_model: encoder checkpoint_repository: repo: michaelfeil/Qwen3-Embedding-0.6B-auto revision: main source: HF max_num_tokens: 32768 num_builder_gpus: 1 quantization_type: fp8 runtime: webserver_default_route: /v1/embeddings ``` This config tells Baseten to build a BEI (Baseten Embeddings Inference) engine for Qwen3 Embedding 0.6B on a single L4, drawing FP8 weights from `michaelfeil/Qwen3-Embedding-0.6B-auto`, a mirror of the official checkpoint with an architecture string compatible with BEI's encoder build path. FP8 quantization on an L4 keeps per-embedding cost low while dynamic batching sustains high throughput. ## Key parameters [Baseten Embeddings Inference](/engines/bei/overview) (BEI) reads these fields from the `trt_llm` block. Each one shapes how the engine is built and served: | Parameter | Value | | - | - | | Quantization | `fp8` | | Base model type | `encoder` | ## Deploy Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it: ```sh Baseten CLI theme={"system"} baseten model push ``` ```sh Truss CLI theme={"system"} uvx truss push ``` You should see output similar to: ```output theme={"system"} Pushing model "qwen3-embedding-0.6b-throughput"... Uploading model... Uploaded model in 0s ✨ Model qwen3-embedding-0.6b-throughput was successfully pushed ✨ Model: qwen3-embedding-0.6b-throughput (abc1d2ef) Deployment: xyz123 Environment: production 🪵 View logs: deployment: baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123 environment: baseten model environment logs --model-id abc1d2ef --environment production (once deployed) app: https://app.baseten.co/models/abc1d2ef/logs/xyz123 🚀 Invoke your model: URL: https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict CLI: baseten model predict --model-id abc1d2ef ``` `baseten model push` prints your **model ID** (`abc1d2ef` in the example). The examples below use it wherever you see `{model_id}`, and read your API key from the `BASETEN_API_KEY` environment variable. ## Call the model Your deployment serves an OpenAI-compatible embeddings API at `/v1/embeddings`. Now call your deployment to generate embeddings: ```python main.py theme={"system"} import os from openai import OpenAI client = OpenAI( api_key=os.environ["BASETEN_API_KEY"], base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1", ) response = client.embeddings.create( model="qwen3-embedding-0.6b", input=[ "Baseten is a fast inference provider", "Embeddings let you do semantic search.", ], ) for item in response.data: print(len(item.embedding), item.embedding[:4]) ``` ```sh theme={"system"} curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/embeddings \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $BASETEN_API_KEY" \ -d '{ "model": "qwen3-embedding-0.6b", "input": [ "Baseten is a fast inference provider", "Embeddings let you do semantic search." ] }' ``` For higher throughput, use the [Baseten Performance Client](https://www.baseten.co/blog/your-client-code-matters-10x-higher-embedding-throughput-with-python-and-rust/), which batches and pipelines requests automatically. [Qwen/Qwen3-Embedding-4B](https://huggingface.co/Qwen/Qwen3-Embedding-4B) is a 4B-parameter dense model. This preset serves Qwen3 Embedding 4B on a single H100 through [Baseten Embeddings Inference](/engines/bei/overview) (BEI) with FP8 weights, optimized for embedding throughput. H100 TRT-LLM ## Write the config Create and move into the project directory: ```sh theme={"system"} mkdir qwen3-embedding-4b-throughput && cd qwen3-embedding-4b-throughput ``` Then create a file named `config.yaml` and paste the following: ```yaml config.yaml theme={"system"} model_metadata: example_model_input: input: - Baseten is a fast inference provider - Embeddings let you do semantic search. model: qwen3-embedding-4b model_name: "model:qwen3-embedding-4b preset:throughput" python_version: py311 resources: instance_type: H100 trt_llm: build: base_model: encoder checkpoint_repository: repo: michaelfeil/Qwen3-Embedding-4B-auto revision: main source: HF max_num_tokens: 32768 num_builder_gpus: 1 quantization_type: fp8 runtime: webserver_default_route: /v1/embeddings ``` This config tells Baseten to build a BEI (Baseten Embeddings Inference) engine for Qwen3 Embedding 4B on a single H100, drawing FP8 weights from `michaelfeil/Qwen3-Embedding-4B-auto`, a mirror of the official checkpoint with an architecture string compatible with BEI's encoder build path. FP8 quantization and dynamic batching keep throughput high for indexing and RAG ingest workloads. ## Key parameters [Baseten Embeddings Inference](/engines/bei/overview) (BEI) reads these fields from the `trt_llm` block. Each one shapes how the engine is built and served: | Parameter | Value | | - | - | | Quantization | `fp8` | | Base model type | `encoder` | ## Deploy Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it: ```sh Baseten CLI theme={"system"} baseten model push ``` ```sh Truss CLI theme={"system"} uvx truss push ``` You should see output similar to: ```output theme={"system"} Pushing model "qwen3-embedding-4b-throughput"... Uploading model... Uploaded model in 0s ✨ Model qwen3-embedding-4b-throughput was successfully pushed ✨ Model: qwen3-embedding-4b-throughput (abc1d2ef) Deployment: xyz123 Environment: production 🪵 View logs: deployment: baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123 environment: baseten model environment logs --model-id abc1d2ef --environment production (once deployed) app: https://app.baseten.co/models/abc1d2ef/logs/xyz123 🚀 Invoke your model: URL: https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict CLI: baseten model predict --model-id abc1d2ef ``` `baseten model push` prints your **model ID** (`abc1d2ef` in the example). The examples below use it wherever you see `{model_id}`, and read your API key from the `BASETEN_API_KEY` environment variable. ## Call the model Your deployment serves an OpenAI-compatible embeddings API at `/v1/embeddings`. Now call your deployment to generate embeddings: ```python main.py theme={"system"} import os from openai import OpenAI client = OpenAI( api_key=os.environ["BASETEN_API_KEY"], base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1", ) response = client.embeddings.create( model="qwen3-embedding-4b", input=[ "Baseten is a fast inference provider", "Embeddings let you do semantic search.", ], ) for item in response.data: print(len(item.embedding), item.embedding[:4]) ``` ```sh theme={"system"} curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/embeddings \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $BASETEN_API_KEY" \ -d '{ "model": "qwen3-embedding-4b", "input": [ "Baseten is a fast inference provider", "Embeddings let you do semantic search." ] }' ``` For higher throughput, use the [Baseten Performance Client](https://www.baseten.co/blog/your-client-code-matters-10x-higher-embedding-throughput-with-python-and-rust/), which batches and pipelines requests automatically. [Qwen/Qwen3-Embedding-8B](https://huggingface.co/Qwen/Qwen3-Embedding-8B) is an 8B-parameter dense model. H100 TRT-LLM ## Write the config Create and move into the project directory: ```sh theme={"system"} mkdir qwen3-embedding-8b && cd qwen3-embedding-8b ``` Then create a file named `config.yaml` and paste the following: ```yaml config.yaml theme={"system"} model_metadata: example_model_input: input: - Baseten is a fast inference provider - Embeddings let you do semantic search. model: qwen3-embedding-8b model_name: "model:qwen3-embedding-8b preset:throughput" python_version: py311 resources: instance_type: H100 trt_llm: build: base_model: encoder checkpoint_repository: repo: michaelfeil/Qwen3-Embedding-8B-auto revision: main source: HF max_num_tokens: 40960 num_builder_gpus: 1 quantization_type: fp8 runtime: webserver_default_route: /v1/embeddings ``` ## Key parameters [Baseten Embeddings Inference](/engines/bei/overview) (BEI) reads these fields from the `trt_llm` block. Each one shapes how the engine is built and served: | Parameter | Value | | - | - | | Quantization | `fp8` | | Base model type | `encoder` | ## Deploy Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it: ```sh Baseten CLI theme={"system"} baseten model push ``` ```sh Truss CLI theme={"system"} uvx truss push ``` You should see output similar to: ```output theme={"system"} Pushing model "qwen3-embedding-8b"... Uploading model... Uploaded model in 0s ✨ Model qwen3-embedding-8b was successfully pushed ✨ Model: qwen3-embedding-8b (abc1d2ef) Deployment: xyz123 Environment: production 🪵 View logs: deployment: baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123 environment: baseten model environment logs --model-id abc1d2ef --environment production (once deployed) app: https://app.baseten.co/models/abc1d2ef/logs/xyz123 🚀 Invoke your model: URL: https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict CLI: baseten model predict --model-id abc1d2ef ``` `baseten model push` prints your **model ID** (`abc1d2ef` in the example). The examples below use it wherever you see `{model_id}`, and read your API key from the `BASETEN_API_KEY` environment variable. ## Call the model Your deployment serves an OpenAI-compatible embeddings API at `/v1/embeddings`. Now call your deployment to generate embeddings: ```python main.py theme={"system"} import os from openai import OpenAI client = OpenAI( api_key=os.environ["BASETEN_API_KEY"], base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1", ) response = client.embeddings.create( model="qwen3-embedding-8b", input=[ "Baseten is a fast inference provider", "Embeddings let you do semantic search.", ], ) for item in response.data: print(len(item.embedding), item.embedding[:4]) ``` ```sh theme={"system"} curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/embeddings \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $BASETEN_API_KEY" \ -d '{ "model": "qwen3-embedding-8b", "input": [ "Baseten is a fast inference provider", "Embeddings let you do semantic search." ] }' ``` For higher throughput, use the [Baseten Performance Client](https://www.baseten.co/blog/your-client-code-matters-10x-higher-embedding-throughput-with-python-and-rust/), which batches and pipelines requests automatically. ## Next steps Endpoint anatomy, authentication, and sync versus async inference Scale replicas with traffic, including scale to zero