> ## Documentation Index > Fetch the complete documentation index at: https://docs.baseten.co/llms.txt > Use this file to discover all available pages before exploring further. # Qwen3 Reranker > Qwen3 Reranker recipes: 3 variants (0.6B, 4B, 8B), Dense architecture.
Reranking Cross-encoder
## Setup Install the Baseten CLI and sign in, then install the Python `requests` library. **Install and sign in to Baseten** ```bash Terminal theme={"system"} brew tap basetenlabs/baseten brew trust --formula basetenlabs/baseten/baseten brew install baseten ``` Download and extract the Windows archive for the latest Baseten CLI release: ```powershell Terminal theme={"system"} $release = Invoke-RestMethod https://api.github.com/repos/basetenlabs/baseten-cli/releases/latest $asset = $release.assets | Where-Object { $_.name -like "*_windows_amd64.zip" } Invoke-WebRequest $asset.browser_download_url -OutFile baseten.zip Expand-Archive -Force baseten.zip . ``` Move `baseten.exe` to a directory on your `PATH`. For other platforms or a specific version, see the [Baseten CLI install reference](/reference/cli/baseten/overview#install). ```sh theme={"system"} baseten auth login ``` **Install requests** ```sh theme={"system"} uv pip install requests ``` To install without Homebrew, see the [Baseten CLI install reference](/reference/cli/baseten/overview#install). Pick the model you want to deploy. Each tab is a self-contained recipe. [Qwen/Qwen3-Reranker-0.6B](https://huggingface.co/Qwen/Qwen3-Reranker-0.6B) is a 0.6B-parameter dense model. This preset serves Qwen3 Reranker 0.6B on a single L4 through [Baseten Embeddings Inference](/engines/bei/overview) (BEI) with FP8 weights, optimized for reranking throughput on low-cost hardware. L4:4x16 TRT-LLM ## Write the config Create and move into the project directory: ```sh theme={"system"} mkdir qwen3-reranker-0.6b-throughput && cd qwen3-reranker-0.6b-throughput ``` Then create a file named `config.yaml` and paste the following: ```yaml config.yaml theme={"system"} # this file was autogenerated by `generate_templates.py` - please do change via template only model_metadata: example_model_input: inputs: - - Baseten is a fast inference provider - - Classify this separately. raw_scores: true truncate: true truncation_direction: Right model_name: "model:qwen3-reranker-0.6b preset:throughput" python_version: py311 resources: instance_type: L4:4x16 trt_llm: build: base_model: encoder checkpoint_repository: repo: michaelfeil/Qwen3-Reranker-0.6B-seq revision: main source: HF max_num_tokens: 32768 num_builder_gpus: 1 quantization_type: fp8 runtime: webserver_default_route: /predict ``` This config tells Baseten to build a BEI (Baseten Embeddings Inference) engine for Qwen3 Reranker 0.6B on a single L4, drawing FP8 weights from `michaelfeil/Qwen3-Reranker-0.6B-seq`, a sequence-classification conversion of the official checkpoint compatible with BEI's encoder build path. The deployment scores query-passage pairs on the `/predict` route with dynamic batching keeping throughput high. ## Key parameters [Baseten Embeddings Inference](/engines/bei/overview) (BEI) reads these fields from the `trt_llm` block. Each one shapes how the engine is built and served: | Parameter | Value | | - | - | | Quantization | `fp8` | | Base model type | `encoder` | ## Deploy Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it: ```sh Baseten CLI theme={"system"} baseten model push ``` ```sh Truss CLI theme={"system"} uvx truss push ``` You should see output similar to: ```output theme={"system"} Pushing model "qwen3-reranker-0.6b-throughput"... Uploading model... Uploaded model in 0s ✨ Model qwen3-reranker-0.6b-throughput was successfully pushed ✨ Model: qwen3-reranker-0.6b-throughput (abc1d2ef) Deployment: xyz123 Environment: production 🪵 View logs: deployment: baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123 environment: baseten model environment logs --model-id abc1d2ef --environment production (once deployed) app: https://app.baseten.co/models/abc1d2ef/logs/xyz123 🚀 Invoke your model: URL: https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict CLI: baseten model predict --model-id abc1d2ef ``` `baseten model push` prints your **model ID** (`abc1d2ef` in the example). The examples below use it wherever you see `{model_id}`, and read your API key from the `BASETEN_API_KEY` environment variable. ## Call the model Your deployment exposes a cross-encoder scoring endpoint at `/predict`. Now call your deployment to score candidates: ```python main.py theme={"system"} import os import requests response = requests.post( "https://model-{model_id}.api.baseten.co/environments/production/sync/predict", headers={"Authorization": f"Bearer {os.environ['BASETEN_API_KEY']}"}, json={ "query": "fast inference platform", "texts": [ "Baseten serves models on dedicated GPUs.", "The Eiffel Tower is in Paris.", "Cold-start latency matters for autoscaling.", ], }, ) for hit in response.json(): print(hit["score"], hit["text"]) ``` ```sh theme={"system"} curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/predict \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $BASETEN_API_KEY" \ -d '{ "query": "fast inference platform", "texts": [ "Baseten serves models on dedicated GPUs.", "The Eiffel Tower is in Paris.", "Cold-start latency matters for autoscaling." ] }' ``` For batch scoring at higher throughput, use the [Baseten Performance Client](https://www.baseten.co/blog/your-client-code-matters-10x-higher-embedding-throughput-with-python-and-rust/). [Qwen/Qwen3-Reranker-4B](https://huggingface.co/Qwen/Qwen3-Reranker-4B) is a 4B-parameter dense model. This preset serves Qwen3 Reranker 4B on a single H100 through [Baseten Embeddings Inference](/engines/bei/overview) (BEI) with FP8 weights, optimized for reranking throughput. H100 TRT-LLM ## Write the config Create and move into the project directory: ```sh theme={"system"} mkdir qwen3-reranker-4b-throughput && cd qwen3-reranker-4b-throughput ``` Then create a file named `config.yaml` and paste the following: ```yaml config.yaml theme={"system"} # this file was autogenerated by `generate_templates.py` - please do change via template only model_metadata: example_model_input: inputs: - - Baseten is a fast inference provider - - Classify this separately. raw_scores: true truncate: true truncation_direction: Right model_name: "model:qwen3-reranker-4b preset:throughput" python_version: py311 resources: instance_type: H100 trt_llm: build: base_model: encoder checkpoint_repository: repo: michaelfeil/Qwen3-Reranker-4B-seq revision: main source: HF max_num_tokens: 32768 num_builder_gpus: 1 quantization_type: fp8 runtime: webserver_default_route: /predict ``` This config tells Baseten to build a BEI (Baseten Embeddings Inference) engine for Qwen3 Reranker 4B on a single H100, drawing FP8 weights from `michaelfeil/Qwen3-Reranker-4B-seq`, a sequence-classification conversion of the official checkpoint compatible with BEI's encoder build path. The deployment scores query-passage pairs on the `/predict` route with dynamic batching keeping throughput high. ## Key parameters [Baseten Embeddings Inference](/engines/bei/overview) (BEI) reads these fields from the `trt_llm` block. Each one shapes how the engine is built and served: | Parameter | Value | | - | - | | Quantization | `fp8` | | Base model type | `encoder` | ## Deploy Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it: ```sh Baseten CLI theme={"system"} baseten model push ``` ```sh Truss CLI theme={"system"} uvx truss push ``` You should see output similar to: ```output theme={"system"} Pushing model "qwen3-reranker-4b-throughput"... Uploading model... Uploaded model in 0s ✨ Model qwen3-reranker-4b-throughput was successfully pushed ✨ Model: qwen3-reranker-4b-throughput (abc1d2ef) Deployment: xyz123 Environment: production 🪵 View logs: deployment: baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123 environment: baseten model environment logs --model-id abc1d2ef --environment production (once deployed) app: https://app.baseten.co/models/abc1d2ef/logs/xyz123 🚀 Invoke your model: URL: https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict CLI: baseten model predict --model-id abc1d2ef ``` `baseten model push` prints your **model ID** (`abc1d2ef` in the example). The examples below use it wherever you see `{model_id}`, and read your API key from the `BASETEN_API_KEY` environment variable. ## Call the model Your deployment exposes a cross-encoder scoring endpoint at `/predict`. Now call your deployment to score candidates: ```python main.py theme={"system"} import os import requests response = requests.post( "https://model-{model_id}.api.baseten.co/environments/production/sync/predict", headers={"Authorization": f"Bearer {os.environ['BASETEN_API_KEY']}"}, json={ "query": "fast inference platform", "texts": [ "Baseten serves models on dedicated GPUs.", "The Eiffel Tower is in Paris.", "Cold-start latency matters for autoscaling.", ], }, ) for hit in response.json(): print(hit["score"], hit["text"]) ``` ```sh theme={"system"} curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/predict \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $BASETEN_API_KEY" \ -d '{ "query": "fast inference platform", "texts": [ "Baseten serves models on dedicated GPUs.", "The Eiffel Tower is in Paris.", "Cold-start latency matters for autoscaling." ] }' ``` For batch scoring at higher throughput, use the [Baseten Performance Client](https://www.baseten.co/blog/your-client-code-matters-10x-higher-embedding-throughput-with-python-and-rust/). [Qwen/Qwen3-Reranker-8B](https://huggingface.co/Qwen/Qwen3-Reranker-8B) is an 8B-parameter dense model. H100 TRT-LLM ## Write the config Create and move into the project directory: ```sh theme={"system"} mkdir qwen3-reranker-8b && cd qwen3-reranker-8b ``` Then create a file named `config.yaml` and paste the following: ```yaml config.yaml theme={"system"} # this file was autogenerated by `generate_templates.py` - please do change via template only model_metadata: example_model_input: inputs: - - Baseten is a fast inference provider - - Classify this separately. raw_scores: true truncate: true truncation_direction: Right model_name: "model:qwen3-reranker-8b preset:throughput" python_version: py311 resources: instance_type: H100 trt_llm: build: base_model: encoder checkpoint_repository: repo: michaelfeil/Qwen3-Reranker-8B-seq revision: main source: HF max_num_tokens: 40960 num_builder_gpus: 1 quantization_type: fp8 runtime: webserver_default_route: /predict ``` ## Key parameters [Baseten Embeddings Inference](/engines/bei/overview) (BEI) reads these fields from the `trt_llm` block. Each one shapes how the engine is built and served: | Parameter | Value | | - | - | | Quantization | `fp8` | | Base model type | `encoder` | ## Deploy Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it: ```sh Baseten CLI theme={"system"} baseten model push ``` ```sh Truss CLI theme={"system"} uvx truss push ``` You should see output similar to: ```output theme={"system"} Pushing model "qwen3-reranker-8b"... Uploading model... Uploaded model in 0s ✨ Model qwen3-reranker-8b was successfully pushed ✨ Model: qwen3-reranker-8b (abc1d2ef) Deployment: xyz123 Environment: production 🪵 View logs: deployment: baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123 environment: baseten model environment logs --model-id abc1d2ef --environment production (once deployed) app: https://app.baseten.co/models/abc1d2ef/logs/xyz123 🚀 Invoke your model: URL: https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict CLI: baseten model predict --model-id abc1d2ef ``` `baseten model push` prints your **model ID** (`abc1d2ef` in the example). The examples below use it wherever you see `{model_id}`, and read your API key from the `BASETEN_API_KEY` environment variable. ## Call the model Your deployment exposes a cross-encoder scoring endpoint at `/predict`. Now call your deployment to score candidates: ```python main.py theme={"system"} import os import requests response = requests.post( "https://model-{model_id}.api.baseten.co/environments/production/sync/predict", headers={"Authorization": f"Bearer {os.environ['BASETEN_API_KEY']}"}, json={ "query": "fast inference platform", "texts": [ "Baseten serves models on dedicated GPUs.", "The Eiffel Tower is in Paris.", "Cold-start latency matters for autoscaling.", ], }, ) for hit in response.json(): print(hit["score"], hit["text"]) ``` ```sh theme={"system"} curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/predict \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $BASETEN_API_KEY" \ -d '{ "query": "fast inference platform", "texts": [ "Baseten serves models on dedicated GPUs.", "The Eiffel Tower is in Paris.", "Cold-start latency matters for autoscaling." ] }' ``` For batch scoring at higher throughput, use the [Baseten Performance Client](https://www.baseten.co/blog/your-client-code-matters-10x-higher-embedding-throughput-with-python-and-rust/). ## Next steps Endpoint anatomy, authentication, and sync versus async inference Scale replicas with traffic, including scale to zero