To install without Homebrew, see the Baseten CLI install reference.Pick the model you want to deploy. Each tab is a self-contained recipe.
0.6B
4B
8B
Qwen/Qwen3-Reranker-0.6B is a 0.6B-parameter dense model.This preset serves Qwen3 Reranker 0.6B on a single L4 through Baseten Embeddings Inference (BEI) with FP8 weights, optimized for reranking throughput on low-cost hardware.
Hardware
L4:4x16
Engine
TRT-LLM
Write the config
Create and move into the project directory:
mkdir qwen3-reranker-0.6b-throughput && cd qwen3-reranker-0.6b-throughput
Then create a file named config.yaml and paste the following:
config.yaml
# this file was autogenerated by `generate_templates.py` - please do change via template onlymodel_metadata: example_model_input: inputs: - - "<|im_start|>system\nJudge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be \"yes\" or \"no\".<|im_end|>\n<|im_start|>user\n<Instruct>: Given a web search query, retrieve relevant passages that answer the query\n<Query>: What is Baseten?\n<Document>: Baseten is an AI inference platform for deploying and serving models.<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n" - - "<|im_start|>system\nJudge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be \"yes\" or \"no\".<|im_end|>\n<|im_start|>user\n<Instruct>: Given a web search query, retrieve relevant passages that answer the query\n<Query>: What is Baseten?\n<Document>: The capital of France is Paris.<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n" raw_scores: true truncate: true truncation_direction: Rightmodel_name: "model:qwen3-reranker-0.6b preset:throughput"python_version: py311resources: instance_type: L4:4x16trt_llm: build: base_model: encoder checkpoint_repository: repo: michaelfeil/Qwen3-Reranker-0.6B-seq revision: main source: HF max_num_tokens: 32768 num_builder_gpus: 1 quantization_type: fp8 runtime: webserver_default_route: /predict
This config tells Baseten to build a BEI (Baseten Embeddings Inference) engine for Qwen3 Reranker 0.6B on a single L4, drawing FP8 weights from michaelfeil/Qwen3-Reranker-0.6B-seq, a sequence-classification conversion of the official checkpoint compatible with BEI’s encoder build path. The deployment scores query-passage pairs on the /predict route with dynamic batching keeping throughput high.
Key parameters
Baseten Embeddings Inference (BEI) reads these fields from the trt_llm block. Each one shapes how the engine is built and served:
Parameter
Value
Quantization
fp8
Base model type
encoder
Deploy
Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:
baseten model push
uvx truss push
You should see output similar to:
Pushing model "qwen3-reranker-0.6b-throughput"...Uploading model...Uploaded model in 0s✨ Model qwen3-reranker-0.6b-throughput was successfully pushed ✨ Model: qwen3-reranker-0.6b-throughput (abc1d2ef) Deployment: xyz123 Environment: production🪵 View logs: deployment: baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123 environment: baseten model environment logs --model-id abc1d2ef --environment production (once deployed) app: https://app.baseten.co/models/abc1d2ef/logs/xyz123🚀 Invoke your model: URL: https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict CLI: baseten model predict --model-id abc1d2ef
baseten model push prints your model ID (abc1d2ef in the example). The examples below use it wherever you see {model_id}, and read your API key from the BASETEN_API_KEY environment variable.
Call the model
Your deployment exposes a cross-encoder scoring endpoint at /predict.Now call your deployment to score candidates:
Python
cURL
main.py
import osimport requestsresponse = requests.post( "https://model-{model_id}.api.baseten.co/environments/production/sync/predict", headers={"Authorization": f"Bearer {os.environ['BASETEN_API_KEY']}"}, json={ "query": "fast inference platform", "texts": [ "Baseten serves models on dedicated GPUs.", "The Eiffel Tower is in Paris.", "Cold-start latency matters for autoscaling.", ], },)for hit in response.json(): print(hit["score"], hit["text"])
curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/predict \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $BASETEN_API_KEY" \ -d '{ "query": "fast inference platform", "texts": [ "Baseten serves models on dedicated GPUs.", "The Eiffel Tower is in Paris.", "Cold-start latency matters for autoscaling." ] }'
Qwen/Qwen3-Reranker-4B is a 4B-parameter dense model.This preset serves Qwen3 Reranker 4B on a single H100 through Baseten Embeddings Inference (BEI) with FP8 weights, optimized for reranking throughput.
Hardware
H100
Engine
TRT-LLM
Write the config
Create and move into the project directory:
mkdir qwen3-reranker-4b-throughput && cd qwen3-reranker-4b-throughput
Then create a file named config.yaml and paste the following:
config.yaml
# this file was autogenerated by `generate_templates.py` - please do change via template onlymodel_metadata: example_model_input: inputs: - - "<|im_start|>system\nJudge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be \"yes\" or \"no\".<|im_end|>\n<|im_start|>user\n<Instruct>: Given a web search query, retrieve relevant passages that answer the query\n<Query>: What is Baseten?\n<Document>: Baseten is an AI inference platform for deploying and serving models.<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n" - - "<|im_start|>system\nJudge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be \"yes\" or \"no\".<|im_end|>\n<|im_start|>user\n<Instruct>: Given a web search query, retrieve relevant passages that answer the query\n<Query>: What is Baseten?\n<Document>: The capital of France is Paris.<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n" raw_scores: true truncate: true truncation_direction: Rightmodel_name: "model:qwen3-reranker-4b preset:throughput"python_version: py311resources: instance_type: H100trt_llm: build: base_model: encoder checkpoint_repository: repo: michaelfeil/Qwen3-Reranker-4B-seq revision: main source: HF max_num_tokens: 32768 num_builder_gpus: 1 quantization_type: fp8 runtime: webserver_default_route: /predict
This config tells Baseten to build a BEI (Baseten Embeddings Inference) engine for Qwen3 Reranker 4B on a single H100, drawing FP8 weights from michaelfeil/Qwen3-Reranker-4B-seq, a sequence-classification conversion of the official checkpoint compatible with BEI’s encoder build path. The deployment scores query-passage pairs on the /predict route with dynamic batching keeping throughput high.
Key parameters
Baseten Embeddings Inference (BEI) reads these fields from the trt_llm block. Each one shapes how the engine is built and served:
Parameter
Value
Quantization
fp8
Base model type
encoder
Deploy
Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:
baseten model push
uvx truss push
You should see output similar to:
Pushing model "qwen3-reranker-4b-throughput"...Uploading model...Uploaded model in 0s✨ Model qwen3-reranker-4b-throughput was successfully pushed ✨ Model: qwen3-reranker-4b-throughput (abc1d2ef) Deployment: xyz123 Environment: production🪵 View logs: deployment: baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123 environment: baseten model environment logs --model-id abc1d2ef --environment production (once deployed) app: https://app.baseten.co/models/abc1d2ef/logs/xyz123🚀 Invoke your model: URL: https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict CLI: baseten model predict --model-id abc1d2ef
baseten model push prints your model ID (abc1d2ef in the example). The examples below use it wherever you see {model_id}, and read your API key from the BASETEN_API_KEY environment variable.
Call the model
Your deployment exposes a cross-encoder scoring endpoint at /predict.Now call your deployment to score candidates:
Python
cURL
main.py
import osimport requestsresponse = requests.post( "https://model-{model_id}.api.baseten.co/environments/production/sync/predict", headers={"Authorization": f"Bearer {os.environ['BASETEN_API_KEY']}"}, json={ "query": "fast inference platform", "texts": [ "Baseten serves models on dedicated GPUs.", "The Eiffel Tower is in Paris.", "Cold-start latency matters for autoscaling.", ], },)for hit in response.json(): print(hit["score"], hit["text"])
curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/predict \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $BASETEN_API_KEY" \ -d '{ "query": "fast inference platform", "texts": [ "Baseten serves models on dedicated GPUs.", "The Eiffel Tower is in Paris.", "Cold-start latency matters for autoscaling." ] }'
Then create a file named config.yaml and paste the following:
config.yaml
# this file was autogenerated by `generate_templates.py` - please do change via template onlymodel_metadata: example_model_input: inputs: - - "<|im_start|>system\nJudge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be \"yes\" or \"no\".<|im_end|>\n<|im_start|>user\n<Instruct>: Given a web search query, retrieve relevant passages that answer the query\n<Query>: What is Baseten?\n<Document>: Baseten is an AI inference platform for deploying and serving models.<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n" - - "<|im_start|>system\nJudge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be \"yes\" or \"no\".<|im_end|>\n<|im_start|>user\n<Instruct>: Given a web search query, retrieve relevant passages that answer the query\n<Query>: What is Baseten?\n<Document>: The capital of France is Paris.<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n" raw_scores: true truncate: true truncation_direction: Rightmodel_name: "model:qwen3-reranker-8b preset:throughput"python_version: py311resources: instance_type: H100trt_llm: build: base_model: encoder checkpoint_repository: repo: michaelfeil/Qwen3-Reranker-8B-seq revision: main source: HF max_num_tokens: 40960 num_builder_gpus: 1 quantization_type: fp8 runtime: webserver_default_route: /predict
Key parameters
Baseten Embeddings Inference (BEI) reads these fields from the trt_llm block. Each one shapes how the engine is built and served:
Parameter
Value
Quantization
fp8
Base model type
encoder
Deploy
Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:
baseten model push
uvx truss push
You should see output similar to:
Pushing model "qwen3-reranker-8b"...Uploading model...Uploaded model in 0s✨ Model qwen3-reranker-8b was successfully pushed ✨ Model: qwen3-reranker-8b (abc1d2ef) Deployment: xyz123 Environment: production🪵 View logs: deployment: baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123 environment: baseten model environment logs --model-id abc1d2ef --environment production (once deployed) app: https://app.baseten.co/models/abc1d2ef/logs/xyz123🚀 Invoke your model: URL: https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict CLI: baseten model predict --model-id abc1d2ef
baseten model push prints your model ID (abc1d2ef in the example). The examples below use it wherever you see {model_id}, and read your API key from the BASETEN_API_KEY environment variable.
Call the model
Your deployment exposes a cross-encoder scoring endpoint at /predict.Now call your deployment to score candidates:
Python
cURL
main.py
import osimport requestsresponse = requests.post( "https://model-{model_id}.api.baseten.co/environments/production/sync/predict", headers={"Authorization": f"Bearer {os.environ['BASETEN_API_KEY']}"}, json={ "query": "fast inference platform", "texts": [ "Baseten serves models on dedicated GPUs.", "The Eiffel Tower is in Paris.", "Cold-start latency matters for autoscaling.", ], },)for hit in response.json(): print(hit["score"], hit["text"])
curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/predict \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $BASETEN_API_KEY" \ -d '{ "query": "fast inference platform", "texts": [ "Baseten serves models on dedicated GPUs.", "The Eiffel Tower is in Paris.", "Cold-start latency matters for autoscaling." ] }'