zerank

Setup

Install the Baseten CLI and sign in, then install the Python requests library.

Install and sign in to Baseten

macOS or Linux
Windows

Terminal

brew tap basetenlabs/baseten
brew trust --formula basetenlabs/baseten/baseten
brew install baseten

Download and extract the Windows archive for the latest Baseten CLI release:

Terminal

$release = Invoke-RestMethod https://api.github.com/repos/basetenlabs/baseten-cli/releases/latest
$asset = $release.assets | Where-Object { $_.name -like "*_windows_amd64.zip" }
Invoke-WebRequest $asset.browser_download_url -OutFile baseten.zip
Expand-Archive -Force baseten.zip .

Move baseten.exe to a directory on your PATH.

For other platforms or a specific version, see the Baseten CLI install reference.

baseten auth login

Install requests

uv pip install requests

To install without Homebrew, see the Baseten CLI install reference. Pick the model you want to deploy. Each tab is a self-contained recipe.

1 Small
2

fred-baseten/zerank-1-small-seq is a 1.7B-parameter decoder model with up to 32K context.This preset builds a sequence-classification checkpoint with FP8 quantization on one L4.

Hardware

L4:4x16

Engine

TRT-LLM

Context

32K

Write the config

Create and move into the project directory:

mkdir zerank-1-small-reranker-throughput && cd zerank-1-small-reranker-throughput

Then create a file named config.yaml and paste the following:

config.yaml

# BEI requires the sequence-classification conversion; see README for prompt and score handling.
model_name: model:zerank-1-small-reranker preset:throughput
model_metadata:
  repo_id: zeroentropy/zerank-1-small-reranker
  example_model_input:
    inputs:
    - '<|im_start|>system

      What is 2+2?<|im_end|>

      <|im_start|>user

      4<|im_end|>

      <|im_start|>assistant

      '
    raw_scores: true
    truncate: false
python_version: py311
resources:
  instance_type: L4:4x16
trt_llm:
  build:
    base_model: encoder
    checkpoint_repository:
      repo: fred-baseten/zerank-1-small-seq
      revision: 65292825034a60e01fb33bf3c437ababc5ca5c26
      source: HF
    max_seq_len: 32768
    max_num_tokens: 32768
    num_builder_gpus: 1
    quantization_type: fp8
  runtime:
    webserver_default_route: /predict

Key parameters

Baseten Embeddings Inference (BEI) reads these fields from the trt_llm block. Each one shapes how the engine is built and served:

Parameter	Value
Max sequence length	`32768`
Quantization	`fp8`
Base model type	`encoder`

Deploy

Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:

baseten model push

uvx truss push

You should see output similar to:

Pushing model "zerank-1-small-reranker-throughput"...
Uploading model...
Uploaded model in 0s
✨ Model zerank-1-small-reranker-throughput was successfully pushed ✨

  Model:       zerank-1-small-reranker-throughput (abc1d2ef)
  Deployment:  xyz123
  Environment: production

🪵 View logs:
   deployment:   baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
   environment:  baseten model environment logs --model-id abc1d2ef --environment production  (once deployed)
   app:          https://app.baseten.co/models/abc1d2ef/logs/xyz123

🚀 Invoke your model:
   URL:  https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
   CLI:  baseten model predict --model-id abc1d2ef

baseten model push prints your model ID (abc1d2ef in the example). The examples below use it wherever you see {model_id}, and read your API key from the BASETEN_API_KEY environment variable.

Call the model

Your deployment exposes a cross-encoder scoring endpoint at /predict.Now call your deployment to score candidates:

Python
cURL

main.py

import os
import requests

response = requests.post(
    "https://model-{model_id}.api.baseten.co/environments/production/sync/predict",
    headers={"Authorization": f"Bearer {os.environ['BASETEN_API_KEY']}"},
    json={
        "query": "fast inference platform",
        "texts": [
            "Baseten serves models on dedicated GPUs.",
            "The Eiffel Tower is in Paris.",
            "Cold-start latency matters for autoscaling.",
        ],
    },
)

for hit in response.json():
    print(hit["score"], hit["text"])

curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/predict \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '{
    "query": "fast inference platform",
    "texts": [
      "Baseten serves models on dedicated GPUs.",
      "The Eiffel Tower is in Paris.",
      "Cold-start latency matters for autoscaling."
    ]
  }'

For batch scoring at higher throughput, use the Baseten Performance Client.The /predict route accepts inputs containing formatted prompts, not a query and texts object. Request raw_scores: true and use the pinned checkpoint’s tokenizer and scoring instructions: Zerank 1 Small checkpoint.

baseten-admin/zerank-2-reranker-seq is a 4B-parameter decoder model with up to 40K context.This preset builds Zerank 2’s public sequence-classification checkpoint with FP8 quantization on one H100.

Hardware

H100

Engine

TRT-LLM v1

Context

40K

Write the config

Create and move into the project directory:

mkdir zerank-2-reranker-throughput && cd zerank-2-reranker-throughput

Then create a file named config.yaml and paste the following:

config.yaml

# Registry copy of the config serving the ZeroEntropy lab listing zeroentropy/zerank-2-reranker.
# Values below are the deployed ones; only model_name (repo convention) and the pinned
# checkpoint revision (main == ac4b5f3c at the time of writing) differ in form, not effect.
model_name: model:zerank-2-reranker preset:throughput
model_metadata:
  repo_id: zeroentropy/zerank-2-reranker
  example_model_input:
    inputs:
    - - Baseten is a fast inference provider
    - - Classify this separately.
    raw_scores: true
    truncate: true
    truncation_direction: Right
python_version: py311
resources:
  instance_type: H100
  cpu: '1'
  memory: 10Gi
runtime:
  predict_concurrency: 1
  streaming_read_timeout: 60
trt_llm:
  inference_stack: v1
  build:
    base_model: encoder
    checkpoint_repository:
      repo: baseten-admin/zerank-2-reranker-seq
      revision: ac4b5f3c452cf22a917aa6b93b0a92c750db284a
      source: HF
    max_batch_size: 256
    max_beam_width: 1
    max_num_tokens: 40960
    num_builder_gpus: 1
    tensor_parallel_count: 1
    pipeline_parallel_count: 1
    sequence_parallel_count: 1
    strongly_typed: false
    quantization_type: fp8
    quantization_config:
      calib_size: 1024
      calib_dataset: abisee/cnn_dailymail
      calib_max_seq_length: 1536
    plugin_configuration:
      paged_kv_cache: false
      use_fp8_context_fmha: false
      use_paged_context_fmha: false
  runtime:
    webserver_default_route: /predict
    total_token_limit: 500000
    batch_scheduler_policy: guaranteed_no_evict
    enable_chunked_context: true
    kv_cache_free_gpu_mem_fraction: 0.9

Key parameters

Baseten Embeddings Inference (BEI) reads these fields from the trt_llm block. Each one shapes how the engine is built and served:

Parameter	Value
Chunked prefill	`enabled`
Quantization	`fp8`
Base model type	`encoder`
Inference stack	`v1`

Deploy

Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:

baseten model push

uvx truss push

You should see output similar to:

Pushing model "zerank-2-reranker-throughput"...
Uploading model...
Uploaded model in 0s
✨ Model zerank-2-reranker-throughput was successfully pushed ✨

  Model:       zerank-2-reranker-throughput (abc1d2ef)
  Deployment:  xyz123
  Environment: production

🪵 View logs:
   deployment:   baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
   environment:  baseten model environment logs --model-id abc1d2ef --environment production  (once deployed)
   app:          https://app.baseten.co/models/abc1d2ef/logs/xyz123

🚀 Invoke your model:
   URL:  https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
   CLI:  baseten model predict --model-id abc1d2ef

Call the model

Your deployment exposes a cross-encoder scoring endpoint at /predict.Now call your deployment to score candidates:

Python
cURL

main.py

import os
import requests

response = requests.post(
    "https://model-{model_id}.api.baseten.co/environments/production/sync/predict",
    headers={"Authorization": f"Bearer {os.environ['BASETEN_API_KEY']}"},
    json={
        "query": "fast inference platform",
        "texts": [
            "Baseten serves models on dedicated GPUs.",
            "The Eiffel Tower is in Paris.",
            "Cold-start latency matters for autoscaling.",
        ],
    },
)

for hit in response.json():
    print(hit["score"], hit["text"])

curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/predict \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '{
    "query": "fast inference platform",
    "texts": [
      "Baseten serves models on dedicated GPUs.",
      "The Eiffel Tower is in Paris.",
      "Cold-start latency matters for autoscaling."
    ]
  }'

For batch scoring at higher throughput, use the Baseten Performance Client.The /predict route accepts inputs containing chat-formatted strings. Put the query in a system turn, the document in a user turn, and end with the assistant prefix, as shown in the examples. The server doesn’t apply this template. With raw_scores: true, each result is a relevance logit; apply a sigmoid for a score between 0 and 1. See the converted checkpoint instructions.

Examples

Models

Engines

Custom Docker servers

Custom Python models

Chains

Baseten for Model Labs

Setup

Hardware

Engine

Context

Write the config

Key parameters

Deploy

Call the model

Hardware

Engine

Context

Write the config

Key parameters

Deploy

Call the model

Next steps

Call your model

Autoscaling

​Setup

Hardware

Engine

Context

​Write the config

​Key parameters

​Deploy

​Call the model

Hardware

Engine

Context

​Write the config

​Key parameters

​Deploy

​Call the model

​Next steps

Call your model

Autoscaling

Setup

Write the config

Key parameters

Deploy

Call the model

Write the config

Key parameters

Deploy

Call the model

Next steps