> ## Documentation Index
> Fetch the complete documentation index at: https://docs.baseten.co/llms.txt
> Use this file to discover all available pages before exploring further.

# zerank

> zerank recipes: 2 variants (1 Small, 2), decoder architecture.

[Reranking](https://docs.baseten.co/examples/models/capabilities/reranking)
[Cross-encoder](https://docs.baseten.co/examples/models/capabilities/cross-encoder)

## Setup

Install the Baseten CLI and sign in, then install the Python `requests` library.

**Install and sign in to Baseten**

**macOS or Linux**

```bash Terminal theme={"system"}
brew tap basetenlabs/baseten
brew trust --formula basetenlabs/baseten/baseten
brew install baseten
```

**Windows**

Download and extract the Windows archive for the latest Baseten CLI release:

```powershell Terminal theme={"system"}
$release = Invoke-RestMethod https://api.github.com/repos/basetenlabs/baseten-cli/releases/latest
$asset = $release.assets | Where-Object { $_.name -like "*_windows_amd64.zip" }
Invoke-WebRequest $asset.browser_download_url -OutFile baseten.zip
Expand-Archive -Force baseten.zip .
```

Move `baseten.exe` to a directory on your `PATH`.

For other platforms or a specific version, see the [Baseten CLI install reference](https://docs.baseten.co/reference/cli/baseten/overview.md#install).

```sh theme={"system"}
baseten auth login
```

**Install requests**

```sh theme={"system"}
uv pip install requests
```

To install without Homebrew, see the [Baseten CLI install reference](https://docs.baseten.co/reference/cli/baseten/overview.md#install).

Pick the model you want to deploy. Each tab is a self-contained recipe.

**1 Small**

[fred-baseten/zerank-1-small-seq](https://huggingface.co/fred-baseten/zerank-1-small-seq) is a 1.7B-parameter decoder model with up to 32K context.

This preset builds a sequence-classification checkpoint with FP8 quantization on one L4.

> **Hardware**
>
> L4:4x16

> **Engine**
>
> TRT-LLM

> **Context**
>
> 32K

## Write the config

Create and move into the project directory:

```sh theme={"system"}
mkdir zerank-1-small-reranker-throughput && cd zerank-1-small-reranker-throughput
```

Then create a file named `config.yaml` and paste the following:

```yaml config.yaml theme={"system"}
# BEI requires the sequence-classification conversion; see README for prompt and score handling.
model_name: model:zerank-1-small-reranker preset:throughput
model_metadata:
  repo_id: zeroentropy/zerank-1-small-reranker
  example_model_input:
    inputs:
    - '<|im_start|>system

      What is 2+2?<|im_end|>

      <|im_start|>user

      4<|im_end|>

      <|im_start|>assistant

      '
    raw_scores: true
    truncate: false
python_version: py311
resources:
  instance_type: L4:4x16
trt_llm:
  build:
    base_model: encoder
    checkpoint_repository:
      repo: fred-baseten/zerank-1-small-seq
      revision: 65292825034a60e01fb33bf3c437ababc5ca5c26
      source: HF
    max_seq_len: 32768
    max_num_tokens: 32768
    num_builder_gpus: 1
    quantization_type: fp8
  runtime:
    webserver_default_route: /predict
```

## Key parameters

[Baseten Embeddings Inference](https://docs.baseten.co/engines/bei/overview.md) (BEI) reads these fields from the `trt_llm` block. Each one shapes how the engine is built and served:

| Parameter | Value |
| - | - |
| Max sequence length | `32768` |
| Quantization | `fp8` |
| Base model type | `encoder` |

## Deploy

Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:

```sh Baseten CLI theme={"system"}
baseten model push
```

```sh Truss CLI theme={"system"}
uvx truss push
```

You should see output similar to:

```output theme={"system"}
Pushing model "zerank-1-small-reranker-throughput"...
Uploading model...
Uploaded model in 0s
✨ Model zerank-1-small-reranker-throughput was successfully pushed ✨

  Model:       zerank-1-small-reranker-throughput (abc1d2ef)
  Deployment:  xyz123
  Environment: production

🪵 View logs:
   deployment:   baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
   environment:  baseten model environment logs --model-id abc1d2ef --environment production  (once deployed)
   app:          https://app.baseten.co/models/abc1d2ef/logs/xyz123

🚀 Invoke your model:
   URL:  https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
   CLI:  baseten model predict --model-id abc1d2ef
```

`baseten model push` prints your **model ID** (`abc1d2ef` in the example). The examples below use it wherever you see `{model_id}`, and read your API key from the `BASETEN_API_KEY` environment variable.

## Call the model

Your deployment exposes a cross-encoder scoring endpoint at `/predict`.

Now call your deployment to score candidates:

**Python**

```python main.py theme={"system"}
import os
import requests

response = requests.post(
    "https://model-{model_id}.api.baseten.co/environments/production/sync/predict",
    headers={"Authorization": f"Bearer {os.environ['BASETEN_API_KEY']}"},
    json={
        "query": "fast inference platform",
        "texts": [
            "Baseten serves models on dedicated GPUs.",
            "The Eiffel Tower is in Paris.",
            "Cold-start latency matters for autoscaling.",
        ],
    },
)

for hit in response.json():
    print(hit["score"], hit["text"])
```

**cURL**

```sh theme={"system"}
curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/predict \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '{
    "query": "fast inference platform",
    "texts": [
      "Baseten serves models on dedicated GPUs.",
      "The Eiffel Tower is in Paris.",
      "Cold-start latency matters for autoscaling."
    ]
  }'
```

For batch scoring at higher throughput, use the [Baseten Performance Client](https://www.baseten.co/blog/your-client-code-matters-10x-higher-embedding-throughput-with-python-and-rust/).

The `/predict` route accepts `inputs` containing formatted prompts, not a `query` and `texts` object. Request `raw_scores: true` and use the pinned checkpoint's tokenizer and scoring instructions: [Zerank 1 Small checkpoint](https://huggingface.co/fred-baseten/zerank-1-small-seq).

**2**

[baseten-admin/zerank-2-reranker-seq](https://huggingface.co/baseten-admin/zerank-2-reranker-seq) is a 4B-parameter decoder model with up to 40K context.

This preset builds Zerank 2's public sequence-classification checkpoint with FP8 quantization on one H100.

> **Hardware**
>
> H100

> **Engine**
>
> TRT-LLM v1

> **Context**
>
> 40K

## Write the config

Create and move into the project directory:

```sh theme={"system"}
mkdir zerank-2-reranker-throughput && cd zerank-2-reranker-throughput
```

Then create a file named `config.yaml` and paste the following:

```yaml config.yaml theme={"system"}
# Registry copy of the config serving the ZeroEntropy lab listing zeroentropy/zerank-2-reranker.
# Values below are the deployed ones; only model_name (repo convention) and the pinned
# checkpoint revision (main == ac4b5f3c at the time of writing) differ in form, not effect.
model_name: model:zerank-2-reranker preset:throughput
model_metadata:
  repo_id: zeroentropy/zerank-2-reranker
  example_model_input:
    inputs:
    - - Baseten is a fast inference provider
    - - Classify this separately.
    raw_scores: true
    truncate: true
    truncation_direction: Right
python_version: py311
resources:
  instance_type: H100
  cpu: '1'
  memory: 10Gi
runtime:
  predict_concurrency: 1
  streaming_read_timeout: 60
trt_llm:
  inference_stack: v1
  build:
    base_model: encoder
    checkpoint_repository:
      repo: baseten-admin/zerank-2-reranker-seq
      revision: ac4b5f3c452cf22a917aa6b93b0a92c750db284a
      source: HF
    max_batch_size: 256
    max_beam_width: 1
    max_num_tokens: 40960
    num_builder_gpus: 1
    tensor_parallel_count: 1
    pipeline_parallel_count: 1
    sequence_parallel_count: 1
    strongly_typed: false
    quantization_type: fp8
    quantization_config:
      calib_size: 1024
      calib_dataset: abisee/cnn_dailymail
      calib_max_seq_length: 1536
    plugin_configuration:
      paged_kv_cache: false
      use_fp8_context_fmha: false
      use_paged_context_fmha: false
  runtime:
    webserver_default_route: /predict
    total_token_limit: 500000
    batch_scheduler_policy: guaranteed_no_evict
    enable_chunked_context: true
    kv_cache_free_gpu_mem_fraction: 0.9
```

## Key parameters

[Baseten Embeddings Inference](https://docs.baseten.co/engines/bei/overview.md) (BEI) reads these fields from the `trt_llm` block. Each one shapes how the engine is built and served:

| Parameter | Value |
| - | - |
| Chunked prefill | `enabled` |
| Quantization | `fp8` |
| Base model type | `encoder` |
| Inference stack | `v1` |

## Deploy

Push the config to Baseten with the Baseten CLI, or with the Truss CLI if you prefer it:

```sh Baseten CLI theme={"system"}
baseten model push
```

```sh Truss CLI theme={"system"}
uvx truss push
```

You should see output similar to:

```output theme={"system"}
Pushing model "zerank-2-reranker-throughput"...
Uploading model...
Uploaded model in 0s
✨ Model zerank-2-reranker-throughput was successfully pushed ✨

  Model:       zerank-2-reranker-throughput (abc1d2ef)
  Deployment:  xyz123
  Environment: production

🪵 View logs:
   deployment:   baseten model deployment logs --model-id abc1d2ef --deployment-id xyz123
   environment:  baseten model environment logs --model-id abc1d2ef --environment production  (once deployed)
   app:          https://app.baseten.co/models/abc1d2ef/logs/xyz123

🚀 Invoke your model:
   URL:  https://model-abc1d2ef.api.baseten.co/deployment/xyz123/predict
   CLI:  baseten model predict --model-id abc1d2ef
```

`baseten model push` prints your **model ID** (`abc1d2ef` in the example). The examples below use it wherever you see `{model_id}`, and read your API key from the `BASETEN_API_KEY` environment variable.

## Call the model

Your deployment exposes a cross-encoder scoring endpoint at `/predict`.

Now call your deployment to score candidates:

**Python**

```python main.py theme={"system"}
import os
import requests

response = requests.post(
    "https://model-{model_id}.api.baseten.co/environments/production/sync/predict",
    headers={"Authorization": f"Bearer {os.environ['BASETEN_API_KEY']}"},
    json={
        "query": "fast inference platform",
        "texts": [
            "Baseten serves models on dedicated GPUs.",
            "The Eiffel Tower is in Paris.",
            "Cold-start latency matters for autoscaling.",
        ],
    },
)

for hit in response.json():
    print(hit["score"], hit["text"])
```

**cURL**

```sh theme={"system"}
curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/predict \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '{
    "query": "fast inference platform",
    "texts": [
      "Baseten serves models on dedicated GPUs.",
      "The Eiffel Tower is in Paris.",
      "Cold-start latency matters for autoscaling."
    ]
  }'
```

For batch scoring at higher throughput, use the [Baseten Performance Client](https://www.baseten.co/blog/your-client-code-matters-10x-higher-embedding-throughput-with-python-and-rust/).

The `/predict` route accepts `inputs` containing chat-formatted strings. Put the query in a system turn, the document in a user turn, and end with the assistant prefix, as shown in the examples. The server doesn't apply this template. With `raw_scores: true`, each result is a relevance logit; apply a sigmoid for a score between 0 and 1. See the [converted checkpoint instructions](https://huggingface.co/baseten-admin/zerank-2-reranker-seq/blob/ac4b5f3c452cf22a917aa6b93b0a92c750db284a/README.md).

## Next steps

> **[Call your model](https://docs.baseten.co/inference/calling-your-model.md)**
>
> Endpoint anatomy, authentication, and sync versus async inference

> **[Autoscaling](https://docs.baseten.co/deployment/autoscaling/overview.md)**
>
> Scale replicas with traffic, including scale to zero