BEI configuration

This reference covers all configuration options for BEI and BEI-Bert deployments. All settings use the trt_llm section in config.yaml.

Configuration structure

trt_llm:
  inference_stack: v1  # Always v1 for BEI
  build:
    base_model: encoder | encoder_bert
    checkpoint_repository: {...}
    max_num_tokens: 16384
    quantization_type: no_quant | fp8 | fp4 | fp4_mlp_only
    quantization_config: {...}
    plugin_configuration: {...}
  runtime:
    webserver_default_route: /v1/embeddings | /v1/systemone | /rerank | /predict | /predict_tokens

Build configuration

Fields are tagged Required, Optional, or Computed. Computed fields are set by the engine; do not configure them manually. The build section configures model compilation and optimization settings.

string

required

Required. The base model architecture determines which BEI variant to use.Options:

encoder: BEI - for causal embedding models (Llama, Mistral, Qwen, Gemma)
encoder_bert: BEI-Bert - for BERT-based models (BERT, RoBERTa, Jina, Nomic)

build:
  base_model: encoder

object

required

Required. Specifies where to find the model checkpoint. Repository must follow the standard HuggingFace structure.Source options:

HF: Hugging Face Hub (default)
GCS: Google Cloud Storage
S3: AWS S3
AZURE: Azure Blob Storage
REMOTE_URL: HTTP URL to tar.gz file
BASETEN_TRAINING: Baseten Training checkpoints

For training checkpoint deployment, see Deploy with optimized inference engines. For cloud storage sources (GCS, S3, Azure), see Deploy from cloud storage.

checkpoint_repository:
  source: HF
  repo: "BAAI/bge-large-en-v1.5"
  revision: main
  runtime_secret_name: hf_access_token  # Optional, for private repos

checkpoint_repository is the weight source for BEI models, and Baseten mirrors it to the Baseten Delivery Network automatically for fast cold starts. Don’t add a top-level weights: section to a BEI config: checkpoint_repository already handles weight loading, so using weights: directly is discouraged.

number

default:"16384 (BEI) / 8192 (BEI-Bert)"

Optional. Maximum number of tokens that can be processed in a single batch. BEI defaults to 16384; BEI-Bert defaults to 8192. BEI and BEI-Bert run without chunked-prefill for performance reasons. This limits the effective context length to the max_position_embeddings value.Range: 65 to 1048576 (gt=64, le=1048576 in schema). Use higher values for long context models. Most models use 16384 as default.

build:
  max_num_tokens: 16384

number

Computed for embedding models. BEI sets it automatically from max_position_embeddings in the model’s config.json and truncates if context length is exceeded; leave it unset for embedding models. SequenceClassification models respect an explicit value.

string

default:"no_quant"

Optional. Specifies the quantization format for model weights. FP8 quantization maintains accuracy within 1% of FP16 for embedding models.Options for BEI:

no_quant: FP16/BF16 precision
fp8: FP8 weights + 16-bit KV cache
fp4: FP4 weights + 16-bit KV cache (B200 only)
fp4_mlp_only: FP4 MLP weights only (B200 only)

Options for BEI-Bert:

no_quant: FP16 precision (only option)

For detailed quantization guidance, see Quantization guide.

build:
  quantization_type: fp8

object

Optional. Configuration for post-training quantization calibration.Fields:

calib_size: Size of calibration dataset (64-16384, multiple of 64)
calib_dataset: HuggingFace dataset for calibration
calib_max_seq_length: Maximum sequence length for calibration

quantization_config:
  calib_size: 1024
  calib_dataset: "abisee/cnn_dailymail"
  calib_max_seq_length: 1536

object

Computed. BEI automatically configures optimal TensorRT-LLM plugin settings. Manual configuration is not required or supported.Automatic optimizations:

XQA kernels for maximum throughput
Dynamic batching for optimal utilization
Memory-efficient attention mechanisms
Hardware-specific optimizations
attention-dp by default for max thoughput

Note: Plugin configuration is managed automatically for BEI; it’s tunable only on Engine-Builder-LLM.

Runtime configuration

The runtime section configures serving behavior.

string

default:"/v1/embeddings"

Optional. The default API endpoint for the deployment.Options:

/v1/embeddings: OpenAI-compatible embeddings endpoint
/v1/systemone: Typed-decision endpoint for decision models such as Laya (BEI-Bert only)
/rerank: Reranking endpoint
/predict: Classification/prediction endpoint
/predict_tokens: Token classification endpoint for named entity recognition (BEI-Bert only)

BEI automatically detects embedding models and sets /v1/embeddings. Classification models default to /predict. Token classification models require /predict_tokens; see Named entity recognition. Typed-decision models use /v1/systemone; see Typed decisions.

runtime:
  webserver_default_route: /v1/embeddings

number

Computed. Available but has no effect for BEI embedding models, which do not use a KV cache. Only relevant for generative (decoder) models.

boolean

Computed. Available but has no effect for BEI embedding models. Only relevant for generative (decoder) models.

string

Computed. Available but has no effect for BEI embedding models. Only relevant for generative (decoder) models.

Hugging Face model repository structure

All model sources (S3, GCS, HuggingFace, or tar.gz) must follow the standard HuggingFace repository structure. Files must be in the root directory, similar to running:

git clone https://huggingface.co/michaelfeil/bge-small-en-v1.5

Choose a checkpoint

Choose an embedding or classification checkpoint for your task, rather than a chat checkpoint from the same model family. The checkpoint’s config.json identifies its architecture. Embedding checkpoints also need the pooling files described below; rerankers and classifiers use a classification head instead. For example, Qwen3-Embedding is an embedding checkpoint. A similarly named text-generation checkpoint doesn’t establish embedding support. Check the architecture and pooling files when evaluating a different model, including Qwen3.5. Adding pooling files doesn’t make an unsupported architecture compatible with the engine.

Model configuration

string[]

required

Architecture names in config.json. The engine infers the model architecture from this field. The supported architecture depends on the engine and task; the name alone doesn’t establish compatibility.

integer

Maximum context size declared by the checkpoint. Input beyond the configured limit is truncated.

object

Required for classification models. Maps each output label ID to a name in the JSON response. Include one entry per output of the final dense layer.

string

Checkpoint weight dtype. Use unquantized float16, bfloat16, or float32 weights. BEI performs quantization during the build; don’t supply a prequantized checkpoint.

Model weights

model.safetensors (preferred)

Or: model.safetensors.index.json + model-xx-of-yy.safetensors (sharded)
Note: Convert to safetensors if you encounter issues with other formats

Tokenizer files

tokenizer_config.json and tokenizer.json

Must be “FAST” tokenizers compatible with Rust
Typically cannot contain custom Python code, will be unread.

Embedding model files (sentence-transformers)

1_Pooling/config.json

Required for embedding models to define pooling strategy

modules.json

Required for embedding models
Shows available pooling layers and configurations

At build time, BEI reads pooling mode from modules.json and 1_Pooling/config.json and maps it to one of the modes below.

Flag in `1_Pooling/config.json`	Pooling mode	BEI	BEI-Bert
`pooling_mode_cls_token: true`	CLS token (first token)	✅	✅
`pooling_mode_mean_tokens: true`	Mean tokens	✅	✅
`pooling_mode_lasttoken: true`	Last token	✅	✅

If either file is missing on an embedding checkpoint, the build fails with a clear error naming the missing path. Sequence classification and reranking models skip pooling detection and use the classification head instead.

Pooling layer support

Engine	Classification Layers	Pooling Types	Notes
BEI	1 layer maximum	Last token, first token	Limited pooling options
BEI-Bert	Multiple layers or 1 layer	Last token, first token, mean, SPLADE pooling	Advanced pooling support

Throughput benchmarks

Measured against TEI and vLLM on the same hardware. Token throughput uses 500 tokens per request; request throughput uses 5 tokens per request. For the full methodology, see Run Qwen3 Embedding on NVIDIA Blackwell GPUs.

Framework	Precision	GPU	Max tokens/s	Max requests/s
TEI	FP16	H100	34,055	824.25
BEI-Bert	FP16	H100	36,520	841.05
vLLM	BF16	H100	36,625	155.23
BEI	BF16	H100	47,549	761.44
BEI	FP8	H100	77,107	855.96
BEI	FP8	B200	121,443	1,310.52

Quantization impact (8B dense models, H100)

Quantization	Speed improvement	Memory reduction	Accuracy impact
FP16/BF16 vLLM	Baseline	None	None
FP16/BF16 BEI	1.3x	None	None
FP8 BEI	2x	50%	~1%
FP4 BEI	3.5x	75%	1-2%

Hardware support

GPU	BEI	BEI-Bert	Recommended for
L4	Full	Full	Cost-effective deployments
A10G, A100	Full	Full	Legacy support
T4	No	Deprecated	Legacy support
RTX_6000	Full	Full	Maximum performance
H100	Full	Full	Maximum performance
H100	Full	Full	Maximum performance
B200	Full	Full	Maximum performance

Complete configuration examples

BEI with `FP8` quantization (embedding model)

model_name: BEI-BGE-Large-FP8
resources:
  accelerator: RTX_PRO_6000:2
  use_gpu: true
trt_llm:
  build:
    base_model: encoder
    checkpoint_repository:
      source: HF
      repo: "Qwen/Qwen3-Embedding-8B"
      revision: main
    max_num_tokens: 16384
    quantization_type: fp8
    quantization_config:
      calib_size: 1536
      calib_dataset: "abisee/cnn_dailymail"
      calib_max_seq_length: 1536
    # plugin_configuration is auto-configured for BEI models.
    # Encoder models disable paged_kv_cache and use_paged_context_fmha automatically.
  runtime:
    webserver_default_route: /v1/embeddings

BEI-Bert for small BERT model

model_name: BEI-Bert-MiniLM-L6
resources:
  accelerator: L4:4
  use_gpu: true
trt_llm:
  build:
    base_model: encoder_bert
    checkpoint_repository:
      source: HF
      repo: "sentence-transformers/all-MiniLM-L6-v2"
      revision: main
    max_num_tokens: 8192
    quantization_type: no_quant
    # plugin_configuration is auto-configured for BEI-Bert models.
    # paged_kv_cache and use_paged_context_fmha are disabled automatically.
  runtime:
    webserver_default_route: /v1/embeddings

BEI for reranking model

model_name: BEI-BGE-Reranker
resources:
  accelerator: H100:2
  use_gpu: true
trt_llm:
  build:
    base_model: encoder
    checkpoint_repository:
      source: HF
      repo: "BAAI/bge-reranker-large"
      revision: main
    max_num_tokens: 16384
    quantization_type: fp8
    quantization_config:
      calib_size: 1024
      calib_dataset: "abisee/cnn_dailymail"
      calib_max_seq_length: 2048
  runtime:
    webserver_default_route: /rerank

BEI-Bert for classification model

model_name: BEI-Bert-Language-Detection
resources:
  accelerator: L4:2
  use_gpu: true
trt_llm:
  build:
    base_model: encoder_bert
    checkpoint_repository:
      source: HF
      repo: "papluca/xlm-roberta-base-language-detection"
      revision: main
    max_num_tokens: 8192
    quantization_type: no_quant
  runtime:
    webserver_default_route: /predict

BEI-Bert for code embeddings (Jina)

model_name: BEI-Bert-Jina-Code
resources:
  accelerator: H100
  use_gpu: true
trt_llm:
  build:
    base_model: encoder_bert
    checkpoint_repository:
      source: HF
      repo: "jinaai/jina-embeddings-v2-base-code"
      revision: main
    max_num_tokens: 8192
    quantization_type: no_quant
  runtime:
    webserver_default_route: /v1/embeddings
    kv_cache_free_gpu_mem_fraction: 0.9
    batch_scheduler_policy: guaranteed_no_evict

BEI-Bert for bidirectional Qwen2 (long sequences)

model_name: BEI-Bert-GTE-Qwen-1.5B
resources:
  accelerator: L4:2
  use_gpu: true
trt_llm:
  build:
    base_model: encoder_bert
    checkpoint_repository:
      source: HF
      repo: "Alibaba-NLP/gte-Qwen2-1.5B-instruct"
      revision: main
    max_num_tokens: 8192
    quantization_type: no_quant
  runtime:
    webserver_default_route: /v1/embeddings
    kv_cache_free_gpu_mem_fraction: 0.85
    batch_scheduler_policy: guaranteed_no_evict

Common configuration errors

Warning: Briton logs: “Compling encoder with a kv-cache dtype is a alpha feature. This may fail.”

Cause: Using a KV quantization type (fp8_kv, fp4_kv) with an encoder model. Encoders do not use a KV cache, so these variants are alpha and may fail the build.
Fix: Use fp8 or no_quant instead.

Error: FP8 quantization is only supported on L4, H100, H200, B200

Cause: Using FP8 quantization on unsupported GPU.
Fix: Use H100 or newer GPU, or use no_quant.

Error: FP4 quantization is only supported on B200

Cause: Using FP4 quantization on unsupported GPU.
Fix: Use B200 GPU or FP8 quantization.

Reference

Inference API

Management API

Loops API

Training API

Frontier Gateway API

Sandbox runtime API

CLI reference

SDKs and clients

Configuration

Metrics

Integrations

BEI configuration

Configuration structure

Build configuration

Runtime configuration

Hugging Face model repository structure

Choose a checkpoint

Model configuration

Model weights

Tokenizer files

Embedding model files (sentence-transformers)

Pooling layer support

Throughput benchmarks

Quantization impact (8B dense models, H100)

Hardware support

Complete configuration examples

BEI with `FP8` quantization (embedding model)

BEI-Bert for small BERT model

BEI for reranking model

BEI-Bert for classification model

BEI-Bert for code embeddings (Jina)

BEI-Bert for bidirectional Qwen2 (long sequences)

Common configuration errors

​Configuration structure

​Build configuration

​Runtime configuration

​Hugging Face model repository structure

​Choose a checkpoint

​Model configuration

​Model weights

​Tokenizer files

​Embedding model files (sentence-transformers)

​Pooling layer support

​Throughput benchmarks

​Quantization impact (8B dense models, H100)

​Hardware support

​Complete configuration examples

​BEI with FP8 quantization (embedding model)

​BEI-Bert for small BERT model

​BEI for reranking model

​BEI-Bert for classification model

​BEI-Bert for code embeddings (Jina)

​BEI-Bert for bidirectional Qwen2 (long sequences)

​Common configuration errors

Configuration structure

Build configuration

Runtime configuration

Hugging Face model repository structure

Choose a checkpoint

Model configuration

Model weights

Tokenizer files

Embedding model files (sentence-transformers)

Pooling layer support

Throughput benchmarks

Quantization impact (8B dense models, H100)

Hardware support

Complete configuration examples

BEI with `FP8` quantization (embedding model)

BEI-Bert for small BERT model

BEI for reranking model

BEI-Bert for classification model

BEI-Bert for code embeddings (Jina)

BEI-Bert for bidirectional Qwen2 (long sequences)

Common configuration errors