Skip to main content
BEI-Bert is a variant of Baseten Embeddings Inference (BEI) for BERT-family architectures. It runs at FP16 or BF16, optimizes cold-start latency, and supports bidirectional attention for sub-4B-parameter encoders.
Bidirectional attention means each token in the input can attend to every other token, in both directions. BERT-family encoders use this pattern, which generally produces better embeddings because each token sees the full context. Causal models like GPT use the opposite pattern: each token attends only to earlier tokens, never to later ones. Some Qwen and Llama checkpoints (the *Bidirectional model variants listed below) are causal LLMs adapted to run in bidirectional mode specifically for embedding use.

​
BEI vs BEI-Bert

Both variants run on the same engine binary. Pick the variant that matches your base architecture.

​
When to use BEI-Bert

Choose BEI-Bert when any of these apply:
  • BERT-family base architecture: BertModel, RobertaModel, ModernBertModel, XLMRobertaModel, or a *Bidirectional adapted checkpoint.
  • Cold-start matters: first-request latency is critical for your traffic shape.
  • Small to medium models: under 4B parameters where FP8/FP4 quantization isn’t needed.
  • 16-bit precision: workloads where FP16 accuracy is preferred over quantized throughput.
  • Token-level classification: NER and other /predict_tokens endpoints run on BEI-Bert only.
For models over 4B parameters, causal embedders, or workloads that need FP8/FP4 quantization, use BEI. See the BEI overview.

​
Supported model families

BEI-Bert runs the following base architectures: BertModel, RobertaModel, ModernBertModel, XLMRobertaModel, Gemma3Bidirectional, Qwen2Bidirectional, Qwen3Bidirectional, LLama3Bidirectional.

​
Additional text models

BEI-Bert supports the following text workloads. Configure deployments with base_model: encoder_bert in the trt_llm section. Architecture support doesn’t establish task accuracy for a base language-model checkpoint. Use a trained embedding or classification checkpoint and validate it on your task. Qwen3.5 support here covers the MoE text backend. See request examples for these tasks.

​
Sentence-transformers

The most common BERT-based embedding models, optimized for semantic similarity.
  • sentence-transformers/all-MiniLM-L6-v2 (384D, 22M params)
  • sentence-transformers/all-mpnet-base-v2 (768D, 110M params)
  • sentence-transformers/multi-qa-mpnet-base-dot-v1 (768D, 110M params)

​
Jina AI

Jina’s BERT-based models for general and code-specific domains.
  • jinaai/jina-embeddings-v2-base-en (512D, 137M params)
  • jinaai/jina-embeddings-v2-base-code (512D, 137M params)
  • jinaai/jina-embeddings-v2-base-es (512D, 137M params)

​
Nomic AI

Nomic’s models with specialized training for text and code.
  • nomic-ai/nomic-embed-text-v1.5 (768D, 137M params)
  • nomic-ai/nomic-embed-code-v1.5 (768D, 137M params)

​
Alibaba GTE and Qwen (bidirectional)

Multilingual models with instruction-tuning and long-context support.
  • Alibaba-NLP/gte-Qwen2-7B-instruct (top-ranked multilingual)
  • Alibaba-NLP/gte-Qwen2-1.5B-instruct (cost-effective alternative)
  • intfloat/multilingual-e5-large-instruct

​
Bidirectional LLM variants

Some Qwen and Llama checkpoints run in bidirectional mode: each token attends to the full input, which often improves embedding quality over causal pooling.
  • Qwen2Bidirectional: Alibaba-NLP/gte-Qwen2-7B-instruct
  • Qwen3Bidirectional: voyageai/voyage-4-nano (contact Baseten for deploy config)
  • Llama3Bidirectional: nvidia/llama-embed-nemotron-8b
Set base_model: encoder_bert. The build applies bidirectional attention automatically.

​
Checkpoint requirements

BEI-Bert builds standard Hugging Face checkpoints only. Repos that require trust_remote_code fail at build time. Pin checkpoint_repository.revision when the model maintainer publishes a compatible config on a non-default branch. For voyageai/voyage-4-nano, the default Hugging Face branch is not compatible with BEI-Bert. Contact your Baseten representative for the current checkpoint_repository settings before you deploy.

​
Reranking

BEI-Bert runs cross-encoder rerankers through /rerank. Recommended:
  • BAAI/bge-reranker-large (XLM-RoBERTa)
  • BAAI/bge-reranker-base (XLM-RoBERTa base)
  • Alibaba-NLP/gte-multilingual-reranker-base
  • Alibaba-NLP/gte-reranker-modernbert-base

​
Classification

BEI-Bert runs sequence classifiers through /predict. The classifier head needs an id2label dictionary in the Hugging Face config. Recommended:
  • SamLowe/roberta-base-go_emotions (sentiment)
  • papluca/xlm-roberta-base-language-detection (language ID)

​
Named entity recognition

Token-level entity classification routes to /predict_tokens and runs on BEI-Bert only. Recommended:
  • dslim/bert-base-NER-uncased (Truss example)
  • tanaos/tanaos-NER-v1
For the full request/response format and Python example, see Named entity recognition.

​
Model selection by constraint

Choose based on your primary constraint: Balanced cost and performance:
  • Alibaba-NLP/gte-Qwen2-7B-instruct: instruction-tuned, ranked #1 for multilingual.
  • Alibaba-NLP/gte-Qwen2-1.5B-instruct: 1/5 the size, still top-tier.
  • Snowflake/snowflake-arctic-embed-m-v2.0: multilingual-optimized, MRL support.
Lightweight (under 500M params):
  • google/embeddinggemma-300m: 300M params, 100+ languages.
  • nomic-ai/nomic-embed-text-v1.5: 137M, minimal latency.
  • sentence-transformers/all-MiniLM-L6-v2: 22M, legacy standard.
Specialized:
  • Code: jinaai/jina-embeddings-v2-base-code.
  • Long sequences: Alibaba-NLP/gte-large-en-v1.5.
  • Reranking: BAAI/bge-reranker-large, Alibaba-NLP/gte-reranker-modernbert-base.

​
Minimal configuration

BEI-Bert deployments set base_model: encoder_bert and quantization_type: no_quant. Pull weights from Hugging Face by default.
For the full schema, including max_num_tokens, GPU support, and complete examples for sentence-transformers, Jina, Nomic, and bidirectional LLM variants, see the BEI configuration reference.

​
Typed decisions

Laya returns typed answers to multiple questions about one plain-text state. It shares the runtime’s batching and GPU replicas. Native messages, images, audio, and video aren’t supported by the Laya adapter. The question types are:
  • choice: select an option from named criteria.
  • noul: return a probability of true for a boolean question.
  • score: return an expected zero-based level from ordered criteria.
See the Laya deployment and multi-question request. A deployment serves its configured checkpoint; the request’s model field doesn’t switch models.

​
Multi-GPU deployments

For throughput on small embedding, reranking, and decision models, start with two or four L4s, or RTX 6000 Ada GPUs where available. An L4 has 24 GB of GPU memory; an RTX 6000 Ada has 48 GB. Specify the RTX generation when selecting hardware because Quadro RTX 6000, RTX 6000 Ada, and RTX PRO 6000 Blackwell require different runtime builds. BEI-Bert loads a complete model on each GPU and distributes batches through a shared queue. Each GPU must fit the full weights and its batch. Adding GPUs increases serving capacity without combining their memory. For example, a 26B-parameter model needs about 52 GB for BF16 weights alone, exceeding an L4 or RTX 6000 Ada’s memory. Send concurrent requests to keep the GPUs busy. Measure throughput and p95 latency with your input lengths before increasing GPU count. See the two-L4 deployment example and available resources.