This configuration builds an inference engine to serve Qwen 2.5 3B on an A10G GPU. It is very similar to the configuration for any other Qwen model, including fine-tuned variants.

Recommended basic GPU configurations for Qwen 2.5 sizes:

Size and variantFP16 unquantizedFP8 quantized
3B (Instruct)A10GN/A
7B (Instruct, Math, Code)H100_40GBN/A
14B (Instruct)H100H100_40GB
32B (Instruct)H100:2H100
72B (Instruct, Math)H100:4H100:2

If you use multiple GPUs, make sure to match num_builder_gpus and tensor_parallel_count in the config. When quantizing, you may need to double the number of builder GPUs.

​
Setup

See the end-to-end engine builder tutorial prerequisites for full setup instructions.

Please upgrade to the latest version of Truss with pip install --upgrade truss before following this example.

pip install --upgrade truss
mkdir qwen-engine
touch qwen-engine/config.yaml

​
Configuration

This configuration file specifies model information and Engine Builder arguments. For a different Qwen model, change the model_name, accelerator, and repo fields, along with any changes to the build arguments.

config.yaml
model_name: Qwen 2.5 3B Instruct
resources:
  accelerator: A10G
  use_gpu: true
trt_llm:
  build:
    base_model: qwen
    checkpoint_repository:
      repo: Qwen/Qwen2.5-3B-Instruct
      source: HF
    max_seq_len: 8192
    num_builder_gpus: 1
    quantization_type: no_quant
    tensor_parallel_count: 1

​
Deployment

truss push --publish

​
Usage

call_model.py
import requests
import os

# Model ID for production deployment
model_id = ""
# Read secrets from environment variables
baseten_api_key = os.environ["BASETEN_API_KEY"]

# Call model endpoint
resp = requests.post(
    f"https://model-{model_id}.api.baseten.co/production/predict",
    headers={"Authorization": f"Api-Key {baseten_api_key}"},
    json={
      "messages": [
        {"role": "system", "content": "You are Qwen, created by Alibaba Cloud. You are a helpful assistant."},
        {"role": "user", "content": "What does Tongyi Qianwen mean?"}
        ],
      "max_tokens": 512
    },
    stream=True
)

# Print the generated tokens as they get streamed
for content in resp.iter_content():
    print(content.decode("utf-8"), end="", flush=True)
prompt
string

The input text prompt to guide the language model’s generation.

One of prompt XOR messages is required.

messages
List[Dict]

A list of dictionaries representing the message history, typically used in conversational contexts.

One of prompt XOR messages is required.

max_tokens
int

The maximum number of tokens to generate in the output. Controls the length of the generated text.

beam_width
int
default: "1"

The number of beams used in beam search. Maximum of 1.

repetition_penalty
float

A penalty applied to repeated tokens to discourage the model from repeating the same words or phrases.

presence_penalty
float

A penalty applied to tokens already present in the prompt to encourage the generation of new topics.

temperature
float

Controls the randomness of the output. Lower values make the output more deterministic, while higher values increase randomness.

length_penalty
float

A penalty applied to the length of the generated sequence to control verbosity. Higher values make the model favor shorter outputs.

end_id
int

The token ID that indicates the end of the generated sequence.

pad_id
int

The token ID used for padding sequences to a uniform length.

runtime_top_k
int

Limits the sampling pool to the top k tokens, ensuring the model only considers the most likely tokens at each step.

runtime_top_p
float

Applies nucleus sampling to limit the sampling pool to a cumulative probability p, ensuring only the most likely tokens are considered.

Mistral with TensorRT-LLMEngine control in Python