> ## Documentation Index
> Fetch the complete documentation index at: https://docs.baseten.co/llms.txt
> Use this file to discover all available pages before exploring further.
# Ornith 1.5
> Ornith 1.5 35B-A3B is an FP8 mixture-of-experts reasoning model for coding and agentic tasks.
Ornith 1.5 35B-A3B is an FP8 mixture-of-experts reasoning model for coding and agentic tasks. It activates about 3B parameters per token and serves OpenAI-compatible multimodal chat, reasoning content, and tool calls with vLLM.
## Setup
Sign in to Baseten with Truss, then install the OpenAI SDK.
**Sign in to Baseten**
```sh theme={"system"}
uvx truss login --browser
```
**Install the OpenAI SDK**
```sh theme={"system"}
uv pip install openai
```
This preset serves Ornith 1.5 35B-A3B on H100:2 with FP8 weights and tensor parallel size 2, optimized for low-latency interactive chat, reasoning, and agent workflows.
vLLM 0.19.1
256K
32
## Write the config
Create and move into the project directory:
```sh theme={"system"}
mkdir ornith-1.5-35b-a3b-latency && cd ornith-1.5-35b-a3b-latency
```
Then create a file named `config.yaml` and paste the following:
```yaml config.yaml theme={"system"}
model_name: "model:ornith-1.5-35b-a3b preset:latency"
model_metadata:
description: >-
Ornith 1.5 35B-A3B is an FP8 mixture-of-experts reasoning model for coding
and agentic tasks. It activates about 3B parameters per token and serves
OpenAI-compatible multimodal chat, reasoning content, and tool calls with vLLM.
repo_id: ornith-ai/Ornith-1.5-35B-A3B-FP8
tags:
- openai-compatible
- multimodal
- reasoning
- tool-use
example_model_input:
model: Ornith-1.5-35B-A3B
messages:
- role: user
content: "Write a one-line Python lambda that squares a number."
max_tokens: 1024
temperature: 0.6
top_p: 0.95
base_image:
image: vllm/vllm-openai:v0.19.1
weights:
- source: "hf://ornith-ai/Ornith-1.5-35B-A3B-FP8@0e048080ccd0ccf4296bfea5638036c196dccc0c"
mount_location: "/app/checkpoint/model"
auth_secret_name: "hf_access_token"
secrets:
hf_access_token: null
environment_variables:
VLLM_LOGGING_LEVEL: WARNING
VLLM_ENGINE_READY_TIMEOUT_S: "3600"
docker_server:
start_command: >-
vllm serve /app/checkpoint/model
--served-model-name Ornith-1.5-35B-A3B
--host 0.0.0.0
--port 8000
--tensor-parallel-size 2
--max-model-len 262144
--gpu-memory-utilization 0.90
--enable-prefix-caching
--enable-auto-tool-choice
--tool-call-parser qwen3_xml
--reasoning-parser qwen3
--trust-remote-code
--limit-mm-per-prompt.image 1
readiness_endpoint: /health
liveness_endpoint: /health
predict_endpoint: /v1/chat/completions
server_port: 8000
resources:
instance_type: H100:2
runtime:
predict_concurrency: 32
health_checks:
restart_check_delay_seconds: 1800
restart_threshold_seconds: 1200
stop_traffic_threshold_seconds: 120
```
This preset deploys the FP8 checkpoint of Ornith 1.5 35B-A3B through vLLM on two H100 GPUs with tensor parallelism. It enables prefix caching and preserves the checkpoint's native 256K context window. The deployment exposes an OpenAI-compatible chat completions endpoint with reasoning, tool calling, and one image per prompt.
## Flags
The `start_command` passes these flags to the engine. Each one controls a runtime or serving behavior:
| Flag | Value | What it does |
| ----------------------------- | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--tensor-parallel-size` | `2` | Number of GPUs to shard the model across. |
| `--max-model-len` | `262144` | Maximum context length (tokens) the server accepts per request. |
| `--gpu-memory-utilization` | `0.90` | Fraction of GPU memory vLLM may use for weights and KV cache. |
| `--enable-prefix-caching` | (no value) | Reuse KV cache across requests that share a prefix. |
| `--enable-auto-tool-choice` | (no value) | Let the model choose when to call tools without requiring `tool_choice: "required"`. |
| `--tool-call-parser` | `qwen3_xml` | Server-side parser that emits structured `tool_calls` on the response. |
| `--reasoning-parser` | `qwen3` | Server-side parser that separates reasoning output into `reasoning_content`. **qwen3:** Qwen3-family thinking format (used by Qwen3, Qwen3.5, and Qwen3.6). |
| `--trust-remote-code` | (no value) | Execute model-specific Python from the checkpoint (required for many Qwen, Phi, and custom architectures). |
| `--limit-mm-per-prompt.image` | `1` | Maximum number of image inputs per prompt. |
## Deploy
Push the config to Baseten:
```sh theme={"system"}
uvx truss push
```
You should see output similar to:
```output theme={"system"}
✨ Model ornith-1.5-35b-a3b-latency was successfully pushed ✨
Model ID: abc1d2ef
Deployment ID: xyz123
Endpoint: model-abc1d2ef.api.baseten.co
Logs: https://app.baseten.co/models/abc1d2ef/logs/xyz123
```
`truss push` prints your **model ID** (`abc1d2ef` in the example). The examples below use it wherever you see `{model_id}`, and read your API key from the `BASETEN_API_KEY` environment variable.
## Call the model
Your deployment serves an OpenAI-compatible API.
Now call your deployment to run inference:
```python main.py theme={"system"}
import os
from openai import OpenAI
client = OpenAI(
api_key=os.environ["BASETEN_API_KEY"],
base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
)
response = client.chat.completions.create(
model="Ornith-1.5-35B-A3B",
messages=[
{"role": "user", "content": "What is machine learning?"}
],
)
print(response.choices[0].message.content)
```
```sh theme={"system"}
curl -s https://model-{model_id}.api.baseten.co/environments/production/sync/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $BASETEN_API_KEY" \
-d '{
"model": "Ornith-1.5-35B-A3B",
"messages": [
{"role": "user", "content": "What is machine learning?"}
]
}'
```
To access the model's chain of thought, enable thinking mode. The server parses the reasoning output into a separate `reasoning_content` field on the response:
```python theme={"system"}
response = client.chat.completions.create(
model="Ornith-1.5-35B-A3B",
messages=[
{"role": "user", "content": "How many r's in strawberry?"}
],
extra_body={"chat_template_kwargs": {"enable_thinking": True}},
)
print(response.choices[0].message.reasoning_content) # chain of thought
print(response.choices[0].message.content) # final answer
```
To let the model call tools, pass a `tools` array. The server returns structured `tool_calls` on the response:
```python theme={"system"}
tools = [{
"type": "function",
"function": {
"name": "get_weather",
"parameters": {
"type": "object",
"properties": {"location": {"type": "string"}},
"required": ["location"],
},
},
}]
response = client.chat.completions.create(
model="Ornith-1.5-35B-A3B",
messages=[
{"role": "user", "content": "What's the weather in Paris?"}
],
tools=tools,
)
print(response.choices[0].message.tool_calls)
```
## Next steps
Endpoint anatomy, authentication, and sync versus async inference
Scale replicas with traffic, including scale to zero