Streaming

Streaming returns model output incrementally so your application can display text before generation finishes. Choose the protocol exposed by your endpoint: Model APIs support OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages; custom Truss models can yield raw chunks. For real-time transcription over WebSocket, use the speech quickstart.

Enable streaming

Set BASETEN_API_KEY to your Baseten API key. Install the dependencies listed in your selected tab. The Model APIs examples use a hosted model; custom Truss streaming requires a deployed model whose code yields output.

Chat Completions
Responses
Anthropic Messages
Custom Truss

To stream a response:Install openai, httpx, and httpx-sse with pip install openai httpx httpx-sse. Run the following script. It reads raw server-sent events so it can check both [DONE] and the finish reason.

# Install dependencies: pip install openai httpx httpx-sse
import json
import os

from httpx_sse import EventSource
from openai import OpenAI

client = OpenAI(
    base_url="https://inference.baseten.co/v1",
    api_key=os.environ["BASETEN_API_KEY"],
)

completed = False
finish_reason = None
with client.chat.completions.with_streaming_response.create(
    model="deepseek-ai/DeepSeek-V4.1-Flash-1017",
    messages=[{"role": "user", "content": "Reply with exactly: Café from Baseten."}],
    stream=True,
) as response:
    for event in EventSource(response.http_response).iter_sse():
        if event.data == "[DONE]":
            completed = True
            break
        chunk = json.loads(event.data)
        if event.event == "error" or "error" in chunk:
            raise RuntimeError(f"Stream error: {chunk}")
        for choice in chunk.get("choices", []):
            print(choice["delta"].get("content") or "", end="", flush=True)
            if choice.get("finish_reason") is not None:
                finish_reason = choice["finish_reason"]

if not completed or finish_reason is None:
    raise RuntimeError("Stream ended before completion; output is partial")
print(f"\nFinish reason: {finish_reason}")

Café from Baseten.
Finish reason: stop

To stream a response:Install the OpenAI SDK with pip install openai. Run the following script. It reads typed events and retains the terminal response status.

# Model APIs: OpenAI Responses endpoint at inference.baseten.co
import os

from openai import OpenAI

client = OpenAI(
    base_url="https://inference.baseten.co/v1",
    api_key=os.environ["BASETEN_API_KEY"],
)

final = None
stream = client.responses.create(
    model="deepseek-ai/DeepSeek-V4.1-Flash-1017",
    input="Reply with exactly: Café from Baseten.",
    stream=True,
)
for event in stream:
    if event.type == "response.output_text.delta":
        print(event.delta, end="", flush=True)
    elif event.type in ("response.completed", "response.incomplete"):
        final = event.response
    elif event.type in ("response.failed", "error"):
        raise RuntimeError(f"Stream error: {event}")

if final is None:
    raise RuntimeError("Stream ended before completion; output is partial")
print(f"\nStatus: {final.status}")
if final.incomplete_details:
    print(f"Incomplete reason: {final.incomplete_details.reason}")

Café from Baseten.
Status: completed

To stream a response:Install the Anthropic SDK with pip install anthropic. Run the following script against the Messages endpoint. It reads events so it can check message_stop and the stop reason.

# Model APIs: Anthropic-compatible endpoint at inference.baseten.co
import os
import anthropic

api_key = os.environ["BASETEN_API_KEY"]

client = anthropic.Anthropic(
    base_url="https://inference.baseten.co",
    api_key=api_key,
    default_headers={"Authorization": f"Bearer {api_key}"},
)

completed = False
stop_reason = None
with client.messages.create(
    model="deepseek-ai/DeepSeek-V4.1-Flash-1017",
    max_tokens=4096,
    stream=True,
    messages=[{"role": "user", "content": "Reply with exactly: Café from Baseten."}],
) as stream:
    for event in stream:
        if event.type == "content_block_delta" and event.delta.type == "text_delta":
            print(event.delta.text, end="", flush=True)
        elif event.type == "message_delta":
            stop_reason = event.delta.stop_reason
        elif event.type == "message_stop":
            completed = True

if not completed or stop_reason is None:
    raise RuntimeError("Stream ended before completion; output is partial")
print(f"\nStop reason: {stop_reason}")

Café from Baseten.
Stop reason: end_turn

To stream a response:Install requests with pip install requests and set BASETEN_MODEL_ID to your model ID. For a test stream, deploy a Truss with this model code:

model/model.py

class Model:
    def predict(self, model_input):
        yield "Caf"
        yield "é"
        yield " from Baseten."

Run the following client. Your production model defines its own payload and chunk format. UTF-8 decoding keeps characters intact even when they span network chunks.

import os
import requests

with requests.post(
    f"https://model-{os.environ['BASETEN_MODEL_ID']}.api.baseten.co/environments/production/predict",
    headers={"Authorization": f"Bearer {os.environ['BASETEN_API_KEY']}"},
    json={"prompt": "Reply with exactly: Café from Baseten.", "stream": True},
    stream=True,
    timeout=60,
) as response:
    response.raise_for_status()
    response.encoding = "utf-8"
    for chunk in response.iter_content(chunk_size=None, decode_unicode=True):
        print(chunk, end="", flush=True)
print()

Café from Baseten.

The text arrives incrementally; the output blocks show the final terminal output. Model wording and chunk boundaries can vary.

Inspect server-sent events with cURL

To inspect the Chat Completions event stream: Run the following command. --no-buffer displays events as they arrive:

curl https://inference.baseten.co/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $BASETEN_API_KEY" \
  -d '{
    "model": "deepseek-ai/DeepSeek-V4.1-Flash-1017",
    "messages": [{"role": "user", "content": "Reply with exactly: Café from Baseten."}],
    "stream": true
  }' \
  --no-buffer

data: {"choices":[{"index":0,"delta":{"content":"Café from Baseten."}}]}

data: {"choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]

The stream contains data: records and ends with data: [DONE]. Check the preceding chunk’s finish_reason as well. For more information, see Chat Completions streaming.

Handle incomplete responses

For Chat Completions and Messages, check the terminal event and stop reason before accepting the text. A token limit can end generation with a completion marker while leaving an unfinished answer. Responses reports this as response.incomplete, with the cause in incomplete_details.reason. The SDK procedures above raise an error if their terminal event is missing. The custom Truss client reads raw response chunks, which have no standard model-completion marker. If your application needs to distinguish complete output from an interrupted generation, define a terminal record in your model’s streaming format and check it in the client. cURL displays the wire format without validating it. Transport and API errors can occur after HTTP 200. Keep partial text separate from a retried generation. For more information, see completion markers and recovery, read timeouts, and bounded retries.

Time to first token

Streaming changes when the caller sees output. Time to first token (TTFT) measures how long the caller waits before receiving the first generated token; total generation time includes the rest of the response. The diagram compares streaming and non-streaming delivery for the same illustrative generation. Both finish together; the streaming client can display partial output earlier. These timings are illustrative, not measured latency.

Get started

Model APIs

Baseten Grid

Inference

Frontier Gateway

Training

Sandboxes

Development

Deployment

Engines

Model Labs

Organization

Hosting options

Observability

Troubleshooting

Enable streaming

Inspect server-sent events with cURL

Handle incomplete responses

Time to first token

​Enable streaming

​Inspect server-sent events with cURL

​Handle incomplete responses

​Time to first token

Enable streaming

Inspect server-sent events with cURL

Handle incomplete responses

Time to first token