Structured outputs

Structured outputs let you generate text that conforms to specific JSON schemas, providing reliable data extraction and controlled text generation. Model APIs support structured outputs except for Muse Spark Open, which accepts json_schema but doesn’t enforce the schema. For self-deployed models, Baseten engines like BIS-LLM and Engine-Builder-LLM support them, as do other inference frameworks like vLLM and SGLang.

Quick start

Define the JSON Schema your application expects, then pass it to a model that enforces structured outputs. The Python examples use Pydantic to create the schema and parse the result. You can also send JSON Schema directly through the API. You need a Baseten API key in BASETEN_API_KEY. Install the Python dependencies with pip install openai pydantic.

Define a schema

Define a Pydantic model whose fields describe the output you want:

schema.py

from typing import Literal
from pydantic import BaseModel

class Task(BaseModel):
    title: str
    priority: Literal["low", "medium", "high"]
    due_date: str
    description: str

Each field requires a type annotation. The schema constrains the types and allowed priority values in a successful, complete response.

Generate structured output

Pass the schema to the parse method and read the typed result. Model APIs accept either OpenAI endpoint; dedicated deployments use your model’s own URL.

Model APIs
Dedicated deployment

To generate a task with Model APIs:Run the example for your API. Point the client at https://inference.baseten.co/v1 and pass a model slug from the supported models table. beta.chat.completions.parse takes the class in response_format, and responses.parse takes it in text_format:

Chat Completions
Responses

To parse a task with Chat Completions:Run the following script:

import os
from typing import Literal
from pydantic import BaseModel
from openai import OpenAI, LengthFinishReasonError

class Task(BaseModel):
    title: str
    priority: Literal["low", "medium", "high"]
    due_date: str
    description: str

client = OpenAI(
    api_key=os.environ['BASETEN_API_KEY'],
    base_url="https://inference.baseten.co/v1"
)

try:
    response = client.beta.chat.completions.parse(
        model="moonshotai/Kimi-K3",
        messages=[
            {"role": "user", "content": "Create a task for: Review the quarterly report by next Friday"}
        ],
        response_format=Task
    )
except LengthFinishReasonError as error:
    raise RuntimeError("Output limit reached. Increase the limit or shorten the request.") from error

message = response.choices[0].message
if message.refusal:
    raise RuntimeError(f"Model refused: {message.refusal}")
task = message.parsed
if task is None:
    raise RuntimeError("No parsed task returned.")
print(f"Task: {task.title}")
print(f"Priority: {task.priority}")

Task: Review the quarterly report
Priority: medium

To parse a task with Responses:Run the following script:

import os
from typing import Literal
from pydantic import BaseModel, ValidationError
from openai import OpenAI

class Task(BaseModel):
    title: str
    priority: Literal["low", "medium", "high"]
    due_date: str
    description: str

client = OpenAI(
    api_key=os.environ['BASETEN_API_KEY'],
    base_url="https://inference.baseten.co/v1"
)

try:
    response = client.responses.parse(
        model="moonshotai/Kimi-K3",
        input="Create a task for: Review the quarterly report by next Friday",
        text_format=Task
    )
except ValidationError as error:
    raise RuntimeError("Output could not be parsed. It may be incomplete or invalid for the schema.") from error

if response.status != "completed":
    details = response.error or response.incomplete_details or response.status
    raise RuntimeError(f"Response did not complete: {details}")
task = response.output_parsed
if task is None:
    raise RuntimeError("No parsed task returned. Check the response for a refusal.")
print(f"Task: {task.title}")
print(f"Priority: {task.priority}")

Task: Review the quarterly report
Priority: high

To generate a task with your deployed model:Set BASETEN_MODEL_ID to the deployment’s model ID and BASETEN_SERVED_MODEL to the model name accepted by its engine or server. Run the following script against a deployment that supports Chat Completions and structured outputs:

import os
from typing import Literal
from pydantic import BaseModel
from openai import OpenAI, LengthFinishReasonError

class Task(BaseModel):
    title: str
    priority: Literal["low", "medium", "high"]
    due_date: str
    description: str

client = OpenAI(
    api_key=os.environ['BASETEN_API_KEY'],
    base_url=f"https://model-{os.environ['BASETEN_MODEL_ID']}.api.baseten.co/environments/production/sync/v1"
)

try:
    response = client.beta.chat.completions.parse(
        model=os.environ["BASETEN_SERVED_MODEL"],
        messages=[
            {"role": "user", "content": "Create a task for: Review the quarterly report by next Friday"}
        ],
        response_format=Task
    )
except LengthFinishReasonError as error:
    raise RuntimeError("Output limit reached. Increase the limit or shorten the request.") from error

message = response.choices[0].message
if message.refusal:
    raise RuntimeError(f"Model refused: {message.refusal}")
task = message.parsed
if task is None:
    raise RuntimeError("No parsed task returned.")
print(f"Task: {task.title}")
print(f"Priority: {task.priority}")

Task: Review the quarterly report
Priority: medium

The output values and wording can vary between runs. Pass your Pydantic class to response_format with beta.chat.completions.parse, or to text_format with responses.parse, instead of calling create and parsing the text yourself. On a successful, complete response, message.parsed for Chat Completions or output_parsed for Responses is a Task object. Check refusals and completion status before using it. A token limit can interrupt generation even when the request asks for a schema. responses.parse can raise Pydantic’s ValidationError while parsing truncated JSON, before it returns a response whose status you can check. For more information, see Inference errors.

Use with LangChain

To generate structured output with LangChain: Install langchain-openai with pip install langchain-openai, then run the following script. ChatOpenAI uses the configured OpenAI-compatible endpoint:

import os
from typing import Literal
from pydantic import BaseModel
from langchain_openai import ChatOpenAI

class Task(BaseModel):
    title: str
    priority: Literal["low", "medium", "high"]
    due_date: str
    description: str

# Model APIs: use "https://inference.baseten.co/v1" and a model slug.
# Dedicated deployment: use your production sync endpoint and the model name
# accepted by its server.
llm = ChatOpenAI(
    api_key=os.environ["BASETEN_API_KEY"],
    base_url="https://inference.baseten.co/v1",
    model="moonshotai/Kimi-K3",
)

structured_llm = llm.with_structured_output(Task)
task = structured_llm.invoke("Create a task for: Review the quarterly report by next Friday")

print(f"Task: {task.title}")
print(f"Priority: {task.priority}")

Task: Review the quarterly report
Priority: medium

See the LangChain integration for other ways to use Baseten with LangChain.

Engine support

Structured outputs are compatible with:

Engine-Builder-LLM, except when Lookahead speculative decoding is configured.
BIS-LLM, except in some configurations, such as when the overlap scheduler is enabled.

Model support

Check the engine and decoding configuration before enabling structured outputs. The exclusions above still apply; a model’s presence on an engine does not override those constraints.

Best practices

Schema design

Keep schemas simple: two to three levels of nesting for best results.
Use basic types: str, int, float, bool when possible.
Optional fields: Use nullable types when a field can be absent from the source input. Follow the endpoint’s supported JSON Schema subset.
Descriptive names: Use clear, descriptive field names.

Prompt engineering

Low temperature: Use 0.1 to 0.3 for consistent outputs.
Provide schema: Dump the model schema and few-shot examples into context.
Provide context: Give background for complex schemas.

Engine-Builder-LLM overview: Dense model documentation.
BIS-LLM overview: MoE model documentation.
Quantization guide: FP8/FP4 trade-offs.

Get started

Model APIs

Inference

Frontier Gateway

Training

Sandboxes

Development

Deployment

Engines

Model Labs

Organization

Hosting options

Observability

Troubleshooting

Structured outputs

Quick start

Define a schema

Generate structured output

Use with LangChain

Engine support

Model support

Best practices

Schema design

Prompt engineering

​Quick start

​Define a schema

​Generate structured output

​Use with LangChain

​Engine support

​Model support

​Best practices

​Schema design

​Prompt engineering

​Related

Quick start

Define a schema

Generate structured output

Use with LangChain

Engine support

Model support

Best practices

Schema design

Prompt engineering

Related