Skip to main content
Call a hosted model with Model APIs or send requests to a model deployed in your workspace. Choose your starting point, then configure how your application sends input and receives results.

Model APIs

Call hosted models with the OpenAI or Anthropic SDK. No deployment required.

Call a deployed model

Use an existing deployment’s endpoint and its model-specific input format.

Deploy a model

Package your model and create a deployment on dedicated compute.
You need a Baseten API key to authenticate requests. Model APIs use the shared base URL https://inference.baseten.co/v1 and a model slug. Dedicated deployments use their own endpoint and the interface configured for that model.

​
Inference API

Choose the API your model exposes:
  • Model APIs: OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages for language models, or the System One API for decision models.
  • OpenAI-compatible deployments: Chat, embeddings, or other routes supported by your deployed engine or custom server. Set the SDK’s base URL to your deployment’s endpoint.
  • Custom Truss models: Send model-specific input to /predict. Your model code defines the input and output formats.
For more information, see Call your model and the Inference API reference.

​
Synchronous inference

Synchronous calls return a full response in one round trip, which fits interactive use (chat, code completion, classification, embeddings) where the client can wait for the answer. See Call your model for predict-style URLs across development, environment, and published deployments.

​
Streaming

Streaming returns output incrementally. OpenAI- and Anthropic-compatible APIs use server-sent events; custom Truss models can yield raw response chunks. Use Streaming to choose the matching client and check that generation completed.

​
Speech and audio

Deploy a dedicated model for transcription, speech generation, or speaker diarization. Start with Speech and audio, or transcribe speech in batch or real time.

​
Asynchronous inference

Async inference queues a request to a dedicated deployment and returns a request ID. Receive the result through a webhook or save it from inside your model. Polling reports request status only; it doesn’t return model output. Use Async inference for long-running work that doesn’t need an open client connection.

​
Structured outputs and tool calling

Choose the response behavior your application needs, then check support for your model:
  • JSON mode: Request JSON with a flexible shape.
  • Structured outputs: Constrain successful, complete responses to a JSON schema.
  • Function calling: Let the model select functions and arguments for your application to execute.
  • Web search: Let Model APIs execute supported search and fetch tools within a request.

​
Client configuration

For batch inference, use the Performance Client to manage connections, retries, and concurrency. If you use an HTTP client directly, see HTTP client configuration to tune connection reuse, timeouts, and parallelism.

​
Files and integrations

  • Binary I/O and File I/O: Send files to custom Truss models and save their outputs.
  • Integrations: Connect coding agents, application frameworks, gateways, and voice clients.
  • Frontier Gateway: Offer an inference API to your customers with API keys, usage limits, and billing events.
For routing, queuing, and cold starts on dedicated deployments, see Request lifecycle.