Inference errors

Read the HTTP status and error body together to decide whether to correct the request or retry it. Recovery also depends on the target: Baseten manages Model APIs infrastructure, while Dedicated Inference gives you control over your deployment’s code, resources, and scaling. Use the quick reference to find the cause and next steps for an HTTP error. The error body depends on the endpoint and the source of the failure. For incomplete streams or failures after async submission, see Streaming and async errors.

How to read an inference error

A failed response has two parts worth reading:

The status code: the broad category, returned in the HTTP response (502, 503, and so on).
The error body: details about the failure, whose shape depends on the endpoint and error source. It can contain a message string, such as {"error": "<message>"}, or a structured error object. Inspect the message and error code rather than inferring the source from the shape alone.

For Model APIs, check your request parameters, model slug and feature support, and model limits. Retry transient failures with bounded backoff. If failures persist, save the model slug, timestamp, status, error body, and x-baseten-request-id response header when available, then contact support. For a dedicated deployment, distinguish an error returned by your model from a failure to reach it. See Is it my model or Baseten?.

Quick reference

Status	What it usually means	Where to look
`400` / `401` / `403` / `404`	Malformed request, invalid API key, wrong model slug or ID, or an incorrect regional endpoint	Your request: check the endpoint and API key
`402`	Billing or payment issue on the account	Billing and usage
`413`	Request body too large; limits depend on the endpoint	Reduce the payload; use files or URLs only when the endpoint and model accept them
`429`	Rate limit exceeded, or no capacity was available for the request	Back off and retry; see 429
`499`	The client disconnected before the response was written	Usually informational; check client-side timeouts
`500`	Internal failure; dedicated model code or routing can also cause it	Choose recovery for your target
`502`	Gateway failure, or a dedicated container crashed or restarted mid-request	Retry or inspect deployment logs
`503`	Container not available yet: draining or routing	Mostly transient: retry with exponential backoff
`504`	The request timed out	Check client timeouts and target-specific recovery
`529`	Baseten shed the request because capacity was exhausted	Back off and retry; see 529

Is it my model or Baseten?

For dedicated deployments, distinguish errors from your model server from failures before the request reaches it. When your model server responds with an error, Baseten passes that response through. When the request doesn’t get an answer from your model, Baseten returns its own error. Your model returned the error: the status code and body come from your model server. Your handler raised an exception, returned a non-2xx status, or timed out internally. Debug it like any application bug, starting from the error body and your model logs. Your model was unreachable: the request failed in front of your container, so you get a message like Error making prediction with a 502 or 503. The container is down, restarting, was killed (for example, out of memory), or is still cold-starting. The most common version of the second case is memory pressure. When you increase your payload or batch sizes, each request uses more memory, and the container can be killed mid-request (OOMKilled). Confirm it by checking your model logs for OOMKilled or repeated restarts, and your metrics for memory pressure and replica restarts. To fix it, reduce per-request memory with smaller batches or payloads, or move to a larger instance type.

Errors by status code

Each section below covers what the status code means, its common causes, and what to check first.

HTTP 400, 401, 403, and 404: request and authentication errors

Check the request, credentials, and target before retrying.

401 / 403: credentials are missing, invalid, revoked, or don’t permit the request. Send a Baseten API key in the Authorization header and confirm it belongs to the right workspace. For dedicated deployments, a 403 can also mean you need the endpoint for the regional environment or regional deployment.
404: for Model APIs, confirm the exact supported model slug. For dedicated deployments, confirm the model or deployment ID exists and you’re calling the right predict endpoint.
400: the request or URL is malformed. Check the request body is valid JSON and the path is correct.

A dedicated OpenAI-compatible custom server such as vLLM returns its own 404 when the request body’s model field doesn’t match a served model:

{
  "error": {
    "code": 404,
    "message": "The model `my-model` does not exist.",
    "param": "model"
  }
}

For vLLM, set model to the --served-model-name value. To list served models, call /v1/models through the sync endpoint:

curl https://model-abc123.api.baseten.co/environments/production/sync/v1/models \
  -H "Authorization: Bearer $BASETEN_API_KEY"

For Truss CLI authentication errors (such as a missing key in ~/.trussrc), see Troubleshooting inference.

HTTP 402: payment required

The account has an unresolved billing or payment issue, such as exhausted credits with no payment method on file. Check your billing and usage settings, or contact your account owner.

HTTP 413: payload too large

The request body exceeded an endpoint’s size limit. For dedicated prediction requests, check these limits:

Edge limit: the ingress proxy caps the request body at 100 MB and rejects larger requests before they reach your model or Chain. It covers the full HTTP body, including the JSON envelope and any base64-encoded media, and isn’t configurable.
Async limit: async requests have a smaller per-organization cap, 256 KiB (262,144 bytes) by default. The message reports both sizes: payload size of X bytes exceeds maximum size of Y bytes.

The async payload limit is set per organization. Contact support to raise it for your organization. The 100 MB edge limit is fixed.

For Model APIs, check the selected endpoint’s limits, including the server-side tool execution limit when enabled. Reduce the payload, or use a file or URL only if the endpoint and model support that input format. For a dedicated model that fetches files, send a file or URL instead of inlining its bytes. See Model I/O with files. A payload that fits under the edge limit but exhausts the container’s memory can return a 502 instead. See Is it my model or Baseten?.

HTTP 429: too many requests

The request exceeded a rate limit, or arrived when no capacity was available. Where the limit lives depends on the surface you’re calling:

Model APIs: the request exceeded the requests-per-minute or tokens-per-minute limit. Check x-ratelimit-remaining-requests and x-ratelimit-remaining-tokens to see which limit reached zero. See Rate limit response headers to learn how the remaining values work.
Loops trainer provisioning: Could not schedule trainer. Please try again later. means no eligible GPU capacity is available. Retry later. See Loops capacity and quota errors to distinguish this response from an organization capacity limit.
Async endpoints: /async_predict allows 12,000 requests per minute per organization, and the status and cancel endpoints allow 100 requests per second. See Async inference.
Dedicated deployments: a response with the CAPACITY_EXCEEDED error code means every replica slot was full. Load shedding can also reject a request because queued payloads create routing memory pressure or the queue crosses its soft limit. See Request queuing and load shedding.

Retry with exponential backoff. Persistent 429 responses mean that backoff alone won’t restore enough capacity. For CAPACITY_EXCEEDED responses on dedicated deployments, increase max replicas or tune the concurrency target. Dedicated load shedding returns 429 for routing memory pressure or the soft queue limit and 529 for the hard queue limit.

HTTP 499: client closed request

A 499 means the client disconnected before the response was written: a user closed a browser tab, a client-side timeout fired, or the calling service cancelled the request. On dedicated deployments, a 499 shows up in your deployment’s request metrics and logs rather than as a response your code handles. The metrics count the request under status 499, and the logs record Client closed connection with the request ID. When the client disconnects, Baseten cancels the in-flight work and aborts the running request on the inference engine, so you don’t pay for GPU time generating output nobody reads. See Request cancellation. Occasional 499s are normal and need no server-side action. A steady stream of them usually means a client-side timeout shorter than your model’s response time, cancelling requests that would have succeeded. Set the client timeout to match your model’s expected latency. See Configure HTTP clients.

HTTP 500: internal server error

For Model APIs, retry with bounded backoff. If the failure persists, contact support with the model slug, timestamp, error body, and request ID. For dedicated deployments, the error message can distinguish these causes:

Internal Server Error (in model/chainlet).: your model code raised an exception during the request. The full traceback is in your model logs; debug it like any application bug.
Failed to route request: Baseten waited for a replica, but none became available before the parking timeout expired. The parking timeout is 1200 seconds by default. A cold start that takes longer than the timeout can cause this error, such as when a model loads weights after scaling from zero. Retry with exponential backoff. If it happens regularly, keep minimum replicas greater than zero to avoid scale-from-zero waits, or see Startup times for ways to make startup faster.

HTTP 502: bad gateway

For Model APIs, retry with bounded backoff and contact support if failures persist. Baseten manages the serving containers. For dedicated deployments, check these causes:

Your container was unreachable: the most common case. The container crashed, restarted, or was killed (for example, OOMKilled) mid-request. The body is a short message like Error making prediction. Check your model logs for a crash or restart. See Is it my model or Baseten?.
A Baseten-side error: a transient problem in the gateway or routing layer. If your model logs are clean and show no crash, retry with exponential backoff.

HTTP 503: service unavailable

Retry a 503 with exponential backoff. For Model APIs, persistent failures need support investigation; server-side tool execution returns 503 when its execution fleet is at capacity. For dedicated deployments, the container isn’t available to take the request yet. Common causes:

Draining: an instance is shutting down during a deploy or scale-down. The message asks you to retry on another instance.
Routing: the request couldn’t be routed to a workload plane, or a circuit breaker is open to protect an unhealthy upstream.

By default, a synchronous request that arrives while the deployment is scaling up from zero doesn’t return a 503. Baseten holds it at the routing layer until a replica is ready, then forwards it. See Request lifecycle.

Async requests return 503 rather than 502 when the async service isn’t set up on the workload plane yet. See Async inference.

HTTP 504: gateway timeout

For Model APIs, retry with bounded backoff. If timeouts persist, reduce the input or requested output length and contact support with the request details. Model APIs don’t expose instance resources or queued async inference. For dedicated deployments, the prediction ran longer than the request timeout (1200 seconds for sync predict). Common causes are a model that’s too slow for the payload, an under-provisioned instance, or a hung request. Profile the model, raise its resources, or move long-running work to the async API, which allows up to 3600 seconds per inference attempt and can retry a timed-out attempt. If you see a timeout sooner than 1200 seconds, check your client’s own timeout. Set it to match your model’s expected response time. See Configure HTTP clients.

HTTP 529: overloaded

A 529 means capacity was exhausted. It doesn’t indicate an account rate limit. For dedicated deployments, one cause is a queue crossing its hard load-shedding limit. Retry with exponential backoff and jitter. If the response carries a Retry-After header, wait at least that long instead of using your own interval. If your client already retries 429, treat 529 the same way. For persistent overload on Model APIs, contact support with the model slug, timestamp, and request ID. For a dedicated deployment, raise max replicas or tune the concurrency target. 529 isn’t a standard HTTP status code. Some clients treat unrecognized 5xx codes as fatal, so check that yours retries 529 instead of failing over.

Streaming and async errors

Not every failure arrives as a status code with a JSON body.

Streaming responses

Baseten can send the 200 response headers before the first output chunk. After sending those headers, it can’t change the HTTP status to report a later failure. A timeout or an unavailable model can end the stream early without an error event. If a streaming request returns a retryable HTTP error before the stream starts, follow Implement retries. Client retry handling is still needed when the routing layer retries internally. After a response starts, check whether the stream completed before restarting it.

Check whether the stream completed

For Model API requests, check both the stream-end marker and the reason generation stopped.

API format	Stream-end marker	Normal stop	Output token limit reached
Chat Completions	`data: [DONE]`	`finish_reason: "stop"`	`finish_reason: "length"`
Responses	`event: response.completed`	`status: "completed"`	`event: response.incomplete`, `incomplete_details.reason: "max_output_tokens"`
Messages	`event: message_stop`	`stop_reason: "end_turn"`	`stop_reason: "max_tokens"`

For Chat Completions and Messages, a token limit ends the stream with its usual marker, but the answer can be incomplete. Inspect the stop reason before accepting the result. A 200 status or a closed connection alone doesn’t establish that the stream completed. These excerpts show raw server-sent events and omit earlier events. The OpenAI parsed iterator consumes [DONE] internally, and Anthropic’s text_stream yields only text. Use the streaming examples to read raw OpenAI events or iterate Anthropic events and check for premature termination.

Chat Completions
Responses
Messages

A Chat Completions stream ends with this marker:

Stream-end marker

data: [DONE]

An interrupted stream might end after a content chunk like this, without the final data: [DONE] marker:

Partial output

data: {"id":"chatcmpl-example","object":"chat.completion.chunk","created":0,"model":"<MODEL_SLUG>","choices":[{"index":0,"delta":{"content":"The result is"},"finish_reason":null}]}

[DONE] marks the end of the stream. Also inspect finish_reason in the preceding chunks to distinguish a normal stop from an output limit or a tool call.

A Responses stream ends with response.completed:

Stream-end marker

event: response.completed
data: {"type":"response.completed","sequence_number":9,"response":{"id":"resp_example","status":"completed"}}

An interrupted stream might end after a text delta like this, without a terminal event:

Partial output

event: response.output_text.delta
data: {"type":"response.output_text.delta","sequence_number":3,"delta":"The result is"}

response.completed is the only normal ending. response.incomplete means generation stopped early: read incomplete_details.reason, which reports max_output_tokens when the output cap ended the answer. response.failed carries the error.

A Messages stream ends with a message_stop event:

Stream-end marker

event: message_stop
data: {"type":"message_stop"}

An interrupted stream might end after a text delta like this, without a final message_stop event:

Partial output

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"The result is"}}

Also inspect stop_reason in the preceding message_delta event to determine why generation stopped.

Recover from an interrupted stream

For Chat Completions, Responses, and Messages, treat an explicit error event or a missing stream-end marker as a failed request. The stream can end without an error object, even if your SDK doesn’t raise an exception. Retrying starts a new request; it doesn’t resume the interrupted stream. Keep partial output separate from the retry’s output. Use bounded retries, and save completed results so a failed request doesn’t require rerunning an entire background job.

For dedicated deployments, check your model logs.
For Model APIs, save the model name, timestamp, and any error details. Include the x-baseten-request-id response header when available. Share these details when contacting support.

Async requests

Queued async inference is available for dedicated models and Chains. Baseten returns submission errors immediately, including an invalid request, authentication failure, payload limit, rate limit, or unavailable async service. After Baseten accepts an async request, a later inference failure appears in the result payload’s errors array, which is empty on success. See Async inference.

Where to look next

For Model APIs, check the model’s limits and save the request details for support. Baseten manages the resources and scaling. For dedicated models and Chains, inspect:

Model logs: crashes, restarts, OOMKilled, and your model’s own error output.
Metrics: memory pressure, replica count and restarts, and queue depth.
Request lifecycle: how queuing, cold starts, and concurrency affect request handling.
Async inference: for payloads or runtimes that shouldn’t go through the synchronous path.

If you’re still stuck, contact support with your model slug or deployment ID, the timestamp, the status code, the error body, and the request ID.

Get started

Model APIs

Inference

Frontier Gateway

Training

Sandboxes

Development

Deployment

Engines

Model Labs

Organization

Hosting options

Observability

Troubleshooting

Inference errors

How to read an inference error

Quick reference

Is it my model or Baseten?

Errors by status code

HTTP 400, 401, 403, and 404: request and authentication errors

HTTP 402: payment required

HTTP 413: payload too large

HTTP 429: too many requests

HTTP 499: client closed request

HTTP 500: internal server error

HTTP 502: bad gateway

HTTP 503: service unavailable

HTTP 504: gateway timeout

HTTP 529: overloaded

Streaming and async errors

Streaming responses

Check whether the stream completed

Recover from an interrupted stream

Async requests

Where to look next

​How to read an inference error

​Quick reference

​Is it my model or Baseten?

​Errors by status code

​HTTP 400, 401, 403, and 404: request and authentication errors

​HTTP 402: payment required

​HTTP 413: payload too large

​HTTP 429: too many requests

​HTTP 499: client closed request

​HTTP 500: internal server error

​HTTP 502: bad gateway

​HTTP 503: service unavailable

​HTTP 504: gateway timeout

​HTTP 529: overloaded

​Streaming and async errors

​Streaming responses

​Check whether the stream completed

​Recover from an interrupted stream

​Async requests

​Where to look next

How to read an inference error

Quick reference

Is it my model or Baseten?

Errors by status code

HTTP 400, 401, 403, and 404: request and authentication errors

HTTP 402: payment required

HTTP 413: payload too large

HTTP 429: too many requests

HTTP 499: client closed request

HTTP 500: internal server error

HTTP 502: bad gateway

HTTP 503: service unavailable

HTTP 504: gateway timeout

HTTP 529: overloaded

Streaming and async errors

Streaming responses

Check whether the stream completed

Recover from an interrupted stream

Async requests

Where to look next