> ## Documentation Index > Fetch the complete documentation index at: https://docs.baseten.co/llms.txt > Use this file to discover all available pages before exploring further. # Inference errors > What each inference error means and where to look next. When an inference request fails, the error message alone often isn't enough to tell you what to fix. The status code is the first clue. It tells you the broad category, and because your request passes through Baseten's inference gateway before it reaches your model, it also points at where the failure happened: your request, the Baseten platform, or your model's own code. Use the [quick reference](#quick-reference) to find the cause and next steps for an HTTP error. The error body depends on the endpoint and the source of the failure. For incomplete streams or failures after async submission, see [Streaming and async errors](#streaming-and-async-errors). ## How to read an inference error A failed response has two parts worth reading: * **The status code**: the broad category, returned in the HTTP response (`502`, `503`, and so on). * **The error body**: `{"error": ""}`. The message is either a short string Baseten generates (such as `Error making prediction`) or, when your model itself returned the error, the raw response your model produced, passed through unchanged. One distinction resolves most failures: did your model return the error, or was your model unreachable? See [Is it my model or Baseten?](#is-it-my-model-or-baseten). ## Quick reference | Status | What it usually means | Where to look | | - | - | - | | `400` / `401` / `403` / `404` | Malformed request, invalid API key, wrong model ID, or the standard URL for a regional environment or deployment | Your request: [check the endpoint and API key](#400-401-403-404-request-and-authentication-errors) | | `402` | Billing or payment issue on the account | [Billing and usage](/organization/billing) | | `413` | Request body too large: 100 MB edge cap (all requests), or 256 KiB async default | Send large inputs by [file or URL](/inference/output-format/files), not inline | | `429` | Rate limit exceeded, or no capacity was available for the request | Back off and retry; see [429](#429-too-many-requests) | | `499` | The client disconnected before the response was written | Usually informational; [check client-side timeouts](#499-client-closed-request) | | `500` | Your model code raised an exception, or no replica became available before the parking timeout | [Model logs](/observability/logs) for the traceback; see [500](#500-internal-server-error) | | `502` | Your container crashed or restarted mid-request, or a Baseten-side error | [Model logs](/observability/logs) for a crash or `OOMKilled`; if clean, retry | | `503` | Container not available yet: draining or routing | Mostly transient: retry with exponential backoff | | `504` | The prediction exceeded the request timeout (1200s sync) | Profile the model, raise resources, or use [async](/inference/async) | | `529` | Baseten shed the request because capacity was exhausted | Back off and retry; see [529](#529-overloaded) | ## Is it my model or Baseten? When a request reaches your model and your model server responds with an error, Baseten passes that response through to you. When the request never gets a real answer from your model, Baseten returns its own error instead. Telling these apart is the fastest way to know where to debug. **Your model returned the error**: the status code and body come from your model server. Your handler raised an exception, returned a non-2xx status, or timed out internally. Debug it like any application bug, starting from the error body and your [model logs](/observability/logs). **Your model was unreachable**: the request failed in front of your container, so you get a message like `Error making prediction` with a `502` or `503`. The container is down, restarting, was killed (for example, out of memory), or is still cold-starting. The most common version of the second case is memory pressure. When you increase your payload or batch sizes, each request uses more memory, and the container can be killed mid-request (`OOMKilled`). Confirm it by checking your [model logs](/observability/logs) for `OOMKilled` or repeated restarts, and your [metrics](/observability/metrics) for memory pressure and replica restarts. To fix it, reduce per-request memory with smaller batches or payloads, or move to a larger instance type. ## Errors by status code Each section below covers what the status code means, its common causes, and what to check first. ### HTTP 400, 401, 403, and 404: request and authentication errors These point to the request itself, not your model. * **`401`**: the request has no credentials; the body reads `Unauthorized`. Send a [Baseten API key](/organization/api-keys) in the `Authorization` header. * **`403`**: either the key is invalid or revoked, or the target requires a regional endpoint. An invalid key returns `Authentication failed`; check the key's value and that it belongs to the right workspace. If the key is valid, use the endpoint for the [regional environment](/deployment/regional-environments#regional-endpoint-formats) or [regional deployment](/deployment/regional-deployments#regional-endpoints). * **`404`**: the model or deployment ID doesn't exist, or the model was deleted. Confirm the ID and that you're calling the right [predict endpoint](/inference/calling-your-model). * **`400`**: the request or URL is malformed. Check the request body is valid JSON and the path is correct. An OpenAI-compatible [custom server](/development/model/custom-server) such as vLLM returns its own `404` when the request body's `model` field doesn't match a served model: ```json theme={"system"} { "error": { "code": 404, "message": "The model `my-model` does not exist.", "param": "model" } } ``` A nested error object identifies a server response. Baseten's own `404` body is flat: `{"error": ""}`. For vLLM, set `model` to the `--served-model-name` value. To list served models, call `/v1/models` through the [sync endpoint](/inference/calling-your-model#sync-api-endpoints): ```bash theme={"system"} curl https://model-abc123.api.baseten.co/environments/production/sync/v1/models \ -H "Authorization: Bearer $BASETEN_API_KEY" ``` For Truss CLI authentication errors (such as a missing key in `~/.trussrc`), see [Troubleshooting inference](/troubleshooting/inference). ### HTTP 402: payment required The account has an unresolved billing or payment issue, such as exhausted credits with no payment method on file. Check your [billing and usage](/organization/billing) settings, or contact your account owner. ### HTTP 413: payload too large The request body exceeded a size limit. Two limits can return a `413`: * **Edge limit**: the ingress proxy caps every inbound request body at [100 MB](/reference/inference-api/overview#request-size) and rejects larger requests before they reach your model or chain. It covers the full HTTP body, including the JSON envelope and any base64-encoded media, and isn't configurable. * **Async limit**: [async requests](/inference/async) have a smaller per-organization cap, 256 KiB (262,144 bytes) by default. The message reports both sizes: `payload size of X bytes exceeds maximum size of Y bytes`. The async payload limit is set per organization. [Contact support](mailto:support@baseten.co) to raise it for your organization. The 100 MB edge limit is fixed. For large inputs, send a file or URL the model fetches instead of inlining the bytes in the request body. See [Model I/O with files](/inference/output-format/files). A payload that fits under the edge limit but exhausts the container's memory returns a `502`, not a `413`. See [Is it my model or Baseten?](#is-it-my-model-or-baseten). ### HTTP 429: too many requests The request exceeded a rate limit, or arrived when no capacity was available. Where the limit lives depends on the surface you're calling: * **Model APIs**: the request exceeded the requests-per-minute limit or one of the model's token rate limits. Check `x-ratelimit-remaining-requests` and the returned token limit headers to see which limit reached zero. Models with a combined token rate limit return `x-ratelimit-remaining-tokens`. Models with split token rate limits return `x-ratelimit-remaining-uncached-input-tokens` and `x-ratelimit-remaining-output-tokens`. See [Rate limit response headers](/inference/model-apis/pricing-and-limits#rate-limit-response-headers) for the full header list. * **Loops trainer provisioning**: `Could not schedule trainer. Please try again later.` means no eligible GPU capacity is available. Retry later. See [Loops capacity and quota errors](/troubleshooting/loops#resolve-capacity-and-quota-errors) to distinguish this response from an organization capacity limit. * **Async endpoints**: `/async_predict` allows 12,000 requests per minute per organization, and the status and cancel endpoints allow 100 requests per second. See [Async inference](/inference/async#rate-limits). * **Dedicated deployments**: a response with the `CAPACITY_EXCEEDED` error code means every replica slot was full. Load shedding can also reject a request because queued payloads create routing memory pressure or the queue crosses its soft limit. See [Request queuing and load shedding](/deployment/autoscaling/request-lifecycle#request-queuing-and-load-shedding). Retry with exponential backoff. Persistent `429` responses mean that backoff alone won't restore enough capacity. For `CAPACITY_EXCEEDED` responses on dedicated deployments, increase [max replicas](/deployment/autoscaling/overview#param-max-replica) or tune the [concurrency target](/deployment/autoscaling/overview#param-concurrency-target). Dedicated load shedding returns `429` for routing memory pressure or the soft queue limit and [`529`](#529-overloaded) for the hard queue limit. ### HTTP 499: client closed request A `499` means the client disconnected before the response was written: a user closed a browser tab, a client-side timeout fired, or the calling service cancelled the request. Because the client has already gone away, a `499` shows up in your deployment's request metrics and logs rather than as a response your code handles. The [metrics](/observability/metrics) count the request under status `499`, and the [logs](/observability/logs) record `Client closed connection` with the request ID. When the client disconnects, Baseten cancels the in-flight work and aborts the running request on the inference engine, so you don't pay for GPU time generating output nobody reads. See [Request cancellation](/deployment/autoscaling/request-lifecycle#request-cancellation). Occasional `499`s are normal and need no server-side action. A steady stream of them usually means a client-side timeout shorter than your model's response time, cancelling requests that would have succeeded. Set the client timeout to match your model's expected latency. See [Configure HTTP clients](/inference/http-client-configuration#set-timeouts). ### HTTP 500: internal server error A `500` has two meanings, distinguished by the error message: * **`Internal Server Error (in model/chainlet).`**: your model code raised an exception during the request. The full traceback is in your [model logs](/observability/logs); debug it like any application bug. * **`Failed to route request`**: Baseten waited for a replica, but none became available before the parking timeout expired. The parking timeout is 1200 seconds by default. A cold start that takes longer than the timeout can cause this error, such as when a model loads weights after scaling from zero. Retry with exponential backoff. If it happens regularly, keep [minimum replicas](/deployment/autoscaling/overview#param-min-replica) greater than zero to avoid scale-from-zero waits, or see [Startup times](/deployment/autoscaling/startup-times) for ways to make startup faster. ### HTTP 502: bad gateway A `502` has two meanings, and they need different responses: * **Your container was unreachable**: the most common case. The container crashed, restarted, or was killed (for example, `OOMKilled`) mid-request. The body is a short message like `Error making prediction`. Check your [model logs](/observability/logs) for a crash or restart. See [Is it my model or Baseten?](#is-it-my-model-or-baseten). * **A Baseten-side error**: a transient problem in the gateway or routing layer. If your model logs are clean and show no crash, retry with exponential backoff. ### HTTP 503: service unavailable The container isn't available to take the request yet. This is usually transient. Retry with exponential backoff. Common causes: * **Draining**: an instance is shutting down during a deploy or scale-down. The message asks you to retry on another instance. * **Routing**: the request couldn't be routed to a workload plane, or a circuit breaker is open to protect an unhealthy upstream. By default, a synchronous request that arrives while the deployment is scaling up from zero doesn't return a `503`. Baseten holds it at the routing layer until a replica is ready, then forwards it. See [Request lifecycle](/deployment/autoscaling/request-lifecycle). Async requests return `503` rather than `502` when the async service isn't set up on the workload plane yet. See [Async inference](/inference/async). ### HTTP 504: gateway timeout The prediction ran longer than the [request timeout](/reference/inference-api/overview#timeouts) (1200 seconds for sync predict). Common causes are a model that's too slow for the payload, an under-provisioned instance, or a hung request. Profile the model, raise its resources, or move long-running work to the [async API](/inference/async), which allows up to 3600 seconds per inference attempt and can retry a timed-out attempt. If you see a timeout sooner than 1200 seconds, check your client's own timeout. Set it to match your model's expected response time. See [Configure HTTP clients](/inference/http-client-configuration#set-timeouts). ### HTTP 529: overloaded A `529` means the routing layer refused the request because the deployment couldn't accept more work. One cause is a queue crossing its hard load-shedding limit. A `529` doesn't indicate an account rate limit. Retry with exponential backoff and jitter. If the response carries a `Retry-After` header, wait at least that long instead of using your own interval. If your client already retries `429`, treat `529` the same way. Persistent `529` responses mean that backoff alone won't restore enough capacity. Raise [max replicas](/deployment/autoscaling/overview#param-max-replica) or tune the [concurrency target](/deployment/autoscaling/overview#param-concurrency-target). `529` isn't a standard HTTP status code. Some clients treat unrecognized 5xx codes as fatal, so check that yours retries `529` instead of failing over. ## Streaming and async errors Not every failure arrives as a status code with a JSON body. ### Streaming responses Baseten can send the `200` response headers before the first output chunk. After sending those headers, it can't change the HTTP status to report a later failure. A timeout or an unavailable model can end the stream early without an error event. If a streaming request returns a retryable HTTP error before the stream starts, follow [Implement retries](/inference/http-client-configuration#implement-retries). Client retry handling is still needed when the routing layer retries internally. After a response starts, check whether the stream completed before restarting it. #### Check whether the stream completed For Model API requests using Chat Completions or Messages, check both the stream-end marker and the reason generation stopped. The following checks cover these two formats, not the Responses API used for [web search](/inference/model-apis/web-search). | API format | Stream-end marker | Normal stop | Output token limit reached | | - | - | - | - | | Chat Completions | `data: [DONE]` | `finish_reason: "stop"` | `finish_reason: "length"` | | Messages | `event: message_stop` | `stop_reason: "end_turn"` | `stop_reason: "max_tokens"` | A token limit ends the stream with its usual marker, but the answer can be incomplete. Inspect the stop reason before accepting the result. A `200` status or a closed connection alone doesn't establish that the stream completed. These excerpts show raw server-sent events and omit earlier events. The OpenAI parsed iterator consumes `[DONE]` internally, and Anthropic's `text_stream` yields only text. Use the [streaming examples](/inference/streaming#enable-streaming) to read raw OpenAI events or iterate Anthropic events and check for premature termination. A [Chat Completions](/reference/inference-api/chat-completions) stream ends with this marker: ```text Stream-end marker theme={"system"} data: [DONE] ``` An interrupted stream might end after a content chunk like this, without the final `data: [DONE]` marker: ```text Partial output theme={"system"} data: {"id":"chatcmpl-example","object":"chat.completion.chunk","created":0,"model":"","choices":[{"index":0,"delta":{"content":"The result is"},"finish_reason":null}]} ``` `[DONE]` marks the end of the stream. Also inspect `finish_reason` in the preceding chunks to distinguish a normal stop from an output limit or a tool call. A [Messages](/reference/inference-api/messages) stream ends with a `message_stop` event: ```text Stream-end marker theme={"system"} event: message_stop data: {"type":"message_stop"} ``` An interrupted stream might end after a text delta like this, without a final `message_stop` event: ```text Partial output theme={"system"} event: content_block_delta data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"The result is"}} ``` Also inspect `stop_reason` in the preceding `message_delta` event to determine why generation stopped. #### Recover from an interrupted stream For Chat Completions and Messages, treat an explicit error event or a missing stream-end marker as a failed request. The stream can end without an error object, even if your SDK doesn't raise an exception. Retrying starts a new request; it doesn't resume the interrupted stream. Keep partial output separate from the retry's output. Use [bounded retries](/inference/http-client-configuration#implement-retries), and save completed results so a failed request doesn't require rerunning an entire background job. * For dedicated deployments, check your [model logs](/observability/logs). * For Model APIs, save the model name, timestamp, and any error details. Include the `x-baseten-request-id` response header when available. Share these details when [contacting support](mailto:support@baseten.co). ### Async requests Baseten returns submission errors immediately, including an invalid request, authentication failure, payload limit, rate limit, or unavailable async service. After Baseten accepts an async request, a later inference failure appears in the result payload's `errors` array, which is empty on success. See [Async inference](/inference/async). ## Where to look next When an error isn't self-explanatory, these are the fastest places to confirm a cause: * **[Model logs](/observability/logs)**: crashes, restarts, `OOMKilled`, and your model's own error output. * **[Metrics](/observability/metrics)**: memory pressure, replica count and restarts, and queue depth. * **[Request lifecycle](/deployment/autoscaling/request-lifecycle)**: how queuing, cold starts, and concurrency affect request handling. * **[Async inference](/inference/async)**: for payloads or runtimes that shouldn't go through the synchronous path. If you're still stuck, contact support with your model or deployment ID, the timestamp, the status code, and the request ID.