predict function after these steps. The request path explains where latency occurs, what each status code means, and how to debug a failed request.
How a request reaches your model
Your request first hits Baseten’s inference gateway, which authenticates it against your API key. If authentication fails, the gateway rejects the request before it reaches any model infrastructure:401 when the request has no credentials, 403 when the key is invalid. A standard endpoint also returns 403 when the environment requires a regional endpoint.
Once authenticated, the request moves to the routing layer. Baseten selects a workload plane where the deployment runs, then distributes traffic among the deployment’s replicas in that plane. The concurrency target defines how many simultaneous requests each replica should handle and informs autoscaling decisions.
When routing succeeds, Baseten forwards the request to the deployment, and a replica runs inference. For deployments that use the predict API, your predict function executes here. The response flows back through the same path to the client. For most requests, the routing overhead is negligible compared to your model’s inference time. The sections below cover what happens when this straightforward path breaks down: when no replica is available, when replicas reach capacity, and when requests fail partway through.
What happens when no replica is available
If your deployment has scaled to zero, no replica is ready to receive a request. By default, Baseten parks a synchronous request at the routing layer and waits for a replica to become available. Once a replica is ready, Baseten forwards the parked request, and the model processes it normally. From the client’s perspective, this wait adds to the normal inference time. This parking behavior makes scale-to-zero practical for deployments that queue requests. You don’t need to retry a request because your deployment was idle. The request waits for a replica, but only until the parking timeout expires. The parking timeout uses the same configured duration as the predict timeout, which is 1200 seconds by default. If no replica becomes available by then, the request fails with a500. To avoid scale-from-zero latency, keep minimum replicas greater than zero.
Async requests follow a different path. The /async_predict endpoint accepts the request into the async service without waiting for a replica. The service dispatches the request later and applies its own inference retry policy.
For strategies to reduce cold start latency, including warm replicas, pre-warming, and the Baseten Delivery Network, see Cold starts.
Request queuing and load shedding
Even when replicas are running, they can fill up. By default, incoming synchronous requests wait for an open replica slot. The autoscaler can add replicas within your configured limits. Load shedding prevents the queue from growing without bound. Baseten rejects a new request with a429 when queued payloads create memory pressure or the queue crosses its soft limit. Baseten returns a 529 when the queue crosses its hard limit. Retry either response with exponential backoff and jitter.
Your client can’t distinguish time spent in the queue from time spent running inference. Set a client-side timeout that matches your latency requirements. If queuing persists, increase max replicas when the deployment reaches its replica ceiling. Raise the concurrency target only when benchmarks show that each replica can handle more simultaneous requests.
Request backpressure policies
A request backpressure policy controls what happens to a new synchronous request when no replica has an open request slot:
Use Queue on full when your clients can tolerate variable latency and you want the deployment to absorb short traffic spikes. Use Reject on full when clients have strict latency limits or can retry requests elsewhere. Rejected requests don’t reach a replica, so they don’t contribute to concurrency-based autoscaling. Set the target utilization percentage below 100% so admitted traffic can trigger scaling before every request slot fills.
If the deployment has scaled to zero, the first request triggers a replica start in the background. That request and later requests receive
529 until a replica is ready. Retry the request with exponential backoff and jitter.
The reject policy applies only to synchronous HTTP requests. It doesn’t change how /async_predict queues and retries inference. Request backpressure policies aren’t available for WebSocket, gRPC, Chain, or Baseten Inference Stack deployments. The Request handling setting doesn’t affect Baseten Inference Stack admission behavior. Baseten Inference Stack uses separate router-level capacity controls, which can return 429 responses based on request shape, such as the number of cached tokens and tokens that aren’t cached.
Policy rejections appear as 5xx responses in the Inference volume chart. They also appear in deployment logs with the error code request_backpressure_rejected.
Backpressure configuration
Request backpressure policies are rolling out gradually. If the Request backpressure policy setting isn’t visible in your deployment or environment settings, contact support to enable it for your organization.
- Baseten CLI
- UI
- Management API
To configure a request backpressure policy:To restore queuing, pass
--policy queue-on-full. Pass --policy null to clear an explicit policy and use the default.For more information, see
deployment update-request-backpressure.Internal retries
When a replica returns a retryable status such as502, 503, or 504, the routing layer retries the request with exponential backoff. Connection failures use the same retry path. Retries stop when they reach the configured maximum number of prediction attempts or the request deadline, whichever comes first. The routing layer doesn’t apply these retries to async requests. The async service has a separate inference retry policy.
From your client’s perspective, retries add latency instead of immediately returning an error. Check the X-BASETEN-MODEL-PREDICTION-ATTEMPTS response header when a request takes longer than expected. A value greater than 1 confirms that Baseten retried the request. Baseten can temporarily suppress retries when the routing layer is under pressure. If a sticky-session request returns a 503, the retry routes to a different replica.
Timeouts
The predict timeout controls how long a sync request can take from the moment it’s forwarded to a replica until a response must be returned. If your model’s inference exceeds this window, the request is cancelled and the client receives a504. The server-side default is 1200 seconds (20 minutes). If you need requests to fail faster than that, set a client-side timeout in your HTTP client.
The async predict timeout is 3600 seconds (1 hour) per inference attempt. A timeout produces a retryable 504 within the async service. Baseten follows the request’s async inference retry policy. After the service exhausts those attempts, it marks the request as failed with a MODEL_PREDICT_TIMEOUT error status and sends the error payload to your webhook.
For a parked synchronous request, the parking timeout controls how long it waits when the deployment has no ready replicas. It uses the same configured duration as the synchronous predict timeout, but the two timeouts apply separately. If routing succeeds before the parking timeout expires, the full predict timeout starts when Baseten forwards the request to a replica. With the default settings, a synchronous request can spend up to 1200 seconds parked and then up to another 1200 seconds running inference.
For streaming responses, timeouts behave differently because the HTTP headers, including the 200 status code, are sent when the stream begins. If the timeout expires mid-stream, the stream stops and the connection closes without an error code, since the status was already written. Most HTTP clients surface this as a connection reset or incomplete response rather than a timeout error.
Requests during a promotion
You can promote a deployment while the current one is serving live traffic without dropping in-flight requests. As traffic shifts to the new deployment, each replica of the previous one stops accepting new requests and finishes the work it’s already doing before shutting down. There’s nothing to configure: draining happens automatically, whether the promotion replaces the deployment immediately or through a rolling deployment. Draining replicas stay alive for up to one hour. This window covers a single in-flight attempt: synchronous prediction has a 1200-second default timeout, and an async prediction attempt has a 3600-second timeout. The async service schedules any retry as a separate attempt. If your model runs its own shutdown logic, such as flushing state in a custom server, it receives aSIGTERM when its replica starts draining.
HTTP status codes
This table summarizes the status codes produced by the routing and lifecycle behavior on this page. Request-side errors such as400 and 413 come from other layers.
For what each error means, how to tell a model failure from a Baseten-side issue, and where to look next, see Inference errors.
A
500 from a sync request during a cold start can mean the parking timeout expired before a replica finished starting. Retry with exponential backoff. If this recurs, keep minimum replicas greater than zero or reduce cold-start time.Request cancellation
When a client disconnects before the response is written, the routing layer detects the closed connection and cancels the in-flight work. The server logs this as a499. In the common case, such as a user closing a browser tab or a client-side timeout firing, this is harmless and the 499 is informational rather than an error.
The more important question is whether cancellation propagates all the way to the GPU. If a client disconnects during a long generation and the model keeps running, you’re paying for GPU time that produces tokens nobody will read. Baseten cancels in-flight work automatically so this doesn’t happen. When the routing layer detects a disconnect, it signals the inference engine, which aborts the running request and frees GPU resources. This works across engines including TRT-LLM and vLLM.
If you’re using a custom model server, you can implement cancellation yourself using Truss request objects. See Request handling for code examples.
Next steps
Cold starts
Reduce cold start latency with warm replicas and pre-warming strategies.
Autoscaling
Configure concurrency targets, replica counts, and scaling dynamics.
Async inference
Fire-and-forget inference with webhook delivery.
Troubleshooting
Diagnose common deployment issues including autoscaling problems.