Reuse client sessions
Creating a new HTTP client per request prevents connection reuse. Repeated connection setup adds TCP and TLS overhead and can exhaust client or NAT resources under load. A reused client maintains a pool of open connections that are ready for subsequent requests. This reduces connection setup overhead. Match pool size and request concurrency to your workload.- Recommended
- Anti-pattern
To reuse a connection pool:Set
BASETEN_MODEL_ID and BASETEN_API_KEY, install httpx with pip install httpx, then run the following script. Replace the payload with your model’s input. For an identity Truss, the request returns the input unchanged:Choose an HTTP client
Your choice of HTTP client library determines which connection management features are available to you. The httpx library is recommended over requests for Baseten workloads because it provides built-in connection pooling, native async support, and optional HTTP/2. Therequests library can achieve connection reuse through its
Session object, but lacks async support and requires more manual
configuration.
The OpenAI Python SDK manages a client connection pool. Reuse the SDK client across requests and close it when your application shuts down.
To configure a shared HTTP client:
Set BASETEN_MODEL_ID and BASETEN_API_KEY, then configure httpx.Client. Close the client when your application shuts down:
client.py
Configure connection pooling
Connection pooling keeps a set of open TCP connections ready for reuse. When your client sends a request, it draws from this pool instead of opening a new connection. This reduces repeated TCP handshakes and TLS negotiations. Measure connection setup latency in your network before choosing pool settings. By default, httpx allows 100 total connections and retains up to 20 idle keep-alive connections across its pool. The idle limit is not a per-host concurrency limit. A workload with more in-flight requests than available connections can exhaust the pool. When the pool is full, new requests block until a connection becomes available, resulting inPoolTimeout
errors or increased latency.
Increase the pool limits based on your peak concurrency using httpx.Limits. The
max_keepalive_connections setting controls how many idle connections stay
open, and keepalive_expiry controls how long idle connections persist before
closing. Choose an idle expiry for your workload and network, including any proxy or load balancer. Configure the limits when you create the client:
client.py
Recommended values
These values apply when calling a single Baseten model endpoint.
If you call multiple models, increase max connections proportionally.
Configure HTTP/2 streams
Baseten allows up to 100 concurrent streams on each HTTP/2 connection. For high-throughput workloads, configure your client to:- Use several connections. Send each request on the busiest connection that has fewer than 100 active streams, and open a new connection when every connection is full.
- Raise the receive windows. HTTP/2 starts each stream with a 64 KiB flow-control window. Set the initial stream and connection windows to 1 MB, or 2 MB for long streaming responses, so the server doesn’t pause for
WINDOW_UPDATEframes.
GOAWAY frame after 999 requests. Use a client that finishes in-flight requests and opens a new connection when it receives GOAWAY; otherwise, requests can fail with REFUSED_STREAM.
To use HTTP/2 with the Performance Client, set http_version=2.
Set timeouts
By default, httpx raises a timeout after 5 seconds of network inactivity. Many inference requests need more time before the first response data arrives. Without a longer read timeout, your client can close the connection before the model finishes processing. Set client timeouts based on your latency budget. Baseten applies separate server-side timeouts before and after routing. A synchronous request can wait for a replica for up to the 1200-second default parking timeout. After Baseten forwards the request to a replica, a separate 1200-second default predict timeout begins. A request that waits for capacity can therefore remain open for longer than 1200 seconds. For details, see Request lifecycle. For a warm deployment, budget enough read time for the expected inference time and a buffer. If the deployment can scale to zero or wait for capacity, also include the expected wait and cold start time. To avoid scale-from-zero latency, keep the deployment’s minimum replicas greater than zero. httpx lets you configure four separate timeout values withhttpx.Timeout. Separating connect and
read timeouts prevents slow network conditions from being confused with slow
model responses. Configure the timeouts when you create the client:
client.py
Timeout guidance by use case
The
read timeout limits how long httpx waits for incoming data. For a
non-streaming request, it must cover the wait before the response arrives. For
a streaming request, it applies while waiting for the response headers and
between response chunks. The default server-side parking and predict phases
can total 2400 seconds, but your client timeout should reflect how long your
client can wait.
For long-running requests that exceed sync timeouts, use async inference
with a webhook to receive the output. If needed, poll the status endpoint to
monitor the request lifecycle; it doesn’t return model output.
Implement retries
Retry transient failures with exponential backoff and jitter. If the response includesRetry-After, wait at least that long before retrying. Use a library such as tenacity to limit retry attempts.
Set an overall deadline for the operation, including requests and backoff delays. Account for your SDK’s automatic retries when adding application retries so nested retry loops don’t multiply attempts.
For Model APIs, handle retryable errors in your client even when the routing layer retries internally. Don’t depend on a fixed number of internal attempts. For failures after a streaming response starts, follow Streaming responses.
Retry the following errors when the cause is transient. Repeated application errors or timeouts need investigation rather than more retries:
- 429 (rate limited)
- 500 (internal server error)
- 502 (bad gateway)
- 503 (service unavailable)
- 504 (gateway timeout)
- 529 (overloaded)
- Connection errors (
ConnectError,ReadTimeout)
- 400 (bad request)
- 401 (unauthorized)
- 403 (forbidden)
- 404 (not found)
- 422 (validation error)
httpx and tenacity with pip install httpx tenacity, then run the following script. It limits a non-streaming request to five attempts with exponential backoff and jitter. It doesn’t implement an overall deadline or handle Retry-After; add those to match your application’s retry budget:
Handle errors
Use the exception type and request logs to identify which phase failed. A pool timeout points to local connection contention; a refused connection can originate at the destination, a proxy, or the network path.Monitor connections
Connection problems tend to surface as intermittent failures rather than complete outages, making them difficult to diagnose without proper monitoring. A gradually exhausting connection pool won’t cause errors until it’s completely full, at which point requests start failing unpredictably. Watch for these signals:- Rising p99 latency without changes to model performance, which can indicate pool contention; compare with model and network latency.
- Sporadic
Connection refusederrors under load, which warrant checking destination availability and client or NAT port usage. - TCP retransmits increasing over time, which suggest connections are being dropped and recreated.
Use with proxies
Enterprise deployments often route traffic through HTTP proxies for security, logging, or network policy enforcement. httpx supports proxy configuration at the client level, so connection pooling and keep-alives continue to work through the proxy. You may need to increase your pool limits when using a proxy, since the additional network hop increases per-request latency, which means connections are held open longer and the pool drains faster under the same concurrency. Pass the proxy URL when you create the client:client.py
Further reading
- Performance Client: Handles connection pooling, retries, and concurrency automatically.
- Async inference: For long-running requests that exceed sync timeout limits.
- Streaming: For streaming model responses.