Async inference user guide
Run asynchronous inference on deployed models
Async requests are a “fire and forget” way of executing model inference requests. Instead of waiting for a response from a model, making an async request queues the request, and immediately returns with a request identifier. Optionally, async request results are sent via a POST
request to a user-defined webhook upon completion.
Use async requests for:
- Long-running inference tasks that may otherwise hit request timeouts.
- Batched inference jobs.
- Prioritizing certain inference requests.
Async fast facts:
- Async requests can be made to any model—no model code changes necessary.
- Async requests can remain queued for up to 12 hours and run for up to 1 hour.
- Async requests are not compatible with streaming model output.
- Async request inputs and model outputs are not stored after an async request has been completed. Instead, model outputs will be sent to your webhook via a
POST
request.
Quick start
There are two ways to use async inference:
- Provide a webhook endpoint where model outputs will be sent via a
POST
request. If providing a webhook, you can use async inference on any model, without making any changes to your model code. - Inside your Truss’
model.py
, save prediction results to cloud storage. If a webhook endpoint is provided, your model outputs will also be sent to your webhook.
Note that Baseten does not store model outputs. If you do not wish to use a webhook, your model.py
must write model outputs to a cloud storage bucket or database as part of its implementation.
Setup webhook endpoint
Set up a webhook endpoint for handling completed async requests. Since Baseten doesn’t store model outputs, model outputs from async requests will be sent to your webhook endpoint.
Before creating your first async request, try running a sample request against your webhook endpoint to ensure that it can consume async predict results properly. Check out this example webhook test.
We recommend using this Repl as a starting point.
Schedule an async predict request
Call /async_predict
on your model. The body of an /async_predict
request includes the model input in model_input
field, with the addition of a webhook endpoint (from the previous step) in the webhook_endpoint
field.
import requests
import os
model_id = "" # Replace this with your model ID
webhook_endpoint = "" # Replace this with your webhook endpoint URL
# Read secrets from environment variables
baseten_api_key = os.environ["BASETEN_API_KEY"]
# Call the async_predict endpoint of the production deployment
resp = requests.post(
f"https://model-{model_id}.api.baseten.co/production/async_predict",
headers={"Authorization": f"Api-Key {baseten_api_key}"},
json={
"model_input": {"prompt": "hello world!"},
"webhook_endpoint": webhook_endpoint
# Optional fields for priority, max_time_in_queue_seconds, etc
},
)
print(resp.json())
Save the request_id
from the /async_predict
response to check its status or cancel it.
{
"request_id": "9876543210abcdef1234567890fedcba"
}
See the async inference API reference for more endpoint details.
Check async predict results
Using the request_id
saved from the previous step, check the status of your async predict request:
import requests
import os
model_id = ""
request_id = ""
# Read secrets from environment variables
baseten_api_key = os.environ["BASETEN_API_KEY"]
resp = requests.get(
f"https://model-{model_id}.api.baseten.co/async_request/{request_id}",
headers={"Authorization": f"Api-Key {baseten_api_key}"}
)
print(resp.json())
Once your model has finished executing the request, the async predict result will be sent to your webhook in a POST
request.
{
"request_id": "9876543210abcdef1234567890fedcba",
"model_id": "my_model_id",
"deployment_id": "my_deployment_id",
"type": "async_request_completed",
"time": "2024-04-30T01:01:08.883423Z",
"data": {
"my_model_output": "hello world!"
},
"errors": []
}
Secure your webhook
We strongly recommend securing the requests sent to your webhooks to validate that they are from Baseten.
For instructions, see our guide to securing async requests.
User guide
Configuring the webhook endpoint
Configure your webhook endpoint to handle POST
requests with async predict results. We require that webhook endpoints use HTTPS.
We recommend running a sample request against your webhook endpoint to ensure that it can consume async predict results properly. Try running this webhook test.
For local development, we recommend using this Repl as a starting point. This code validates the webhook request and logs the payload.
Making async requests
import requests
import os
model_id = "" # Replace this with your model ID
webhook_endpoint = "" # Replace this with your webhook endpoint URL
# Read secrets from environment variables
baseten_api_key = os.environ["BASETEN_API_KEY"]
# Call the async_predict endpoint of the production deployment
resp = requests.post(
f"https://model-{model_id}.api.baseten.co/production/async_predict",
headers={"Authorization": f"Api-Key {baseten_api_key}"},
json={
"model_input": {"prompt": "hello world!"},
"webhook_endpoint": webhook_endpoint
# Optional fields for priority, max_time_in_queue_seconds, etc
},
)
print(resp.json())
Create an async request by calling a model’s /async_predict
endpoint. See the async inference API reference for more endpoint details.
Getting and canceling async requests
import requests
import os
model_id = ""
request_id = ""
# Read secrets from environment variables
baseten_api_key = os.environ["BASETEN_API_KEY"]
resp = requests.get(
f"https://model-{model_id}.api.baseten.co/async_request/{request_id}",
headers={"Authorization": f"Api-Key {baseten_api_key}"}
)
print(resp.json())
Manage async requests using the get async request API endpoint and the cancel async request API endpoint.
Processing async predict results
Baseten does not store async predict results. Ensure that prediction outputs are either processed by your webhook, or saved to cloud storage in your model code (for example, in your model’s postprocess
method).
If a webhook endpoint was provided in the /async_predict
request, the async predict results will be sent in a POST
request to the webhook endpoint. Errors in executing the async prediction will be included in the errors
field of the async predict result.
Async predict result schema:
request_id
(string): the ID of the completed async request. This matches therequest_id
field of the/async_predict
response.model_id
(string): the ID of the model that executed the requestdeployment_id
(string): the ID of the deployment that executed the requesttype
(string): the type of the async predict result. This will always be"async_request_completed"
, even in error cases.time
(datetime): the time in UTC at which the request was sent to the webhookdata
(dict or string): the prediction outputerrors
(list): any errors that occurred in processing the async request
Example async predict result:
{
"request_id": "9876543210abcdef1234567890fedcba",
"model_id": "my_model_id",
"deployment_id": "my_deployment_id",
"type": "async_request_completed",
"time": "2024-04-30T01:01:08.883423Z",
"data": {
"my_model_output": "hello world!"
},
"errors": []
}
Observability
Metrics for async request execution are available on the Metrics tab of your model dashboard.
- Async requests are included in inference latency and volume metrics.
- A time in async queue chart displays the time an async predict request spent in the
QUEUED
state before getting processed by the model. - A async queue size chart displays the current number of queued async predict requests.
The time in async queue chart.