Async inference user guide

Async requests are a “fire and forget” way of executing model inference requests. Instead of waiting for a response from a model, making an async request queues the request, and immediately returns with a request identifier. Async inference results are sent via a POST request to a user-defined webhook upon completion.

Use async requests for:

• Long-running inference tasks that may otherwise hit request timeouts.
• Batched inference jobs.
• Prioritizing certain inference requests.

​

Quick start

import requests
import os

model_id = ""  # Replace this with your model ID
webhook_endpoint = ""  # Replace this with your webhook endpoint URL
# Read secrets from environment variables
baseten_api_key = os.environ["BASETEN_API_KEY"]

# Call the async_predict endpoint of the production deployment
resp = requests.post(
    f"https://model-{model_id}.api.baseten.co/production/async_predict",
    headers={"Authorization": f"Api-Key {baseten_api_key}"},
    json={
      "model_input": {"prompt": "hello world!"},
      "webhook_endpoint": webhook_endpoint
      # Optional fields for priority, max_time_in_queue_seconds, etc
    }
)

print(resp.json())

{
  "request_id": "9876543210abcdef1234567890fedcba"
}

import requests
import os

model_id = ""
request_id = ""
# Read secrets from environment variables
baseten_api_key = os.environ["BASETEN_API_KEY"]

resp = requests.get(
    f"https://model-{model_id}.api.baseten.co/async_request/{request_id}",
    headers={"Authorization": f"Api-Key {baseten_api_key}"}
)

print(resp.json())

{
  "request_id": "9876543210abcdef1234567890fedcba",
  "model_id": "my_model_id",
  "deployment_id": "my_deployment_id",
  "type": "async_request_completed",
  "time": "2024-04-30T01:01:08.883423Z",
  "data": {
    "my_model_output": "hello world!"
  },
  "errors": []
}

​

User guide

​

Configuring the webhook endpoint

Configure your webhook endpoint to handle POST requests with async predict results. We require that webhook endpoints use HTTPS.

We recommend running a sample request against your webhook endpoint to ensure that it can consume async predict results properly. Try running this webhook test.

For local development, we recommend using this Repl as a starting point. This code validates the webhook request and logs the payload.

​

Making async requests

• Production deployment
• Development deployment
• Other published deployments

import requests
import os

model_id = ""
webhook_endpoint = ""
# Read secrets from environment variables
baseten_api_key = os.environ["BASETEN_API_KEY"]

resp = requests.post(
    f"https://model-{model_id}.api.baseten.co/production/async_predict",
    headers={"Authorization": f"Api-Key {baseten_api_key}"},
    json={
      "model_input": {"prompt": "hello world!"},
      "webhook_endpoint": webhook_endpoint
      # Optional fields for priority, max_time_in_queue_seconds, etc
    }
)

print(resp.json())

Create an async request by calling a model’s /async_predict endpoint. See the async inference API reference for more endpoint details.

​

Getting and canceling async requests

• Get async request details
• Cancel async request

import requests
import os

model_id = ""
request_id = ""
# Read secrets from environment variables
baseten_api_key = os.environ["BASETEN_API_KEY"]

resp = requests.get(
    f"https://model-{model_id}.api.baseten.co/async_request/{request_id}",
    headers={"Authorization": f"Api-Key {baseten_api_key}"}
)

print(resp.json())

Manage async requests using the get async request API endpoint and the cancel async request API endpoint.

​

Processing async predict results

Once the model has finished executing the async prediction, the async predict results will be sent in a POST request to your webhook endpoint provided in the /async_predict request. Errors in executing the async prediction will be added to the errors field and sent to your webhook.

Baseten does not store these async predict results, ensure they are processed by your webhook.

Async predict result schema:

• request_id (string): the ID of the completed async request. This matches the request_id field of the /async_predict response.
• model_id (string): the ID of the model that executed the request
• deployment_id (string): the ID of the deployment that executed the request
• type (string): the type of the async predict result. This will always be "async_request_completed", even in error cases.
• time (datetime): the time in UTC at which the request was sent to the webhook
• data (dict or string): the prediction output
• errors (list): any errors that occurred in processing the async request

Example async predict result:

{
  "request_id": "9876543210abcdef1234567890fedcba",
  "model_id": "my_model_id",
  "deployment_id": "my_deployment_id",
  "type": "async_request_completed",
  "time": "2024-04-30T01:01:08.883423Z",
  "data": {
    "my_model_output": "hello world!"
  },
  "errors": []
}

​

Observability

Metrics for async request execution are available on the Metrics tab of your model dashboard.

• Async requests are included in inference latency and volume metrics.
• A time in async queue chart displays the time an async predict request spent in the QUEUED state before getting processed by the model.
• A async queue size chart displays the current number of queued async predict requests.