Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.baseten.co/llms.txt

Use this file to discover all available pages before exploring further.

Truss processes client requests by extracting and validating payloads. For advanced use cases, you can access the raw request object to:
  • Customize payload deserialization (for example, binary protocol buffers).
  • Handle disconnections and cancel long-running predictions.
You can mix request objects with standard inputs or use requests exclusively for performance optimization.

​
Using request objects in Truss

You can define request objects in preprocess, predict, and postprocess: 
import fastapi

class Model:
    def preprocess(self, request: fastapi.Request):
        ...

    def predict(self, inputs, request: fastapi.Request):
        ...

    def postprocess(self, inputs, request: fastapi.Request):
        ...

​
Rules for using requests

  • The request must be type-annotated as fastapi.Request.
  • If only requests are used, Truss skips payload extraction for better performance.
  • If both request objects and standard inputs are used:
    • Request must be the second argument.
    • Preprocessing transforms inputs, but the request object remains unchanged.
    • postprocess can’t use only the request. It must receive the model’s output.
    • If predict only uses the request, preprocess cannot be used.
import fastapi, asyncio, logging

class Model:
    async def predict(self, inputs, request: fastapi.Request):
        await asyncio.sleep(1)
        if await request.is_disconnected():
            logging.warning("Cancelled before generation.")
            return  # Cancel request on the model engine here.

        for i in range(5):
            await asyncio.sleep(1.0)
            logging.warning(i)
            yield str(i)  # Streaming response
            if await request.is_disconnected():
                logging.warning("Cancelled during generation.")
                return  # Cancel request on the model engine here.
You must implement request cancellation at the model level, which varies by framework.

​
Cancelling requests in specific frameworks

​
TRT-LLM (polling-based cancellation)

For TensorRT-LLM, use response_iterator.cancel() to terminate streaming requests: 
async for request_output in response_iterator:
    if await is_cancelled_fn():
        logging.info("Request cancelled. Cancelling Triton request.")
        response_iterator.cancel()
        return
See full example in TensorRT-LLM Docs.

​
vLLM (abort API)

For vLLM, use engine.abort() to stop processing: 
async for request_output in results_generator:
    if await request.is_disconnected():
        await engine.abort(request_id)
        return
See full example in vLLM Docs.

​
Unsupported request features

  • Streaming file uploads: Use URLs instead of embedding large data in the request.
  • Client-side headers: Most headers are stripped; include necessary metadata in the payload.