Truss integration

Chains can be combined with existing Truss models using Stubs. A Stub acts as a substitute (client-side proxy) for a remotely deployed dependency, either a Chainlet or a Truss model. The Stub performs the remote invocations as if it were local by taking care of the transport layer, authentication, data serialization and retries. Stubs can be integrated into Chainlets by passing in a URL of the deployed model. They also require context to be initialized (for authentication). The following Chainlet wraps a deployed model with a Stub:

my_chainlet.py

import truss_chains as chains


class LLMClient(chains.StubBase):

    async def run_remote(self, prompt: str) -> str:
        # Call the deployed model
        resp = await self.predict_async(inputs={
            "messages": [{"role": "user", "content": prompt}],
            "stream"  : False
        })
        # Return a string with the model output
        return resp["output"]


LLM_URL = ...


class MyChainlet(chains.ChainletBase):

    def __init__(
        self,
        context: chains.DeploymentContext = chains.depends_context(),
    ):
        self._llm = LLMClient.from_url(LLM_URL, context)

There are various ways how you can make a call to the other deployment:

Input as JSON dict (like above) or pydantic model.
Automatic parsing of the response into a pydantic model using the output_model argument.
predict_async (recommended) or predict_sync.
Streaming responses using predict_async_stream which returns an async bytes iterator.
Customized with RPCOptions.

See the StubBase reference for all APIs.

TrussChainlet and TrussHandle

You can integrate existing Truss models directly into a chain without rewriting them as ChainletBase subclasses. Use TrussChainlet to wrap a Truss directory and TrussHandle to manage connections to it. TrussChainlet wraps an existing Truss directory as a non-entry leaf chainlet. This is useful for including models that use custom servers (like vLLM) or existing model.py implementations.

my_chainlet.py

import truss_chains as chains


class STT(chains.TrussChainlet):
    truss_dir = "./a_truss_model"

To call a TrussChainlet from another chainlet, use chains.depends() as a default argument in __init__. At runtime this provides a TrussHandle, which supplies the arguments for raw HTTP or WebSocket calls. TrussChainlet cannot be used as an entrypoint and cannot declare its own dependencies.

my_chainlet.py

from truss_chains.remote_chainlet.truss_chainlet import TrussHandle


class MyChainlet(chains.ChainletBase):

    def __init__(self, stt: TrussHandle = chains.depends(STT)):
        self._stt = stt

    async def run_remote(self, audio_data: bytes) -> str:
        # Get HTTP call arguments
        url, headers = self._stt.http_call_args()
        # Use url and headers with your preferred HTTP client
        ...

TrussHandle supports Bring Your Own Client (BYOC) scenarios by exposing raw connection details:

http_call_args(): Returns the URL and headers for HTTP requests. Use prefer_internal=True to use the workload-plane URL with the correct Host header, or sync_path to rewrite the URL for platform passthrough endpoints.
ws_call_args(): Returns the URL and headers for WebSocket connections.

Overview

Get started

Model APIs

Inference

Development

Deployment

Engines

Frontier Gateway

Training

Organization

Observability

Troubleshooting

Truss integration

TrussChainlet and TrussHandle

​TrussChainlet and TrussHandle

TrussChainlet and TrussHandle