aynscio
-tasks that run concurrently.
That way you get the most out of the parallelism that Chains offers. This
design pattern is extensively used in the
audio transcription example.
While using
asyncio
is essential for performance, it can also be tricky.
Here are a few caveats to look out for:- Executing operations in an async function that block the event loop for
more than a fraction of a second. This hinders the “flow” of processing
requests concurrently and starting RPCs to other Chainlets. Ideally use
native async APIs. Frameworks like vLLM or triton server offer such APIs,
similarly file downloads can be made async and you might find
AsyncBatcher
useful. If there is no async support, consider running blocking code in a thread/process pool (as an attribute of a Chainlet). - Creating async tasks (e.g. with
asyncio.ensure_future
) does not start the task immediately. In particular, when starting several tasks in a loop,ensure_future
must be alternated with operations that yield to the event loop that, so the task can be started. If the loop is notasync for
or contains otherawait
statements, a “dummy” await can be added, for exampleawait asyncio.sleep(0)
. This allows the tasks to be started concurrently.