Skip to main content

View on GitHub

In this example, we deploy a dockerized model for infinity embedding server, a high-throughput, low-latency REST API server for serving vector embeddings.

​
Set up the config.yaml

To deploy a dockerized model, all you need is a config.yaml. It specifies how to build your Docker image, start the server, and manage resources. Let’s break down each section.

​
Base image

Sets the foundational Docker image to a lightweight Python 3.11 environment.
config.yaml

​
Docker server configuration

Configures the server’s startup command, health check endpoints, prediction endpoint, and the port on which the server will run.
config.yaml

​
Build commands (optional)

Pre-downloads model weights during the build phase to ensure the model is ready at container startup.
config.yaml

​
Configure resources

Note that we need an L4 to run this model.
config.yaml

​
Requirements

Lists the Python package dependencies required for the infinity embedding server.
config.yaml

​
Runtime settings

Sets the server to handle up to 40 concurrent inferences to manage load efficiently.
config.yaml

​
Environment variables

Defines essential environment variables including the Hugging Face access token, request batch size, queue size limit, and a flag to disable tracking.
config.yaml

​
Deploy dockerized model

Deploy the model like you would other Trusses, with:
docker_server configs deploy as published deployments and don’t support development mode. Using truss push --watch with a docker_server config returns an error. Use truss push without --watch to deploy to production.