How to use base images
A guide to configuring a base image for your truss
Model serving enviroments will often be standardized as container images to avoid wrangling python, system, and other requirements needed to run your model on every deploy. Leverage your existing container artifacts by bringing your own base image to Truss.
While the default image for the truss server in adequate for most use cases, there may come a time when you require a custom base image for your truss. For example, maybe the python packages required for you project are not compatible with the ones installed by default on Truss. In a situation like this, you can create your own base image based on the default truss image.
Setting a base image in config.yaml
To specify a base image to build a truss container image from in your config.yaml
configure a base_image
.
base_image:
image: <image_name:tag>
python_executable_path: <path-to-python>
where python_executable_path
is a path to a Python executable with which to run your server.
Example usage
One great use case for base images is to more easily package and deploy models from sources like the NeMo Toolkit without having to recreate all of the dependencies yourself.
This example demonstrates how to properly configure a base image for NVIDIA NeMo TitaNet, a speaker recognition model from NVIDIA.
base_image:
image: nvcr.io/nvidia/nemo:23.03
python_executable_path: /usr/bin/python
apply_library_patches: true
requirements:
- PySoundFile
resources:
accelerator: T4
cpu: 2500m
memory: 4512Mi
use_gpu: true
secrets: {}
system_packages:
- python3.8-venv
Configuring private base images with build time secrets
Secrets of the form DOCKER_REGISTRY_<REGISTRY_URL>
will be supplied to your model build to authenticate image pulls from private container registries.
For information on where to store secret values see the secrets guide.
For example, to configure docker credentials to a private dockerhub repository your config.yaml
should include the following secret and placeholder:
secrets:
DOCKER_REGISTRY_https://index.docker.io/v1/: null
along with a configured Baseten secret DOCKER_REGISTRY_https://index.docker.io/v1/
with a base64 encoded username:password
secret value:
echo -n 'username:password' | base64
To add docker credentials for Google Cloud Artifact Registry provide an access token as the secret value.
For example, to configure authentication for a repository in us-west2
your config.yaml
should include the following secret and placeholder:
secrets:
DOCKER_REGISTRY_us-west2-docker.pkg.dev: null
Note that since access tokens are short-lived, you may want to leverage the Baseten Secrets API to automatically check for expired access tokens before every deploy, and update them as necessary.
If you donβt have a base image but want to create one, the following section will guide you through that process.
Creating a custom base image
While itβs possible to create your own base image from scratch, it may be easier to use a truss server image as a starting point.
All of the base images used by truss can be found by going to docker hub.
Each image tag is tied to a specific python version and may support a GPU. For example, the image with the tag 3.11-gpu-v0.7.16
is for Python 3.11 and has GPU support.
On the other hand, image 3.9-v0.7.15
is for Python 3.9 and does not have GPU support. Based on your project requirements you can select the appropriate base image.
Next, we can write our Dockerfile
.
FROM baseten/truss-server-base:3.11-gpu-v0.7.16
RUN pip uninstall cython -y
RUN pip install cython==0.29.30
In the example dockerfile above, we use the truss-server-base
as our base image. This base image comes with some python dependencies installed.
If you want to override the versions of these default python dependencies you can simply uninstall the package and reinstall it using the RUN pip install
command.
You can even add you own files or directories to this Dockerfile if required. Once you have your Dockerfile set up, you can build, tag, and push the image to your own docker registry.
Build, tag, and push your custom base image
Docker installation required
For this portion, docker needs to be installed and running on your system. Additionally, you will need docker installed on your command line.
To build the image run the command:
docker build -t my-custom-base-image:0.1 .
You can replace my-custom-base-image
with the name for your docker image.
Next, to tag the image you can run the command:
docker tag my-custom-base-image:0.1 your-docker-username/my-custom-base-image:0.1
Lastly, to push the image to docker hub run the following command:
docker push your-docker-username/my-custom-base-image:0.1