Model with system packages
Deploy a model with both Python and system dependencies
View on Github
In this example, we build a Truss with a model that requires specific system packages.
To add system packages to your Truss, you can add a system_packages
key to your config.yaml file,
for instance:
To add system packages to your model serving environment, open config.yaml and
update the system_packages key with a list of apt-installable Debian packages:
system_packages:
- tesseract-ocr
For this example, we use the LayoutLM Document QA model, a multimodal model that answers questions about provided invoice documents. This model requires a system package, tesseract-ocr, which needs to be included in the model serving environment.
Setting up the model.py
For this model, we use the HuggingFace transformers library, and the document-question-answering task.
from transformers import pipeline
class Model:
def __init__(self, **kwargs) -> None:
self._model = None
def load(self):
self._model = pipeline(
"document-question-answering",
model="impira/layoutlm-document-qa",
)
def predict(self, model_input):
return self._model(model_input["url"], model_input["prompt"])
Setting up the config.yaml file
The main items that need to be configured in the config.yaml file are requirements
and system_packages
sections.
environment_variables: {}
external_package_dirs: []
model_metadata:
example_model_input: {"url": "https://templates.invoicehome.com/invoice-template-us-neat-750px.png", "prompt": "What is the invoice number?"}
model_name: LayoutLM Document QA
python_version: py39
Specify the versions of the Python requirements that are needed.
requirements:
- Pillow==10.0.0
- pytesseract==0.3.10
- torch==2.0.1
- transformers==4.30.2
resources:
cpu: "4"
memory: 16Gi
use_gpu: false
accelerator: null
secrets: {}
The system_packages section is the other important bit here, you can
add any package thatβs available via apt
on Debian.
system_packages:
- tesseract-ocr
Deploy the model
$ truss push
You can then invoke the model with:
$ truss predict -d '{"url": "https://templates.invoicehome.com/invoice-template-us-neat-750px.png", "prompt": "What is the invoice number?"}'