Setting config.yaml
Set the behavior of your model server.
Truss is configurable to its core. Every Truss must include a file
config.yaml in its root directory, which is automatically generated when the Truss is created. However, configuration is optional. Every configurable value has a sensible default, and a completely empty config file is valid.description: An instruction-following LLM Using Evol-Instruct.
environment_variables: {}
model_metadata: {}
model_name: WizardLM
python_version: py39
requirements:
- torch==1.9.0
- peft==0.3.0
- sentencepiece==0.1.99
- accelerate==0.20.3
- bitsandbytes==0.39.0
- transformers==4.20.1
resources:
cpu: "3"
memory: 14Gi
use_gpu: true
accelerator: A10G
secrets: {}
system_packages: []
This reference covers the settings in
config.yaml that may need to be modified in the course of packaging a model as a Truss.YAML syntax can be a bit non-obvious when dealing with empty lists and dictionaries. You may notice the following in the default Truss config file:
requirements: []
secrets: {}
When you fill them in with values, lists and dictionaries should look like this:
requirements:
- dep1
- dep2
secrets:
key1: default_value1
key2: default_value2
Default:
nullDescribe your model for documentation purposes.
Default:
{}Do not store secret values directly in environment variables (or anywhere in the config file). See the section on secrets for information on properly managing secrets.
Any environment variables can be provided here as key value pairs and are exposed to the environment that the model executes in. Many Python libraries can be customized using environment variables, so this field can be quite handy in those scenarios.
environment_variables:
ENVIRONMENT: Staging
DB_URL: https://my_database.example.com/
Default:
{}Set any additional metadata in this catch-all field. The entire contents of the config file are available to the model at runtime, so this is a good place to store any custom information that model needs. For example, scikit-learn models include a flag here that indicates whether the model supports returning probabilities alongside predictions.
model_metadata:
supports_predict_proba: true
Default:
nullThe model's name, for documentation purposes.
Default:
py39Set the version of Python to use for serving the model. Currently supported Python version are:
py38-- Python 3.8py39-- Python 3.9py310-- Python 3.10py311-- Python 3.11
Default:
[]List the Python dependencies that the model depends on. The requirements should be provided in the pip requirements file format, but as a yaml list.
requirements:
- scikit-learn==1.0.2
- threadpoolctl==3.0.0
- joblib==1.1.0
- numpy==1.20.3
- scipy==1.7.3
YAML syntax can be a bit non-obvious when dealing with empty lists. You may notice the following in the default Truss config file:
requirements: []
When you fill them in with values, lists should look like this:
requirements:
- dep1
- dep2
Default:
{}Specify model server runtime resources such as CPU, RAM and GPU. For more, see model resource documentation.
resources:
cpu: "3"
memory: 14Gi
use_gpu: true
accelerator: A10G
Default:
{}This field can be used to specify the keys for such secrets and dummy default values -- Never store actual secret values in the config. Dummy default values are instructive of what the actual values look like and thus act as documentation of the format.
A model may depend on certain secret values that can't be bundled with the model and need to be bound securely at runtime. For example, a model may need to download information from s3 and may need access to AWS credentials for that.
Default:
[]Specify any system packages that you would typically install using
apt on a Debian operating system.system_packages:
- ffmpeg
- libsm6
- libxext6
Last modified 3mo ago