Tracking model metadata
Use artifacts, configs, and metrics to store and interact with model metadata.
Once you've started using Baseten for model deployment, you might wish to associate pieces of information with models that are deployed in the Baseten infrastructure. We provide three pieces of functionality in our Python client to support model metadata information: artifacts, configs, and metrics.
Artifacts are effectively general purpose BLOB storage. They can be created and associated to models and model versions. Constructing an artifact entails giving it a name and a list of files. The files will be compressed and stored and can be subsequently retrieved. The artifact can be associated with a model as a whole (using
model_id) or a particular version of a model (using model_version_id).import baseten
model = baseten.deployed_model_version_id(model_version_id='a12b34c')
# artifacts can be associated directly on the model object
model.add_artifact(
name='Train/Test Data'
list_of_files=['data/train.h5', 'data/test.h5', 'README.txt'],
description='All the files needed for model reproduction'
)
# or an artifact can be created with just a model (or model version) ID
artifact = baseten.create_artifact(
name='Model binary',
list_of_files=['model.joblib'],
description='Model object stored with joblib',
model_version_id=model_version_id
)
# artifacts can be linked to many models
artifact.create_link(model_version_id='a_different_model_version_id')
Once created and uploaded, interacting with artifacts and their associated models is easy.
# a presigned URL to download the artifact's .tgz
artifact_url = artifact.url()
# extracts the artifact binary into the given directory
artifact.download(target_directory='model_binary_download/')
# update the artifact with new data
artifact.update(
description='Model object stored as h5',
list_of_files=['model.h5'],
name='Model h5 binary'
)
# see what models are linked to this artifact
artifact.links()
# or see what artifacts are linked to a model
model.artifacts()
# delete the artifact and it's data
artifact.delete()
The
url method on the artifact will give the end user a presigned URL to download and manage the archive created by the upload. The update method will replace all the existing artifact data with the data provided. The delete method will destroy the artifact and delete all the data associated with it. Use the methods carefully.Baseten provides an API for storing model configuration. This feature will store JSON serializable data and provide access to that information. As such it is a good place to store information such as hyperparameters and other configuration information to track with particular models.
model = baseten.deployed_model_version_id(model_version_id='a12b34c')
# a sample configuration
model.log_config({
'batch_size': 16,
'layers': 64,
'epochs': 20,
'learning_rate': 0.0001
})
# the latest configuration associated with the model
model.get_config()
Model metrics and other offline evaluation metrics can also be stored via JSON serialization. To track evaluation metrics with a deployed model pass a JSON serializable object to the
log_metrics method in the client library.model = baseten.deployed_model_version_id(model_version_id='a12b34c')
# a sample set of metrics to track
model.log_metrics({
'accuracy': 94.5,
'log_loss': 0.222222
})
# the latest metrics associated with the model
model.get_metrics()
To facilitate use of these features we provide a high level solution for listing your models and their versions. Simply call
models_summary() from the client library and observe a list of dictionaries that describe the models and model versions you currently have deployed on Baseten.import baseten
baseten.models_summary()
# sample response
[{'id': 'YNBEDqa',
'name': 'Iris Random Forest Classifier',
'description': 'Predict Iris class with Random Forest Classifier',
'versions': [{'id': 'prwny3y',
'created': '2022-02-16T01:05:35.563227+00:00',
'configuration': {'sha': 'foobarbaz'}}]}]
Last modified 7mo ago