Skip to main content
baseten volume sync copies a remote source into a volume as a new sealed version. The sync runs as a durable background job on Baseten, so nothing downloads to your machine.

Start a sync

Pass a provider URI such as hf://<organization>/<repository> with --source and a bdn: volume reference with --dest. The destination is in the organization associated with your Baseten credentials. A tag on --dest tags the version the sync publishes. To copy a Hugging Face repository into a volume, run the following command:
The source’s URI scheme selects the provider: Repeat --include and --exclude with glob patterns to filter source-relative paths:
Without --wait, the command returns as soon as Baseten creates the job and prints its sync ID. With --wait, it follows the job until it’s READY, FAILED, or CANCELED, writes progress to stderr, and prints the result, including the new version’s reference, to stdout. Pressing Ctrl+C stops waiting without cancelling the sync, and prints the commands to check or cancel it.

Check a sync

Check the status of a sync using the ID returned when you started it:
A READY sync includes the reference of the version it published, and a FAILED sync includes an error code and message.

List syncs

List syncs for a destination volume:
The command shows syncs newest first. --dest matches one exact destination reference.

Cancel a sync

Cancel a sync using its ID:
Cancelling a sync that already finished returns it unchanged and doesn’t delete a version it published.

Fix a failed sync

When the source rejects a sync, Baseten stops the job instead of retrying it and reports one of these error codes. The message names the source URI and, when the provider returned one, its error code, such as NoSuchBucket or HTTP_404. Any other failure reports SYNC_FAILED. Baseten retries transient errors, such as throttling and timeouts, before it fails a sync.

Authenticate with a secret

Public sources need no credentials. For a private source, store its credentials as a Baseten secret in the same team, and pass the secret’s name with --auth-secret-name:
The secret’s value depends on the source: These are the same formats that weights sources use. Baseten Training sources authenticate automatically.

Authenticate with OIDC or AWS AssumeRole

Use short-lived credentials to read a private source. Pass the flags for one authentication method per sync.

AWS OIDC

Set up AWS OIDC, scope the trust to volume sync, and grant the role read access to the source bucket and prefix. Run the sync with the role ARN and AWS region.

GCP OIDC

Set up GCP OIDC, scope the trust to volume sync, and grant the service account read access to the source bucket and prefix. Run the sync with the service account and workload identity provider.

Hugging Face OIDC

Set up Hugging Face trust with the type claim set to volume_sync. Set --auth-hf-oidc-resource to the private repository ID or to the Hugging Face username with access to the gated repository.

AWS AssumeRole

Set up AWS AssumeRole with read access to the source bucket and prefix. Run the sync with the role ARN and region.
Confirm that the sync reaches READY and returns the new volume version’s reference. For all flags, see baseten volume sync.