baseten volume sync copies a remote source into a volume as a new sealed version. The sync runs as a durable background job on Baseten, so nothing downloads to your machine.
Start a sync
Pass a provider URI such ashf://<organization>/<repository> with --source and a bdn: volume reference with --dest. The destination is in the organization associated with your Baseten credentials. A tag on --dest tags the version the sync publishes.
To copy a Hugging Face repository into a volume, run the following command:
Repeat
--include and --exclude with glob patterns to filter source-relative paths:
--wait, the command returns as soon as Baseten creates the job and prints its sync ID. With --wait, it follows the job until it’s READY, FAILED, or CANCELED, writes progress to stderr, and prints the result, including the new version’s reference, to stdout. Pressing Ctrl+C stops waiting without cancelling the sync, and prints the commands to check or cancel it.
Check a sync
Check the status of a sync using the ID returned when you started it:READY sync includes the reference of the version it published, and a FAILED sync includes an error code and message.
List syncs
List syncs for a destination volume:--dest matches one exact destination reference.
Cancel a sync
Cancel a sync using its ID:Fix a failed sync
When the source rejects a sync, Baseten stops the job instead of retrying it and reports one of these error codes. The message names the source URI and, when the provider returned one, its error code, such asNoSuchBucket or HTTP_404.
Any other failure reports
SYNC_FAILED. Baseten retries transient errors, such as throttling and timeouts, before it fails a sync.
Authenticate with a secret
Public sources need no credentials. For a private source, store its credentials as a Baseten secret in the same team, and pass the secret’s name with--auth-secret-name:
These are the same formats that
weights sources use. Baseten Training sources authenticate automatically.
Authenticate with OIDC or AWS AssumeRole
Use short-lived credentials to read a private source. Pass the flags for one authentication method per sync.AWS OIDC
Set up AWS OIDC, scope the trust to volume sync, and grant the role read access to the source bucket and prefix. Run the sync with the role ARN and AWS region.GCP OIDC
Set up GCP OIDC, scope the trust to volume sync, and grant the service account read access to the source bucket and prefix. Run the sync with the service account and workload identity provider.Hugging Face OIDC
Set up Hugging Face trust with thetype claim set to volume_sync. Set --auth-hf-oidc-resource to the private repository ID or to the Hugging Face username with access to the gated repository.
AWS AssumeRole
Set up AWS AssumeRole with read access to the source bucket and prefix. Run the sync with the role ARN and region.READY and returns the new volume version’s reference. For all flags, see baseten volume sync.