Skip to main content
The training cache enables you to persist data between training jobs. This can significantly improve iteration speed by skipping expensive downloads and data transformations.

How to use the training cache

Set the cache configuration in your Runtime:

Cache directory

By default, the cache will be mounted in two locations:
  • /root/.cache/user_artifacts, which can be accessed through the $BT_PROJECT_CACHE_DIR environment variable. This cache is shared by all jobs in a project.
  • /root/.cache/team_artifacts, which can be accessed through the $BT_TEAM_CACHE_DIR environment variable. A team’s jobs on the same cluster share this cache. (See Cache scope.)

Cache scope

Each cache lives on one cluster, so jobs reuse cached data when they run on that cluster. When you enable caching, your project pins to the cluster where its first cache-enabled job ran. New projects prefer their team’s existing cache location for the same GPU type. By default (cache_config.require_cache_affinity=true), requesting a GPU type the pinned cluster doesn’t offer fails the job. Set require_cache_affinity=false to allow another cluster, where you may need to re-seed the cache.

Prepare data on CPUs for GPU jobs

For CPU jobs that prepare cached data for GPU jobs, set TrainingProject.project_accelerator to the GPU type that consumes the data, such as Accelerator.H100.

Hugging Face cache mount

You can mount your cache to the Hugging Face cache directory by setting HF_HOME to one of the provided mount points plus /huggingface. For example, you can set HF_HOME=$BT_PROJECT_CACHE_DIR/huggingface to use the project cache directory. However, there are considerable technical pitfalls when trying to read from the cache with multiple processes, as Hugging Face doesn’t work well with distributed filesystems. To help enable this use case, ensure your dataset processors or process count is set to 1 to minimize the number of concurrent readers.

Cache seeding

Seeding the cache once avoids repeating a large download on every training job. See Seed large datasets into the cache for the seed job and download script.

Cache management

Use baseten train project cache describe --project <project_name or project_id> to inspect cached files and remove artifacts you no longer need.
When you delete a project, all data in the project’s training cache ($BT_PROJECT_CACHE_DIR) is permanently deleted with no archival or recovery option. See Management for details.