Skip to main content

​
Deprecated

b10cache is deprecated. For model weight caching, use the new weights configuration which offers faster cold starts through multi-tier caching.For torch.compile caching, see Torch Compile Cache.

​
Early Access

Please contact our support team for access to b10cache.
Deployments sometimes have cache or other files that are useful to other replicas. Using torch.compile results in a cache that can speed up future torch.compile on the same function. This can speed up other replicas’ cold start times. These files can be stored via b10cache. b10cache is a volume mounted over the network onto each of your pods. There are two ways files can be stored:

​
1. /cache/org/

This directory is shared, and can be written to or accessed by every pod you deploy. Simply move a file into here and it will be accessible.

​
2. /cache/model/

This directory is shared by every pod within the scope of your deployment. This is excellent for keeping filesystems clean and limiting access.

​
Not a persistent object storage

While b10cache is very reliable, it should not be used as a persistent object storage or database. It should be considered a cache that can be shared by deployments, meaning there should always be a fallback plan if the b10cache path does not exist.
See two features built on b10cache:
  1. model cache
  2. torch compile cache