Startup cache

Much of a model’s cold start, especially an LLM’s, goes to compiling kernels: torch.compile, Triton and CUDA just-in-time (JIT) compilation, and autotuning. The startup cache saves what a deployment’s first replica compiles and restores it on later replicas, so they skip that work without changes to your model code or image.

​

How the startup cache works

1. The first replica of a deployment starts cold and compiles as usual. After it passes its readiness check, Baseten saves the compile caches it wrote.
2. Later replicas restore those caches before the model starts, so the framework finds its kernels already compiled.

The cache is best-effort. If no cache exists yet or a restore fails, the replica starts cold, as it would without the cache. Baseten saves only what the first replica writes before it becomes ready. It doesn’t add kernels compiled later, during warmup after the readiness check or under traffic, and it doesn’t update a saved cache.

​

When replicas reuse the cache

Each deployment keeps its own cache, keyed on its GPU type, GPUs per replica, nodes per replica, and the Truss build it runs. A replica that matches the key restores the cache. The first replica with a new key starts cold and saves a new cache. Baseten saves CUDA driver JIT output separately for each NVIDIA driver version, because the driver discards it when its version changes.

​

What gets cached

Baseten points these framework cache variables at the directory it saves: A model that compiles nothing into these paths before it’s ready, such as a PyTorch model without torch.compile, has nothing to cache and starts cold every time. If you set one of these variables in environment_variables, your value wins and Baseten doesn’t cache that path. For example, a model that caches torch.compile output in b10cache with TORCHINDUCTOR_CACHE_DIR: /cache/org/inductor keeps using b10cache for it. To let the startup cache handle that path, remove the variable from your config. The startup cache isn’t shared writable storage. Replicas can’t write files for each other through it, and it saves only the paths in the preceding table. For files your application downloads at runtime, such as per-request data from object storage, use BDN proxy or attach a BDN volume while the deployment runs. The startup cache needs no code and covers one deployment. b10cache’s organization scope shares files across deployments, so it can also warm the first replica of a new push.

​

Supported deployments

The startup cache works with:

• Single-node Truss deployments, including custom servers, with an HTTP or gRPC readiness check.
• Baseten Inference Stack (BIS) engine deployments, including multi-node ones.

Multi-node Truss deployments and other unsupported deployments deploy without the cache instead of failing.

​

Apply the cache to existing deployments

The startup cache needs no config.yaml setting or code change; Baseten turns it on for your organization or model. Baseten applies the setting when it creates a deployment, on a new push or a promotion. To use the cache with a model whose deployments predate the setting, push or promote a new deployment. For more information, see Runtime caching and BDN.