> ## Documentation Index
> Fetch the complete documentation index at: https://docs.baseten.co/llms.txt
> Use this file to discover all available pages before exploring further.

# Startup cache

> Reuse the kernels a deployment's first replica compiles so later replicas start without recompiling

Much of a model's cold start, especially an LLM's, goes to compiling kernels:
`torch.compile`, Triton and CUDA just-in-time (JIT) compilation, and
autotuning. The startup cache saves what a deployment's first replica compiles
and restores it on later replicas, so they skip that work without changes to
your model code or image.

> [!NOTE]
>
> The startup cache is in public preview and turned on for some organizations.
> To turn it on or off for your organization or a model, [Contact us](mailto:support@baseten.co).

## How the startup cache works

1. The first replica of a deployment starts cold and compiles as usual. After
   it passes its readiness check, Baseten saves the compile caches it wrote.
2. Later replicas restore those caches before the model starts, so the
   framework finds its kernels already compiled.

The cache is best-effort. If no cache exists yet or a restore fails, the
replica starts cold, as it would without the cache.

Baseten saves only what the first replica writes before it becomes ready. It
doesn't add kernels compiled later, during warmup after the readiness check or
under traffic, and it doesn't update a saved cache.

## When replicas reuse the cache

Each deployment keeps its own cache, keyed on its GPU type, GPUs per replica,
nodes per replica, and the Truss build it runs. A replica that matches the key
restores the cache. The first replica with a new key starts cold and saves a
new cache. Baseten saves CUDA driver JIT output separately for each NVIDIA
driver version, because the driver discards it when its version changes.

| Event | Result |
| - | - |
| Scale-up, scale from zero, or replica restart | Restores the cache |
| Promotion | Restores the source deployment's cache |
| Instance type change | Starts the first replica on the new GPU layout cold |
| Development deployment rebuild | Starts the first replica after the rebuild cold |
| New push | Starts with an empty cache |

## What gets cached

Baseten points these framework cache variables at the directory it saves:

| Variable | Caches |
| - | - |
| `VLLM_CACHE_ROOT` | vLLM `torch.compile` artifacts |
| `TORCHINDUCTOR_CACHE_DIR` | TorchInductor output |
| `TRITON_CACHE_DIR` | Triton kernels |
| `CUDA_CACHE_PATH` | CUDA driver JIT output |
| `TORCH_EXTENSIONS_DIR` | PyTorch C++ and CUDA extensions |
| `TLLM_AUTOTUNER_CACHE_PATH` | TensorRT-LLM autotuner results |
| `DG_JIT_CACHE_DIR` | DeepGEMM JIT kernels |
| `SGLANG_CACHE_DIR`, `SGLANG_JIT_CACHE_DIR`, `SGLANG_DG_CACHE_DIR`, `TILELANG_CACHE_DIR` | SGLang caches and JIT kernels |
| `FLASHINFER_WORKSPACE_BASE` | FlashInfer JIT kernels |
| `CUTE_DSL_CACHE_DIR` | CuTe DSL kernels |
| `NUMBA_CACHE_DIR` | Numba compiled functions |
| `JAX_COMPILATION_CACHE_DIR` | JAX compilation output |
| `MPLCONFIGDIR` | Matplotlib configuration and font cache |
| `COMPILE_CACHE_DIR`, `VQ_CACHE_DIR` | Fish Audio serving image caches |

A model that compiles nothing into these paths before it's ready, such as a
PyTorch model without `torch.compile`, has nothing to cache and starts cold
every time.

If you set one of these variables in `environment_variables`, your value wins
and Baseten doesn't cache that path. For example, a model that caches
`torch.compile` output in [b10cache](https://docs.baseten.co/development/model/runtime-caching.md) with
`TORCHINDUCTOR_CACHE_DIR: /cache/org/inductor` keeps using b10cache for it. To
let the startup cache handle that path, remove the variable from your config.

The startup cache isn't shared writable storage. Replicas can't write files for
each other through it, and it saves only the paths in the preceding table. For files your
application downloads at runtime, such as per-request data from object
storage, use [BDN proxy](https://docs.baseten.co/development/model/bdn.md#bdn-proxy) or
[attach a BDN volume while the deployment runs](https://docs.baseten.co/development/model/bdn.md#hotload-attach-bdn-volumes-while-a-deployment-runs).

The startup cache needs no code and covers one deployment. b10cache's
organization scope shares files across deployments, so it can also warm the
first replica of a new push.

## Supported deployments

The startup cache works with:

* Single-node Truss deployments, including custom servers, with an HTTP or
  gRPC readiness check.
* Baseten Inference Stack (BIS) engine deployments, including multi-node ones.

Multi-node Truss deployments and other unsupported deployments deploy without
the cache instead of failing.

## Apply the cache to existing deployments

The startup cache needs no `config.yaml` setting or code change; Baseten turns
it on for your organization or model. Baseten applies the setting when it
creates a deployment, on a new push or a promotion. To use the cache with a model whose deployments
predate the setting, push or promote a new deployment.

For more information, see [Runtime caching](https://docs.baseten.co/development/model/runtime-caching.md)
and [BDN](https://docs.baseten.co/development/model/bdn.md).