> ## Documentation Index
> Fetch the complete documentation index at: https://docs.baseten.co/llms.txt
> Use this file to discover all available pages before exploring further.

# Multinode training

> Learn how to configure and run multinode training jobs with Training Jobs.

Training Jobs supports multinode training over InfiniBand for distributed training across multiple nodes.

## Configure multinode training

**To deploy a multinode training job**:

Set `node_count` on the `Compute` resource in your `TrainingJob` to the number of nodes you'd like to use, for example 2:

```python theme={"system"}
from truss_train import definitions

compute = definitions.Compute(
    node_count=2,  # Use 2 nodes for multinode training
    # ... other compute configuration options
)
```

## Environment variables

Make sure you've properly integrated with the
[Baseten provided environment variables](/reference/sdk/training#baseten-provided-environment-variables)
for distributed training.

## Network configuration

Baseten provides high-speed InfiniBand networking between nodes to ensure efficient communication during distributed training. This enables:

* Fast gradient synchronization.
* Efficient parameter updates.
* Low-latency communication between nodes.

## Checkpointing in multinode training

Checkpointing behavior varies across training frameworks in multinode setups. One common pattern is to use the shared cache directory that all nodes can access:

```bash theme={"system"}
# Use shared volume with job name for checkpointing
ckpt_dir="${BT_PROJECT_CACHE_DIR}/${BT_TRAINING_JOB_NAME}"
```

Then ensure you write to `ckpt_dir`. This ensures all nodes write to the same checkpoint location. For comprehensive framework-specific examples and patterns, see the [Training Cookbook](https://github.com/basetenlabs/ml-cookbook).
Baseten doesn't back up these checkpoints because they live outside `$BT_CHECKPOINT_DIR`. Copy them there to preserve them.

## Common practices

When setting up multinode training:

* **Data loading**: Ensure your data loading is properly distributed across nodes.
* **Seeding**: Use consistent seeding across all nodes for reproducible results.
* **Monitoring**: Monitor training metrics across all nodes to ensure balanced training.

## Interactive multi-node clusters

For interactive multi-node work, `truss train workstation --node-count N` provisions the same multi-node compute as an SSH workstation with a Slurm cluster bootstrapped across the nodes. See [Slurm workstations](/training/slurm).