InfiniBand Interconnects
What is InfiniBand?
InfiniBand (IB) is a high-speed, low-latency network interconnect standard designed specifically for HPC and data center environments. Unlike Ethernet, which was designed for general-purpose networking, InfiniBand was built from the ground up for deterministic latency, high bandwidth, and RDMA — direct memory-to-memory transfers without CPU involvement.
InfiniBand is used in more than 60% of TOP500 systems. After NVIDIA acquired Mellanox in 2020, it became the dominant vendor for both the HPC adapters (ConnectX series) and switches (Quantum series).
InfiniBand achieves ~600 nanoseconds latency vs ~5 microseconds for high-speed Ethernet. At scale — thousands of nodes doing MPI_Allreduce — this 8× latency advantage compounds into significantly faster iteration times for both simulation and AI training.
Generations and Speeds
Speeds are per-port. Modern HPC nodes typically use a single HDR (200 Gb/s) or NDR (400 Gb/s) port. Switches aggregate many ports — a Quantum-2 NDR switch provides 64 × 400 Gb/s ports = 25.6 Tb/s total switching capacity.
RDMA — The Core Advantage
Remote Direct Memory Access (RDMA) allows a process on one node to read or write data directly into the memory of another node, bypassing both operating systems and CPUs entirely. This eliminates kernel overhead and memory copies, delivering:
- Latency: ~600 ns (vs ~5 µs TCP/IP)
- CPU overhead: near zero (vs significant for TCP)
- Bandwidth: full line rate utilization
InfiniBand supports three RDMA verbs: RDMA Write, RDMA Read, and Send/Receive. MPI implementations (MVAPICH2, OpenMPI with UCX) use these natively for all inter-node communication.
GPUDirect RDMA
GPUDirect RDMA extends RDMA to GPU memory — enabling a network adapter to transfer data directly from one GPU's HBM memory to another GPU's HBM memory on a remote node, without touching CPU DRAM at all.
| Transfer path | Latency | Bandwidth | CPU involvement |
|---|---|---|---|
| GPU → CPU DRAM → Network → CPU DRAM → GPU | ~50 µs | ~10 GB/s | High |
| GPU → Network → GPU (GPUDirect RDMA) | ~5 µs | ~25 GB/s | None |
For large-scale AI training with frequent gradient synchronization across thousands of GPUs, GPUDirect RDMA is not optional — it is the difference between practical and impractical training times.
Network Topology: Fat-Tree
The dominant topology in InfiniBand HPC clusters is the fat-tree. In a fat-tree, bandwidth increases toward the root of the tree — ensuring that any two nodes in the cluster can communicate at full bisection bandwidth.
A two-level fat-tree (leaf + spine switches) is common for clusters up to ~1,000 nodes. Three-level fat-trees scale to tens of thousands of nodes — used in systems like Perlmutter and LUMI.
Fat-tree provides full bisection bandwidth — any communication pattern runs at full speed. Dragonfly (used in Slingshot/Cray systems) uses fewer cables and switches but relies on adaptive routing to handle adversarial traffic patterns. Fat-tree is simpler and more predictable; Dragonfly scales better to very large systems.
Key Hardware Components
| Component | Example products | Role |
|---|---|---|
| Host Channel Adapter (HCA) | ConnectX-7 (NDR), ConnectX-6 (HDR) | PCIe card in server — provides IB ports and RDMA engine |
| Switch | Quantum-2 (NDR, 64-port), Quantum (HDR, 40-port) | Interconnects nodes and other switches |
| Cable | Active Optical Cable (AOC), DAC copper | Connects HCAs to switches — AOC for distances >3m |
| Subnet Manager | OpenSM (open source), MLNX-UFM | Manages fabric routing and QoS policies |
InfiniBand vs High-Speed Ethernet
| Property | InfiniBand NDR | 400GbE |
|---|---|---|
| Latency | ~600 ns | ~3–5 µs |
| Bandwidth (per port) | 400 Gb/s | 400 Gb/s |
| RDMA protocol | Native IB verbs | RoCEv2 (RDMA over Ethernet) |
| Congestion control | Built-in, deterministic | Requires careful DCQCN tuning |
| Ecosystem | HPC-specific, mature | Broader, converged with storage/cloud |
| Cost | Higher | Lower (commodity) |
| Best for | Latency-sensitive MPI, tightly coupled HPC | AI training at scale, cloud HPC, storage |
Key Takeaways
- InfiniBand dominates HPC due to sub-microsecond latency and native RDMA
- NDR (400 Gb/s) is the current standard; XDR (800 Gb/s) is emerging for 2025+ systems
- GPUDirect RDMA is essential for high-performance AI training clusters
- Fat-tree topology provides full bisection bandwidth — every node can communicate at full speed
- For cost-sensitive deployments, RoCEv2 over 400GbE is a viable alternative, especially for AI training workloads less sensitive to latency