// Deep Dive · HPC Programming

MPI — Message Passing Interface

📅 Updated: June 2026 ⏱ 10 min read 🏷 Parallel Programming · Distributed Memory · Supercomputing

What is MPI?

The Message Passing Interface (MPI) is the de facto standard for parallel programming on distributed-memory systems — meaning HPC clusters where each compute node has its own private memory. Processes running on different nodes communicate by explicitly sending and receiving messages over the network.

MPI is not a language or a compiler — it is a library specification defining over 400 functions. Programs written in C, C++, or Fortran call MPI routines to coordinate parallel work. The first standard (MPI-1) was released in 1994; the current standard is MPI-4.1 (2023).

// Why MPI dominates HPC

Every major supercomputer on the TOP500 runs MPI workloads. Its explicit communication model gives programmers precise control over data movement — critical when squeezing performance from 10,000+ node systems where even microseconds of latency matter.

The Core Concept: Ranks and Communicators

When an MPI program launches, it starts N processes simultaneously. Each process gets a unique integer identifier called a rank (0 to N-1). Processes are organized into communicators — groups that can exchange messages. The default communicator MPI_COMM_WORLD contains all processes.

// Minimal MPI program — every process prints its rank
#include <mpi.h>
#include <stdio.h>

int main(int argc, char** argv) {
    MPI_Init(&argc, &argv);           // Initialize MPI

    int rank, size;
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);  // My rank (0..N-1)
    MPI_Comm_size(MPI_COMM_WORLD, &size);  // Total processes

    printf("Hello from rank %d of %d\n", rank, size);

    MPI_Finalize();                   // Clean up MPI
    return 0;
}
  

Run on 4 nodes with 32 processes each: mpirun -np 128 ./hello — 128 independent processes each knowing their own rank and the total count.

Point-to-Point Communication

The fundamental MPI operations are send and receive between pairs of processes:

Function	Type	Behavior
MPI_Send	Blocking	Returns when buffer can be reused — may buffer internally
MPI_Recv	Blocking	Returns only when message is fully received
MPI_Isend	Non-blocking	Returns immediately; use MPI_Wait to confirm completion
MPI_Irecv	Non-blocking	Posts receive buffer; use MPI_Wait to confirm arrival
MPI_Sendrecv	Blocking	Send and receive simultaneously — avoids deadlock in ring patterns

// Deadlock warning

If process 0 calls MPI_Send to process 1 and process 1 calls MPI_Send to process 0 simultaneously — both block waiting for a receive that never comes. Always use non-blocking or MPI_Sendrecv for bidirectional exchanges.

Collective Operations

Collectives coordinate all processes in a communicator simultaneously — they are the backbone of parallel algorithms. Every process must call the collective, making them implicit synchronization points.

MPI_Bcast

One process sends identical data to all others. Used to distribute parameters, configuration, or model weights.

MPI_Scatter

Root distributes different chunks of an array to each process. Essential for data parallelism.

MPI_Gather

All processes send data to root, which assembles the complete result.

MPI_Reduce

Combines values from all processes (sum, max, min, etc.) into a single result on root.

MPI_Allreduce

Reduce + broadcast to all — the most critical collective for distributed AI training (gradient averaging).

MPI_Alltoall

Every process sends unique data to every other process — used in FFTs and matrix transposes.

MPI_Barrier

Synchronization point — all processes wait until every process has reached the barrier.

MPI_Scan

Prefix reduction — process i receives the reduction of values from ranks 0 through i.

MPI_Allreduce in AI Training

MPI_Allreduce is arguably the single most important HPC communication pattern for modern AI. During distributed training, each GPU computes gradients on its local data batch. Before the weight update, gradients must be averaged across all GPUs — this is an Allreduce operation. At scale (thousands of GPUs), the efficiency of this operation directly determines training throughput.

// Distributed gradient averaging — simplified
double local_gradient = compute_gradient(my_batch);
double global_gradient;

MPI_Allreduce(
    &local_gradient,    // Send buffer
    &global_gradient,   // Receive buffer
    1,                  // Count
    MPI_DOUBLE,         // Datatype
    MPI_SUM,            // Operation
    MPI_COMM_WORLD      // Communicator
);
global_gradient /= num_processes;
update_weights(global_gradient);
  

MPI Implementations

Implementation	Maintained by	Key strengths	Common use
OpenMPI	Open-source consortium	Broad hardware support, modular, excellent InfiniBand	Most university/research clusters
MPICH	Argonne National Lab	Reference implementation, tight MPI standard conformance	National labs, basis for other MPIs
Intel MPI	Intel (oneAPI)	Highly optimized for Intel Xeon, Omni-Path fabric	Intel-based enterprise HPC
MVAPICH2	Ohio State University	Best-in-class InfiniBand/RDMA optimization	InfiniBand clusters, GPU-aware MPI
Cray MPICH	HPE/Cray	Optimized for Slingshot interconnect	Frontier, Perlmutter, HPE Cray systems

Hybrid MPI + OpenMP

Modern HPC nodes have 64–256 CPU cores sharing memory. Running one MPI process per core wastes memory (each process has independent address space and MPI overhead). The best practice is hybrid programming: one MPI process per socket or per node, with OpenMP threads filling the cores.

#!/bin/bash — SLURM hybrid job script
#SBATCH --nodes=16
#SBATCH --ntasks-per-node=2      # 2 MPI ranks per node (1 per socket)
#SBATCH --cpus-per-task=64       # 64 OpenMP threads per rank

export OMP_NUM_THREADS=64
export OMP_PLACES=cores
export OMP_PROC_BIND=close

mpirun ./simulation
  

This launches 32 MPI processes total across 16 nodes, each with 64 threads — fully utilizing a dual-socket node with 128 cores while minimizing MPI overhead.

GPU-Aware MPI

Traditional MPI requires data to be in CPU memory before sending. GPU-aware MPI (supported by MVAPICH2, OpenMPI with UCX, Cray MPICH) allows direct GPU-to-GPU transfers over InfiniBand using GPUDirect RDMA — bypassing the CPU entirely and dramatically reducing latency for AI training communication.

// Performance impact

GPUDirect RDMA reduces the latency of GPU-to-GPU transfers across nodes from ~50µs (via host memory) to ~5µs, and increases bandwidth from ~10 GB/s to the full InfiniBand line rate (~25 GB/s for HDR). For large-scale AI training, this translates directly into faster iteration times.

MPI-4: What's New (2021–2023)

Persistent collectives — pre-initialize collective communication patterns, then trigger them repeatedly with reduced overhead
Large count support — counts now use MPI_Count (64-bit) instead of int, enabling transfers larger than 2 billion elements
Partitioned communication — send/receive large messages in parts as they become ready
MPI_T tool interface improvements — better performance monitoring and tuning APIs
Sessions model — alternative to MPI_Init enabling more flexible initialization in library contexts

Performance Tuning Tips

Use non-blocking collectives (MPI_Iallreduce) to overlap computation with communication
Minimize synchronization points — every MPI_Barrier stalls all processes
Tune message sizes — small messages (<4KB) are latency-bound; large messages are bandwidth-bound. Match your algorithm accordingly
Enable RDMA — ensure your MPI installation uses the InfiniBand/UCX transport, not TCP fallback
Pin processes to NUMA nodes — use --bind-to socket in mpirun to avoid cross-NUMA memory access
Profile with tools — Intel Trace Analyzer, Vampir, or Score-P to find communication hotspots

Key Takeaways

MPI is the universal language of distributed HPC — every major supercomputer runs it
The rank/communicator model gives explicit control over all inter-process communication
Collective operations — especially MPI_Allreduce — are performance-critical for both scientific simulation and AI training
Hybrid MPI/OpenMP is the recommended programming model for modern multi-core, multi-node clusters
GPU-aware MPI with GPUDirect RDMA is essential for high-performance AI training clusters