GPU Computing in HPC
Why GPUs Dominate HPC
GPUs were originally designed for rendering graphics — but their massively parallel architecture (thousands of cores operating simultaneously) makes them ideal for the matrix multiplication operations at the heart of both scientific simulation and AI training. A modern NVIDIA H100 delivers over 2,000 TFLOPS of FP16 tensor core performance — roughly 60× more than a high-end CPU for AI workloads.
Since 2012, GPU computing has transformed HPC. Today, all top 10 supercomputers on the TOP500 are GPU-accelerated. The shift from CPU-only to GPU-accelerated HPC is complete.
NVIDIA H100 SXM: 3,958 TFLOPS FP16 Tensor Core, 3.35 TB/s HBM3 memory bandwidth, 80 GB HBM3. AMD MI300X: 5,220 TFLOPS FP16, 5.3 TB/s HBM3, 192 GB HBM3. Compare to Intel Xeon Platinum 8592+: ~8 TFLOPS FP16. The GPU advantage is 500–650×.
H100 vs MI300X: The Current Generation
| Spec | NVIDIA H100 SXM | AMD MI300X |
|---|---|---|
| Architecture | Hopper (GH100) | CDNA3 |
| FP16 Tensor TFLOPS | 3,958 | 5,220 |
| FP8 TFLOPS | 7,916 | 10,440 |
| HBM capacity | 80 GB HBM3 | 192 GB HBM3e |
| Memory bandwidth | 3.35 TB/s | 5.3 TB/s |
| TDP | 700 W | 750 W |
| GPU interconnect | NVLink 4.0 (900 GB/s) | Infinity Fabric (896 GB/s) |
| Software ecosystem | CUDA (dominant) | ROCm (growing) |
| Best for | AI training/inference, broad HPC | Large model training (192 GB fits bigger models) |
CUDA vs ROCm
The software ecosystem is the critical differentiator — not just the hardware specs.
CUDA (NVIDIA): 15+ years of development, the world's largest GPU computing software ecosystem. Virtually every AI framework (PyTorch, TensorFlow, JAX), HPC library (cuBLAS, cuFFT, NCCL), and tool has native CUDA support. Closed-source but free to use.
ROCm (AMD): Open-source GPU computing platform with HIP (CUDA-compatible API). PyTorch and TensorFlow have ROCm support. Growing rapidly — Frontier runs entirely on ROCm. The main challenge is ecosystem maturity and the long tail of CUDA-only libraries.
If your code or framework is CUDA-based and you have a choice, NVIDIA wins on ecosystem compatibility. For new greenfield deployments focused on large model training, MI300X's 192 GB memory makes it compelling — fitting LLaMA-70B in a single GPU vs requiring model parallelism on H100.
Tensor Cores and Precision
Modern AI training rarely uses FP64 (double precision). The key precision formats and their use cases:
| Precision | Bits | H100 TFLOPS | Use case |
|---|---|---|---|
| FP64 | 64 | 67 | Scientific simulation (CFD, molecular dynamics) |
| TF32 | 19 (10+8+1) | 989 | AI training default on NVIDIA (automatic) |
| BF16 | 16 | 1,979 | AI training (same range as FP32, less mantissa) |
| FP16 | 16 | 1,979 | AI training and inference |
| FP8 | 8 | 3,958 | AI inference, some training (Hopper+) |
| INT8 | 8 | 3,958 | AI inference quantization |
NVLink — GPU-to-GPU Interconnect
Within a single node, multiple GPUs must communicate fast. PCIe (the standard interface) provides ~64 GB/s — far too slow for tight GPU coupling. NVLink 4.0 (H100) provides 900 GB/s total bandwidth per GPU in an 8-GPU NVLink domain via the NVSwitch.
An 8× H100 SXM node with NVSwitch creates a fully connected GPU mesh — every GPU can communicate with every other GPU at full 900 GB/s simultaneously. This is why DGX H100 nodes are the standard building block for large AI training clusters.
GPU Cluster Architecture
A modern GPU HPC cluster has a clear hierarchy:
- Node level: 4–8 GPUs connected via NVLink/NVSwitch + 2–4 CPUs + HBM GPU memory + DRAM system memory
- Rack level: 8–16 nodes per rack, each node with a high-speed network adapter (InfiniBand NDR or 400GbE)
- Cluster level: Fat-tree InfiniBand fabric connecting all nodes; shared Lustre parallel filesystem; SLURM for job scheduling
For AI training specifically, the communication-to-compute ratio is critical. With tensor/pipeline/data parallelism across thousands of GPUs, the interconnect must keep up with gradient synchronization. This is why NDR InfiniBand (400 Gb/s) with GPUDirect RDMA is the standard for frontier AI clusters.
Programming Models
| Model | Vendor | Level | Best for |
|---|---|---|---|
| CUDA C/C++ | NVIDIA | Low (explicit kernels) | Maximum performance, custom algorithms |
| HIP | AMD | Low (CUDA-compatible) | Portable code across NVIDIA/AMD |
| OpenCL | Khronos | Low (portable) | Cross-vendor portability (less common in HPC) |
| OpenACC | PGI/NVIDIA | High (directives) | Porting existing CPU code to GPU with minimal changes |
| OpenMP target | Standard | High (directives) | Portable GPU offload in Fortran/C++ |
| PyTorch/JAX | Meta/Google | Very high (framework) | AI training and ML workloads |
Key Performance Considerations
- Memory bandwidth is often the bottleneck — most HPC kernels are memory-bound, not compute-bound. Maximize data reuse and minimize HBM accesses
- Occupancy matters — fill the GPU with enough warps to hide memory latency
- Minimize CPU-GPU data transfers — PCIe transfers (16 GB/s) are 200× slower than HBM bandwidth
- Use libraries — cuBLAS, cuFFT, cuDNN, NCCL are already tuned; custom kernels rarely beat them for standard operations
- Profile before optimizing — use NVIDIA Nsight Systems and Nsight Compute to find actual bottlenecks
Key Takeaways
- GPUs deliver 500–650× the AI throughput of CPUs — all top-10 supercomputers are GPU-accelerated
- H100 and MI300X are the current generation — MI300X wins on memory capacity, H100 on ecosystem
- NVLink within a node and InfiniBand with GPUDirect RDMA between nodes are the critical interconnects
- CUDA dominates the software ecosystem; ROCm is maturing rapidly driven by Frontier and AMD investment
- For AI training, BF16/FP8 precision with tensor cores delivers 30–60× the throughput of FP64