Exascale Computing

What is Exascale?

Exascale computing refers to systems capable of performing at least 10¹⁸ floating-point operations per second — one quintillion FLOPS, or one ExaFLOP/s. To put this in perspective:

• 1 ExaFLOP/s = 1,000 PetaFLOP/s = 1,000,000 TeraFLOP/s
• If every person on Earth performed one calculation per second, it would take 4 months to match what an exascale machine does in one second
• The world's fastest personal computer in 2010 was roughly 1,000,000× slower

The exascale milestone was first crossed in May 2022 when Frontier at Oak Ridge National Laboratory achieved 1.102 ExaFLOP/s on the HPL LINPACK benchmark.

The Exascale Systems (2022–2026)

Frontier — The First Exascale Machine

Frontier at Oak Ridge National Laboratory (ORNL) consists of 9,408 compute nodes, each containing one AMD EPYC 7A53 CPU and four AMD Instinct MI250X GPUs. The nodes are connected by HPE Slingshot 11 (200 Gb/s) in a dragonfly topology.

Key numbers: 700 petabytes of storage (Lustre), 9,400+ nodes, 37,632 GPUs, ~21 MW power consumption, cooled entirely by water at 80% efficiency.

Aurora — Intel's Exascale Entry

Aurora at Argonne National Laboratory (ANL) uses Intel's Ponte Vecchio (Data Center GPU Max) accelerators — Intel's first discrete GPU designed for HPC and AI. Each node has two Intel Xeon CPU Max processors and six Intel GPU Max 1550 accelerators. Aurora reached exascale milestone in 2024 after significant delays in Intel's GPU development.

The Engineering Challenges

Power and Energy

Frontier consumes approximately 21 megawatts — enough to power around 20,000 homes. This is the dominant constraint for exascale systems. The DOE ECP set an energy efficiency target of 20–30 MW for exascale systems, compared to the ~9 MW for the Titan petascale predecessor.

Power is managed at every level: voltage regulators on every chip, dynamic frequency scaling, workload-aware power capping, and facility-level power distribution systems that can respond in milliseconds.

Cooling

Air cooling cannot handle rack power densities exceeding ~30–40 kW/rack. Frontier's racks run at ~70 kW per rack, requiring direct liquid cooling (DLC) on every CPU and GPU. Water at ~15°C flows through cold plates directly attached to the chips, removing heat at the source before it can raise ambient temperatures.

Reliability at Scale

With 37,000+ GPUs, statistically, hardware failures occur multiple times per day. At this scale, the system cannot simply stop when a component fails — it must:

• Detect failures within seconds via health monitoring daemons on every node
• Checkpoint running applications regularly (every 30–60 minutes) so work is not lost
• Drain failed nodes and reschedule jobs transparently
• Maintain spare nodes for rapid replacement

Memory and Storage

Exascale systems generate and consume data at unprecedented rates. Frontier's 700 PB Lustre filesystem delivers over 5 TB/s — and still struggles to keep pace with peak compute throughput. The memory hierarchy (HBM on GPU → DRAM → NVMe burst buffer → Lustre → tape archive) must be carefully managed to avoid I/O becoming the bottleneck.

Science Enabled by Exascale

Programming Exascale Systems

Programming exascale machines is fundamentally different from conventional HPC. The key challenges are:

• Heterogeneous programming — every node has both CPU and GPU; code must efficiently utilize both
• Portable performance — Frontier (AMD ROCm), Aurora (Intel oneAPI), and future systems all use different GPU programming models. HIP, SYCL, and OpenMP target offload aim for portability
• Deep memory hierarchy — HBM, DRAM, NVMe, Lustre, and tape each require different access patterns and data placement strategies
• Fault tolerance — applications must checkpoint frequently and restart cleanly

What Comes After Exascale?

The HPC roadmap continues. Zettascale (10²¹ FLOPS) is the next milestone — roughly 1,000× Frontier. Current projections suggest zettascale systems are feasible in the 2035–2040 timeframe, contingent on breakthroughs in memory technology, interconnect bandwidth, and energy efficiency. Quantum computing may complement (but not replace) classical HPC for specific problem classes.

Key Takeaways

• Exascale = 10¹⁸ FLOPS — first achieved by Frontier (ORNL) in 2022
• The key challenges are power (~21 MW), cooling, and reliability at unprecedented scale
• Exascale enables qualitatively new science — not just faster, but previously impossible simulations
• Programming requires heterogeneous CPU+GPU code with portable performance across different vendor ecosystems
• The next milestone is zettascale — projected for the mid-2030s