High-Throughput Computing (HTC)
Maximizing total job volume through HTCondor & Cycle Scavenging.
The Power of "Embarrassingly Parallel" Tasks
In 2026, High-Throughput Computing (HTC) focuses on executing a massive volume of independent tasks over long periods. Unlike traditional HPC, which prioritizes the speed of a single simulation, HTC prioritizes the total number of jobs completed per month or year.
ClassAds & Matchmaking
HTCondor uses a unique "Classified Advertisements" system instead of a top-down queue:
- Resource Offers: Machines advertise CPU, RAM, and availability (e.g., "Idle for 15 min").
- Resource Requests: Users specify requirements (Memory > 2GB) and rank preferences.
- The Matchmaker: A central Negotiator daemon brokers the best matches in real-time.
Built for Resilience & Autonomy
| Feature | Mechanism | Impact in 2026 |
|---|---|---|
| Cycle Scavenging | Detects idle workstations. | Turns office PCs into a supercomputer at night. |
| Check-pointing | Saves job state to disk. | Resumes jobs on different nodes without data loss. |
| File Transfer | Native data movement. | Works without shared filesystems (Cloud-ready). |
| DAGMan | Acyclic Graph Manager. | Manages complex dependencies between millions of jobs. |
HTCondor in the 2026 Cloud-Hybrid Era
Condor Annex
Elastically "annexes" thousands of nodes from AWS, Azure, or Google Cloud. HTCondor shuts them down automatically when the queue is empty to optimize costs.
Spot Instance Optimization
Perfect for 90% cheaper "Spot Instances." Due to native checkpointing, terminated jobs simply move back to the queue until the next resource is available.
HPC Middleware vs. HTC Middleware
HPC (Slurm / PBS)
Goal: Minimize wall-clock time for one job.
Connectivity: High-speed low-latency fabric.
Workload: Tightly coupled MPI.
HTC (HTCondor)
Goal: Maximize total throughput for many jobs.
Connectivity: Standard Ethernet / Internet.
Workload: Independent Parameter Sweeps.
2026 Implementation Best Practices
Vanilla Universe
Use for maximum portability of standard binaries without specialized libraries.
Internal Transfer
Use transfer_input_files to reduce file server load and ensure cloud-readiness.
Match Analysis
Debug with condor_q -analyze to see why specific ClassAds fail to match.