Specialized hardware is needed in practice because training large machine learning models demands vast numbers of repeated mathematical operations and rapid movement of data and parameters. A CPU can perform this work, but GPUs and TPUs make it far faster and more energy-efficient.
The Reasoning and Mechanism
During training, a model processes inputs, calculates predictions, measures error with a loss function, and updates its parameters through repeated iterations. A large model may have billions of parameters, making matrix and vector calculations the dominant workload.
A central processing unit (CPU) has relatively few powerful cores designed for varied and sequential tasks. A graphics processing unit (GPU) contains many smaller cores that execute similar calculations simultaneously. This parallel processing suits the matrix operations used in machine learning.
A tensor processing unit (TPU) is an application-specific processor optimized for tensor and matrix calculations. GPUs and TPUs also provide high memory bandwidth, allowing data and parameters to reach processing units quickly.
| Hardware need | Why it matters | Typical solution |
|---|---|---|
| Processing throughput | Training repeats many matrix calculations | Parallel GPU or TPU cores |
| Memory capacity and bandwidth | Parameters, activations, and optimizer data must be stored and moved rapidly | Accelerator memory |
| Scalability | One device may be insufficient | Distributed computing across accelerators |
Distributed computing splits training across multiple accelerators, although communication between devices can become a bottleneck. Specialized hardware also reduces the time and electricity required for long training runs.
A common misconception is that large models cannot run on ordinary CPUs. They often can, but training may take impractical time and energy. Inference, using an already-trained model, usually requires less computation than training.
IB Exam Technique
For A4.1 Machine learning fundamentals, link workload to architecture: large-scale matrix operations benefit from parallel processing, high memory bandwidth, and distributed computing. Do not merely say GPUs are "faster"; explain why their architecture suits training.
