The GPU

The GPU is not a single inventor's triumph but a collective engineering achievement. Nvidia's founders—Jensen Huang, Chris Malachowsky, Curtis Priem (1993)—recognized that the massively parallel architecture needed to shade millions of pixels could be repurposed for scientific computing. By 2006, with CUDA (Compute Unified Device Architecture), Nvidia opened the GPU to general-purpose computation. Yet the true hero is the architecture itself: thousands of small cores working in lockstep, a radical departure from the CPU's sequential logic. This design, born from the constraints of real-time graphics, proved perfectly suited to the matrix multiplications that underpin deep learning. The GPU did not invent the neural network—that lineage traces to Rosenblatt (1958), Backpropagation (Rumelhart, Hinton, Williams, 1986)—but it made them *practical at scale*. Without the GPU's parallel hunger, the transformer (Vaswani et al., 2017) would have remained a theoretical curiosity. The device thus occupies a liminal space: tool and revolution, constraint and liberation.

The GPU's engineering triumph lies in its inversion of the CPU's design philosophy. Where a CPU optimizes for low latency and single-threaded speed, the GPU sacrifices latency for throughput: it accepts a 10–100× longer response time from a single instruction in exchange for executing thousands of instructions in parallel. This is enabled by a hierarchical memory system—registers (fastest, smallest), shared memory (fast, per-block), global memory (slow, largest)—and a programming model (CUDA, OpenCL, HIP) that exposes this hierarchy to developers. The Streaming Multiprocessor (SM), Nvidia's core building block, contains 128–192 CUDA cores, each capable of integer and floating-point operations. Modern GPUs (2020 onward) add Tensor Cores—specialized units for matrix multiplication—which accelerate the 4D tensor operations central to neural networks. The interconnect between GPU and CPU is critical: early GPUs communicated via PCIe (bandwidth ~16 GB/s in PCIe 3.0), a bottleneck. Nvidia's NVLink (2015 onward) raised inter-GPU bandwidth to 600 GB/s, enabling true multi-GPU training. Thermal management is acute: a 700 W GPU dissipates as much heat as a space heater; modern data centers employ liquid cooling loops. The engineering challenge is not raw performance but *efficiency*—operations per watt—because training a large language model consumes megawatt-hours.

The GPU's history divides into three eras: Graphics (1980–2005), Compute (2006–2016), and AI (2017–present). In the Graphics era, GPUs evolved from fixed-function rasterizers (Nvidia GeForce 256, 1999; ATI Radeon 8500, 2000) to programmable shaders. The GeForce 3 (2001) introduced vertex and pixel shaders, allowing developers to write custom graphics code. This programmability was the seed: researchers realized that the same parallel architecture could accelerate scientific simulations. In 2006, Nvidia released CUDA, a C-like language exposing the GPU's parallel architecture to general-purpose programmers. Early adopters were computational physicists and climate modelers; GPUs accelerated molecular dynamics, seismic imaging, and weather forecasting by 10–100×. The Compute era saw GPUs adopted in high-performance computing (HPC) clusters. Nvidia's Fermi architecture (2010) added ECC memory and better double-precision performance, making GPUs viable for scientific computing. The AI era began in earnest in 2012, when Geoffrey Hinton's team at the University of Toronto won the ImageNet competition using a deep convolutional neural network (AlexNet) trained on two Nvidia GTX 580 GPUs. This was the inflection point: deep learning, long dormant, suddenly became practical and powerful. By 2017, the transformer architecture (Vaswani et al., 2017) emerged, and GPUs became the indispensable substrate. Nvidia's market cap grew from $7 billion (2012) to $1+ trillion (2024), driven almost entirely by AI demand. The GPU had become the bottleneck and the enabler of the AI revolution.

The GPU was born from a specific constraint: rendering real-time 3D graphics requires computing the color of millions of pixels 60 times per second. A CPU, optimized for sequential execution and low latency, is poorly suited to this task. In the 1990s, as 3D gaming exploded (Quake, 1996; Half-Life, 1998), graphics cards evolved from simple frame buffers to dedicated processors. The key insight was that pixel shading is embarrassingly parallel: each pixel's color is computed independently, so thousands of cores can work in parallel without synchronization. This constraint—parallelism for graphics—created an architecture that happened to be ideal for other parallel workloads: matrix multiplication, convolution, and the forward/backward passes of neural networks. The GPU's existence was thus contingent on the commercial success of 3D gaming. Without Nvidia's competition with ATI (now AMD) to push graphics performance, the GPU might have remained a niche accelerator. Instead, the GPU's trajectory was: graphics → scientific computing → machine learning → artificial intelligence. Each transition was driven by researchers recognizing that the GPU's parallel architecture could solve their bottleneck. By 2012, with AlexNet, the feedback loop became self-reinforcing: AI researchers bought GPUs, which drove Nvidia's R&D, which produced faster GPUs, which enabled larger models, which drove more demand. The GPU thus exists because of a happy accident: an architecture optimized for one problem (pixel shading) proved optimal for another (neural network training).

In the Age of Revolutions exhibit, the GPU's 'daily use' is metaphorical—it is not wielded by a human operator but by software. A researcher training a large language model (LLM) interacts with the GPU indirectly: she writes Python code (using PyTorch, TensorFlow, or JAX), which is compiled to CUDA kernels, which are dispatched to the GPU. The GPU then executes billions of multiply-accumulate operations in parallel, returning results to the CPU. A typical training run for a 7-billion-parameter model might consume 100–1000 GPU-hours, costing $1000–$10,000 in cloud compute. During this time, the GPU is fully utilized: all cores executing instructions, all memory bandwidth saturated. The GPU's 'daily use' is thus continuous, intensive, and invisible. In inference (using a trained model), the GPU's role is different: a user submits a prompt (e.g., 'Write a poem about the Industrial Revolution'), and the GPU executes the model's forward pass, generating tokens one at a time. For a 70-billion-parameter model, this might take 5–30 seconds per response, consuming 50–100 GB of VRAM. In data centers, GPUs are racked in clusters of 8–1024 units, cooled by liquid loops, and accessed via APIs (OpenAI's GPT-4, Anthropic's Claude, Meta's Llama). The 'daily use' is thus: training (batch processing, hours to weeks), inference (interactive, seconds), and research (experimentation, minutes to hours). The GPU is never idle; its utilization is measured in percentage of peak FLOPS, and modern data centers aim for 80–95% utilization to justify the $10,000–$15,000 per-unit capital cost.

The GPU is not crewed but orchestrated by a distributed team: hardware engineers (Nvidia, AMD, Intel), software engineers (CUDA, PyTorch, TensorFlow developers), researchers (machine learning scientists), and operators (data-center engineers). Nvidia's GPU architecture is designed by teams in Santa Clara, California; Austin, Texas; and Bangalore, India. A single GPU architecture (e.g., Hopper, 2023) involves hundreds of engineers over 3–4 years. The CUDA ecosystem—compiler, runtime, libraries—is maintained by Nvidia's CUDA team (~100 engineers). PyTorch and TensorFlow, the dominant deep-learning frameworks, are developed by Meta and Google respectively, with contributions from thousands of open-source developers. A machine-learning researcher using a GPU is both operator and user: she writes code, monitors training loss, adjusts hyperparameters, and debugs failures. In large-scale training (e.g., training GPT-4), a team of 50–100 engineers manages the pipeline: data engineers prepare datasets, ML engineers design models, systems engineers optimize performance, and operators monitor hardware health. The GPU thus requires a 'crew' spanning hardware design, software infrastructure, machine learning, and operations—a supply chain of expertise. The most critical role is the ML researcher or engineer who understands both the model (transformer, diffusion, etc.) and the GPU's constraints (memory, bandwidth, latency), and can optimize the code to achieve 50–80% of peak FLOPS.

A modern GPU (e.g., Nvidia H100, 2023) is constructed in the following sequence: (1) Design: Nvidia's architects define the instruction set, microarchitecture, and memory hierarchy in SystemVerilog, a hardware description language. This phase takes 12–18 months and involves thousands of design reviews. (2) Verification: The design is simulated in software to catch bugs before silicon. Simulation can take years of compute time. (3) Tape-out: The verified design is sent to a foundry (TSMC, Samsung) as a 'tape-out,' the final design file. (4) Fabrication: TSMC manufactures the GPU on a 5 nm process node (as of 2023). A wafer contains hundreds of dies; each die is tested for defects. Yield (fraction of working dies) is typically 50–80%. (5) Assembly: Working dies are bonded to a substrate, wire-bonded to pads, and packaged in a BGA (Ball Grid Array) or LGA (Land Grid Array) package. (6) Testing: Each packaged GPU is tested for functionality and performance. (7) Integration: The GPU is mounted on a PCB (printed circuit board) with memory chips (HBM, high-bandwidth memory), power delivery, and cooling. (8) Validation: The assembled GPU is tested in a system, and firmware is loaded. The entire process, from design to shipping, takes 3–4 years. The cost of designing a GPU is $1–2 billion; manufacturing costs are ~$5,000 per unit (for a $15,000 retail price). The GPU is thus a capital-intensive product, requiring massive R&D investment and manufacturing scale.

GPU architectures vary by vendor (Nvidia, AMD, Intel), generation, and target market. Nvidia dominates with four product lines: (1) GeForce (consumer gaming), (2) RTX (professional graphics and AI), (3) Tesla/A-series (data-center AI training), (4) H-series (large-scale AI). Each line has multiple SKUs (stock-keeping units) with different core counts, memory, and power. For example, the RTX 4090 (consumer, 2022) has 16,384 CUDA cores and 24 GB VRAM; the H100 (data center, 2023) has 14,080 CUDA cores but 80 GB HBM3 memory and 3× the bandwidth. AMD's RDNA and CDNA architectures compete in gaming and AI respectively. Intel's Arc (consumer) and Ponte Vecchio (data center) are newer entrants. Variations also exist in memory type: GDDR6X (consumer, fast but power-hungry), HBM2/HBM3 (data center, high bandwidth), and LPDDR (mobile). Cooling varies: air (consumer), liquid (high-end), and immersion (experimental). Interconnect varies: PCIe (standard), NVLink (Nvidia multi-GPU), Infinity Fabric (AMD). Some GPUs are optimized for inference (smaller, lower power), others for training (larger, more memory). Specialized variants exist for specific workloads: tensor-optimized (AI), ray-tracing-optimized (graphics), and sparse-optimized (irregular compute). The GPU landscape is thus highly fragmented, with no single 'standard' GPU—only a family of designs optimized for different constraints.

Nvidia H100 (2023): The flagship data-center GPU, with 80 GB HBM3 memory, 14,080 CUDA cores, and 1.4 PFLOPS peak Tensor performance. Used in OpenAI's GPT-4 training and inference clusters. Cost: ~$15,000 per unit. Nvidia A100 (2020): The previous generation, with 80 GB HBM2 and 2 TFLOPS peak. Still widely used; more affordable than H100. Cost: ~$10,000. Nvidia RTX 4090 (2022): The flagship consumer GPU, with 24 GB GDDR6X and 16,384 CUDA cores. Popular for local AI experimentation and gaming. Cost: ~$1,600. AMD MI300X (2024): AMD's competitor to H100, with 192 GB HBM3 memory and 1.5 PFLOPS Tensor performance. Emerging alternative to Nvidia dominance. Cost: ~$12,000. Intel Ponte Vecchio (2023): Intel's data-center GPU, with 128 GB memory and 52 TFLOPS peak. Limited adoption; Intel's entry into AI accelerators. Cost: ~$10,000. Tesla V100 (2017): The GPU that trained AlexNet and early transformer models; 32 GB HBM2, 7 TFLOPS peak. Now vintage but historically significant. Cost: ~$8,000 (2017). Nvidia GeForce RTX 2080 Ti (2018): Consumer GPU that enabled local deep learning; 11 GB GDDR6, 14 TFLOPS peak. Widely used for research and gaming. Cost: ~$1,200. Each of these GPUs represents a generation of AI capability; the progression from V100 (2017) to H100 (2023) shows a 200× increase in peak Tensor performance and a 2.5× increase in memory bandwidth.

The GPU has no archaeological past—it is a contemporary artifact. However, the *history* of GPU design is preserved in several archives: (1) Nvidia's design documents and patents (filed with the U.S. Patent Office) detail the evolution of CUDA, Tensor Cores, and NVLink. (2) Academic papers, particularly Lindholm et al. (2008) on CUDA and Vaswani et al. (2017) on transformers, document the GPU's role in AI. (3) Oral histories: interviews with Nvidia founders Jensen Huang, Chris Malachowsky, and Curtis Priem (conducted by the Computer History Museum) provide firsthand accounts. (4) Hardware specimens: early GPUs (GeForce 256, GeForce 3, Fermi) are preserved in the Computer History Museum in Mountain View, California, and the Smithsonian's National Museum of American History. (5) Software artifacts: the CUDA toolkit source code, PyTorch and TensorFlow repositories, and training logs from landmark models (AlexNet, GPT-3) are archived on GitHub and in institutional repositories. (6) Performance data: benchmarks and performance reports from Nvidia, MLPerf, and academic papers provide quantitative records of GPU evolution. Unlike physical artifacts, the GPU's 'archaeology' is largely digital and distributed.

The GPU is often compared to the CPU (Central Processing Unit) and the TPU (Tensor Processing Unit). CPU vs. GPU: A CPU (e.g., Intel Core i9, AMD Ryzen) has 8–64 cores optimized for sequential execution and low latency; a GPU has thousands of cores optimized for parallel execution and throughput. A CPU executes one instruction per core per cycle; a GPU executes thousands. For AI workloads, a GPU is 10–100× faster than a CPU. However, a CPU is better for irregular, branching code (e.g., parsing, control flow). GPU vs. TPU: A TPU (Tensor Processing Unit, Google's custom AI accelerator) is a specialized chip designed *only* for neural networks, with no graphics capability. TPUs are more power-efficient than GPUs (50–100 W vs. 350–700 W) but less flexible; they cannot run arbitrary code. TPUs are used internally by Google (for Gemini, Bard) but not sold to customers. GPUs are more widely available and more programmable. GPU vs. ASIC: An ASIC (Application-Specific Integrated Circuit) is a custom chip designed for a single task (e.g., Bitcoin mining). ASICs are more efficient than GPUs but cannot be reprogrammed. GPUs are the middle ground: more flexible than ASICs, more efficient than CPUs. The GPU's advantage is *programmability with parallelism*—it can run any code, but is optimized for parallel workloads. This flexibility is why GPUs dominate AI: they can run any neural network architecture, from CNNs to transformers to diffusion models.