KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#developer-tools #typescript #codex #ai-agents #claude-code #dsh #dsh-plugin #cli #deepseek-harness #open-source #macos #ai-agent

BEFORE-GPU-ENGINEERING

★ 88 Open GitHub ↗

A first-principles climb from the absolute bottom of hardware to the top of GPU architecture.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<div align="center">
 <h1>Before GPU Engineering</h1>
 A first-principles climb from the absolute bottom of hardware to the top of GPU architecture.
 
 
 <a href="https://before-gpu-engineering.github.io/BEFORE-GPU-ENGINEERING/toc.html" style="color: #ff0000;">READ THE LIVE DIGITAL NOTEBOOK HERE</a>
 
</div>

> **New to GPU Engineering? Start Here.**
> If you are diving into CUDA, kernel programming, or LLM inference, consider this your ultimate prerequisite. Before battling complex textbooks or dense research papers, read this series. You will build a rock-solid, first-principles foundation to actually understand advanced literature and write aggressively fast code.

---

### 📖 The Philosophy
This repository documents my journey learning GPU engineering, built in public, note by note, from the ground up. **No hand-waving or abstract jargon.** Just logic gates, memory buses, and first principles.

### 🎯 Who Is This For?
**Developers, students, and self-learners** who want to understand *how* and *why* GPUs are so fast for AI and heavy compute, without getting immediately bogged down in complex CUDA syntax or graphics APIs. If you understand basic programming but GPUs feel like black magic, this is for you.

### 🚀 What Will You Learn?
By the end of this series, you will no longer see the GPU as a black box. You will be able to:
- **Reason from first principles** about architecture (warps, SMs, memory hierarchy, tensor cores).
- **Diagnose bottlenecks** intuitively (compute-bound vs. memory-bound, latency hiding).
- **Learn CUDA/Triton faster**, by understanding the *why* behind thread blocks and memory coalescing.
- **Optimize AI workloads** with a deep mechanical sympathy for hardware.

### 💡 How to Use This Resource
Treat this as a structured book. **Start from Chapter 1 and read sequentially.** The concepts build heavily on each other. You cannot understand matrix tiling without understanding shared memory, and you cannot understand shared memory without understanding the memory wall.

---

## 📚 Syllabus

> 🚧 **Work in Progress:** This series is actively being written and continuously updated. There are **5 phases** in total. **57 chapters are live so far across Phases 1–4**; Phase 5 is in progress.

### Phase 1

#### Part 1: The Architecture

- [Chapter 1: Why CPUs Can't Have 10,000 Cores](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/architecture/chapter-1.html)
- [Chapter 2: Why a GPU Multiplies Matrices Faster](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/architecture/chapter-2.html)
- [Chapter 3: What "Parallelizable" Actually Means](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/architecture/chapter-3.html)

#### Part 2: Feeding the Beast

- [Chapter 4: Why GPUs Care So Much About Memory Bandwidth](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/feeding-the-beast/chapter-4.html)
- [Chapter 5: Compute-Bound vs. Memory-Bound](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/feeding-the-beast/chapter-5.html)
- [Chapter 6: The GPU Memory Hierarchy](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/feeding-the-beast/chapter-6.html)

#### Part 3: Inside the Silicon

- [Chapter 7: What Problem Do Warps Solve?](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/inside-the-silicon/chapter-7.html)
- [Chapter 8: Why Branching Is Bad on GPUs](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/inside-the-silicon/chapter-8.html)
- [Chapter 9: What Is SIMT? (And How It Differs From SIMD)](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/inside-the-silicon/chapter-9.html)
- [Chapter 10: Why Occupancy Matters](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/inside-the-silicon/chapter-10.html)

#### Part 4: Putting It Together

- [Chapter 11: Why AI Runs on GPUs but Databases Don't](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/putting-it-together/chapter-11.html)
- [Chapter 12: What Is Latency Hiding?](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/putting-it-together/chapter-12.html)
- [Chapter 13: What if a GPU Had CPU-Style Caches?](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/putting-it-together/chapter-13.html)
- [Chapter 14: The Memory Wall & HBM](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/putting-it-together/chapter-14.html)
- [Chapter 15: Designing an LLM Training Processor](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/putting-it-together/chapter-15.html)

### Phase 2

#### The Translation Barrier

- [Chapter 16: Why C++ Won't Auto-Compile to GPU](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/the-translation-barrier/chapter-16.html)
- [Chapter 17: A Kernel Is Not a Function](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/the-translation-barrier/chapter-17.html)
- [Chapter 18: Why Thread Indexing Exists](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/the-translation-barrier/chapter-18.html)

#### The Hardware Hierarchy

- [Chapter 19: Why Threads Group Into Blocks](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/the-hardware-hierarchy/chapter-19.html)
- [Chapter 20: Why Shared Memory Is Fast](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/the-hardware-hierarchy/chapter-20.html)
- [Chapter 21: Why RAM Isn't Visible to the GPU](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/the-hardware-hierarchy/chapter-21.html)
- [Chapter 22: Kernel Launch Overhead](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/the-hardware-hierarchy/chapter-22.html)

#### Synchronization & Hazards

- [Chapter 23: What Syncthreads Protects](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/synchronization-hazards/chapter-23.html)
- [Chapter 24: Why Race Conditions Happen](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/synchronization-hazards/chapter-24.html)
- [Chapter 25: What Makes CUDA Code "Correct"?](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/synchronization-hazards/chapter-25.html)

#### Scale & Performance

- [Chapter 26: Why the GPU Context Switch Is Free but CPU's Is Slow](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/scale-and-performance/chapter-26.html)
- [Chapter 27: The Step-by-Step Anatomy of a Kernel Launch](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/scale-and-performance/chapter-27.html)
- [Chapter 28: Why NVIDIA Made CUDA Asynchronous](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/scale-and-performance/chapter-28.html)
- [Chapter 29: Coalesced vs. Strided Memory Access](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/scale-and-performance/chapter-29.html)
- [Chapter 30: Why GPU Programs Actually Slow Down](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/scale-and-performance/chapter-30.html)
- [Chapter 31: Why Matrix Add Scales but a Linked List Dies](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/scale-and-performance/chapter-31.html)
- [Chapter 32: When Not to Use a GPU, and the Reduction Tree](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/scale-and-performance/chapter-32.html)

### Phase 3

#### The Memory Wall

- [Chapter 33: Your ALUs Are Idle. Memory Is the Real Bottleneck.](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/the-memory-wall/chapter-33.html)
- [Chapter 34: What Is Arithmetic Intensity? The Currency of GPU Optimization.](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/the-memory-wall/chapter-34.html)
- [Chapter 35: Two Hard Ceilings. One Tells You to Fix Your Math. The Other Says: Fix Your Memory.](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/the-memory-wall/chapter-35.html)

#### The Register Physics

- [Chapter 36: Why Not Build a GPU Entirely From Registers?](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/the-register-physics/chapter-36.html)
- [Chapter 37: Where Should Data Live? The Register-Occupancy Tradeoff.](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/the-register-physics/chapter-37.html)
- [Chapter 38: What Is Occupancy, Really? It's a Ratio, Not a Count, and 100% Is Not the Goal.](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/the-register-physics/chapter-38.html)
- [Chapter 39: Why Higher Occupancy Can Make Performance Worse.](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/the-register-physics/chapter-39.html)

#### Register Spilling

- [Chapter 40: What Is Register Spilling? The Cliff Between 1 ms and 50 ms.](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/register-spilling/chapter-40.html)
- [Chapter 41: Splitting the Giant](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/register-spilling/chapter-41.html)
- [Chapter 42: When Spilling Is Acceptable, and When It Destroys Performance.](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/register-spilling/chapter-42.html)

#### Global Memory & Coalescing

- [Chapter 43: Why Global Memory Latency Is 350 to 1,000 Cycles: The Physics of Distance.](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/global-memory-and-coalescing/chapter-43.html)
- [Chapter 44: What Is Memory Coalescing? 32 Threads, 1 Transaction vs. 32 Transactions.](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/global-memory-and-coalescing/chapter-44.html)
- [Chapter 45: DRAM Row-Buffer Thrashing: Why Scattered Access Is Structurally Broken.](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/global-memory-and-coalescing/chapter-45.html)

#### Shared Memory & Tiling

- [Chapter 46: Why Shared Memory Is Powerful](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/shared-memory-and-tiling/chapter-46.html)
- [Chapter 47: Bank Conflicts: The Hidden Trap](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/shared-memory-and-tiling/chapter-47.html)
- [Chapter 48: Memory Reuse: Escape the Wall](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/shared-memory-and-tiling/chapter-48.html)
- [Chapter 49: What Is Tiling?](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/shared-memory-and-tiling/chapter-49.html)

#### GEMM & the Synthesis

- [Chapter 50: Why Matmul Is King](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/gemm-and-the-synthesis/chapter-50.html)
- [Chapter 51: Throughput vs. Latency Mindset](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/gemm-and-the-synthesis/chapter-51.html)
- [Chapter 52: What Is Latency Hiding?](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/gemm-and-the-synthesis/chapter-52.html)
- [Chapter 53: Why Profiling Beats Intuition](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/gemm-and-the-synthesis/chapter-53.html)

### Phase 4

#### The Measurement Mandate

- [Chapter 54: The 91 Missing TFLOPS](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/the-measurement-mandate/chapter-54.html)
- [Chapter 55: Observe, Don't Assume](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/the-measurement-mandate/chapter-55.html)
- [Chapter 56: Nsight Systems vs. Compute](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/the-measurement-mandate/chapter-56.html)
- [Chapter 57: What Kernel Time Measures](https://BEFORE-GPU-ENGINEERING.github.io/BEFORE-GPU-ENGINEERING/the-measurement-mandate/chapter-57.html)

### Phase 5

> 🚧 Coming soon.

---

### 💬 Join the Discussion
Learning GPU architecture is hard. Don't do it alone. **[Join the GitHub Discussions](https://github.com/BEFORE-GPU-ENGINEERING/BEFORE-GPU-ENGINEERING/discussions)** to ask questions, share your "aha!" moments, and show off your custom kernels.

## 🤝 How to Contribute

This is a living, public resource, and **community contributions are highly encouraged!**

You can help improve this project in several ways:
- **Adding or Fixing Diagrams:** Many chapters are missing visual aids, or existing diagrams could be made clearer. If you can create a great visual for a concept, feel free to add it!
- **Improving Notes & Information:** If you spot a typo, a technical inaccuracy, or know a better/simpler way to explain a concept, open a PR to update the text.
- **Enhancing the UI:** The custom digital notebook interface itself is open to improvements in design, responsiveness, or readability.

### To Contribute:
1. Fork the repository.
2. Create a new branch for your feature or fix (`git checkout -b feature/improve-diagram`).
3. Commit your changes (`git commit -m "Add coalescing diagram to Chapter 44"`).
4. Push to the branch (`git push origin feature/improve-diagram`).
5. Open a Pull Request!

---

## 📜 License

This project is licensed under the [MIT License](LICENSE).

BEFORE-GPU-ENGINEERING

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README