100-days-of-cuda

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

100-days-of-cuda

★ 39 Open GitHub ↗

Will write CUDA for 100 days

Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# 100 days of CUDA

Will write CUDA for 100 days

https://github.com/hkproj/100-days-of-gpu/blob/main/CUDA.md

### Progress by Day

| Day   | Summary                                                                                                                                                                                                                                                                                                                                                                                                         |
| ----- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| day01 | Read chapter 1 and 2 of PMPP and implemented CUDA kernel for vector addition                                                                                                                                                                                                                                                                                                                                    |
| day02 | Implemented CUDA kernel for RGB to gray kernel                                                                                                                                                                                                                                                                                                                                                                  |
| day03 | Read chapter 3 of PMPP and implemented three different types of matrix multiplication kernels, each thread for each output cell, each thread of each output row and each thread for each output column                                                                                                                                                                                                          |
| day04 | Read chapter 4 of PMPP and integrated C++/Cuda with pytorch to test previously coded rgb2gray and blur kernels.                                                                                                                                                                                                                                                                                                 |
| day05 | Coded CUDA kernel to generate mandelbrot set image and integrated it with pytorch to visualize the final image using matplotlib.                                                                                                                                                                                                                                                                                |
| day06 | Create a simple interactive website to render mandelbrot using WASM (works only on CPU)                                                                                                                                                                                                                                                                                                                         |
| day07 | Read chapter 5 of PMPP and implemented CUDA kernel for tiled matrix multiplication                                                                                                                                                                                                                                                                                                                              |
| day08 | Implemented sigmoid CUDA kernel                                                                                                                                                                                                                                                                                                                                                                                 |
| day09 | Started implementing a neural network with core operations in CUDA to train on MNIST data                                                                                                                                                                                                                                                                                                                       |
| day10 | Implement conv1d CUDA kernel                                                                                                                                                                                                                                                                                                                                                                                    |
| day11 | Finished implementing the CPU part of my tensor library and I can create simple neural networks with it. Added basic CUDA kernels like ReLU and matrix add to the library                                                                                                                                                                                                                                       |
| day12 | Implemented CUDA kernels for cross entropy and softmax operations. Optimized the softmax kernel using online algorithm, shared memory, and parallel reduction                                                                                                                                                                                                                                                   |
| day13 | Wrote layer norm CUDA kernel and optimized it using shared memory and parallel reduction                                                                                                                                                                                                                                                                                                                        |
| day14 | Finished implementing all the kernels and functions for my tensor library that were required to train a simple neural network on MNIST data. After lots of debugging I was able to train the NN on both CPU and GPU. On CPU the training took 1 min 30 sec to train the network whereas on GPU it literally trained in 1.5 secs. Also wrote a simple gui using raylib to predict handwritten number in realtime |
| day15 | Implemented Adam optimizer's step CUDA kernel                                                                                                                                                                                                                                                                                                                                                                   |
| day16 | Implemented GeLU CUDA kernel                                                                                                                                                                                                                                                                                                                                                                                    |
| day17 | Revised chapter 4 and read chapter 5 and 6 of PMPP and implemented tiled matrix multiplication using block coarsening                                                                                                                                                                                                                                                                                           |
| day18 | Read chapter 7 of PMPP and implemented conv2d kernel using constant memory                                                                                                                                                                                                                                                                                                                                      |
| day19 | Implemented tiled conv2d kernel                                                                                                                                                                                                                                                                                                                                                                                 |
| day20 | Implement a naive matrix multiplication kernel in PTX                                                                                                                                                                                                                                                                                                                                                           |
| day21 | Implemented naive stencil 3d kernel                                                                                                                                                                                                                                                                                                                                                                             |
| day22 | Learned how fuzzy search works and its scoring algorithm. Tried implementing CUDA kernel for wagner fischer algorithm                                                                                                                                                                                                                                                                                           |
| day23 | Implemented wavefront parallelization and fixed yesterday's wagner fischer implementation                                                                                                                                                                                                                                                                                                                       |
| day24 | Started working on my fuzzy match library fcuk (fuzzy cuda kernel). Implemented a matching and scoring algorithm for strings using dynamic programming similar to wagner fischer's algorithm with affine gap penalty. The scoring system is similar to fzy. Right now I have only implemented the CUDA kernel for the scoring algorithm                                                                         |
| day25 | Fused compute bonus and scoring kernel and fixed previous logical bugs. Also implemented a matching kernel using parallel reduction                                                                                                                                                                                                                                                                             |
| day26 | Compared my results with other fuzzy match libraries and fixed some bugs in my scoring algorithm                                                                                                                                                                                                                                                                                                                |
| day27 | Wrote a single fused kernel for scoring all the matched strings and implemented several optimizations, such as using pinned memory and reducing the quadratic memory usage of wavefront parallelism to linear shared memory                                                                                                                                                                                     |
| day28 | Implemented fused matching kernel for my fuzzy matcher library                                                                                                                                                                                                                                                                                                                                                  |
| day29 | Read chapter 7 of PMPP. Reimplemented naive and tiled conv2d kernel and implemented tiled conv2d kernel using l1 and l2 cache                                                                                                                                                                                                                                                                                   |
| day30 | Implemented a colorful 2d metaball CUDA kernel                                                                                                                                                                                                                                                                                                                                                                  |
| day31 | Tried implementing scaled dot product attention (failed)                                                                                                                                                                                                                                                                                                                                                        |
| day32 | Still stuck on scaled dot product attention                                                                                                                                                                                                                                                                                                                                                                     |
| day33 | Read flash attention paper and finally implemented flash attention CUDA kernel                                                                                                                                                                                                                                                                                                                                  |
| day34 | Implemented flash attention 2                                                                                                                                                                                                                                                                                                                                                                                   |
| day35 | Reached rank 2 on GPU mode's grayscale leaderboard                                                                                                                                                                                                                                                                                                                                                              |
| day36 | Reached rank 1 on GPU mode's grayscale leaderboard on A100, L4 and T4                                                                                                                                                                                                                                                                                                                                           |
| day37 | Added batch and multihead support to flash attention 2 implementation                                                                                                                                                                                                                                                                                                                                           |
| day38 | Learned about cooperative groups CUDA api and implemented softmax using cooperative groups api.                                                                                                                                                                                                                                                                                                                 |
| day39 | Learned about asynchronous data transfer and tensor memory accelerator (TMA) operations introduced in hopper architecture. Implemented grayscale kernel using TMA instructions                                                                                                                                                                                                                                  |
| day40 | Implemented inclusive and exclusive scan using kogge stone algorithm                                                                                                                                                                                                                                                                                                                                            |
| day41 | Implemented inclusive and exclusive scan using brent kung algorithm                                                                                                                                                                                                                                                                                                                                             |
| day42 | Learned about rotary positional embedding (RoPE) and implemented its CUDA kernel                                                                                                                                                                                                                                                                                                                                |
| day43 | Learned about privatization and atomic operations. Implemented color histogram CUDA kernel                                                                                                                                                                                                                                                                                                                      |
| day44 | Implemented merge CUDA kernel                                                                                                                                                                                                                                                                                                                                                                                   |
| day45 | Implemented radix sort CUDA kernel utilizing kogge stone algorithm for exclusive scan                                                                                                                                                                                                                                                                                                                           |
| day46 | Learned about warp level primitives and implemented them in softmax CUDA kernel. Also learned about streams and implemented a pipelined vector add kernel using multiple streams.                                                                                                                                                                                                                               |
| day47 | Read RMSNorm paper and implemented its CUDA kernel                                                                                                                                                                                                                                                                                                                                                              |
| day48 | Implemented SiLU CUDA kernel                                                                                                                                                                                                                                                                                                                                                                                    |
| day49 | Basic CuBLAS GEMM kernel                                                                                                                                                                                                                                                                                                                                                                                        |
| day50 | Optimizing GEMM to match CuBLAS performance, implemented tiled matrix multiplication with 2d coarsening                                                                                                                                                                                                                                                                                                         |
| day51 | Fixed bugs in GEMM kernel                                                                                                                                                                                                                                                                                                                                                                                       |
| day52 | Naive transpose and thread coarsened CUDA kernel                                                                                                                                                                                                                                                                                                                                                                |
| day53 | Implemented transpose kernel using shared memory with padding                                                                                                                                                                                                                                                                                                                                                   |
| day54 | Fixed transpose kernel benchmarking and added vectorized load gemm with shared blocktiling                                                                                                                                                                                                                                                                                                                      |
| day55 | Implemented vector addition HIP kernel                                                                                                                                                                                                                                                                                                                                                                          |
| day56 | Wrote swish kernel and reached no. 1 on tensara for it                                                                                                                                                                                                                                                                                                                                                          |
| day57 | Wrote hard softmax kernel and reached no. 1 on tensara for it                                                                                                                                                                                                                                                                                                                                                   |
| day58 | Finally understood warp tiling and implemented gemm kernel using warp tiling and achieve rank 1 on for square matrix multiplication tensara                                                                                                                                                                                                                                                                     |
| day59 | Read about hopper's matrix multiply accumulate instruction and TMA and implemented ReLU kernel on tensara                                                                                                                                                                                                                                                                                                       |
| day60 | Started working on FP8 GEMM kernel for AMD challenge                                                                                                                                                                                                                                                                                                                                                            |
| ...   | ...                                                                                                                                                                                                                                                                                                                                                                                                             |
| day97 | Reached rank 1 on AMD challenge FP8 GEMM kernel                                                                                                                                                                                                                                                                                                                                                                 |

100-days-of-cuda

Download

Today's picks

Was this content helpful?

Recent Feedback

100-days-of-cuda

Download

Today's picks

Was this content helpful?

Recent Feedback

Download README