HPC-Paper-Notes

KO | EN

gitlite — search

#react #ai #typescript #go #deep-learning #linux #docker #nodejs #windows #awesome #cli #kubernetes #mcp #openai

HPC-Paper-Notes

(★ 27)

My notes on various HPC papers.

HPC-Paper-Notes Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# List of Papers

1. [Energy Efficient Architecture for Graph Analytics Accelerators](papers/Energy-Efficient-Architecture-for-Graph-Analytics-Accelerators)
2. [A Template Based Design Methodology for Graph Parallel Hardware Accelerators](papers/A-Template-Based-Design-Methodology-for-Graph-Parallel-Hardware-Accelerators)
3. [System Simulation with gem5 and SystemC](papers/System-Simulation-with-gem5-and-SystemC)
4. [GAIL: The Graph Algorithm Iron Law](papers/The-Graph-Algorithm-Iron-Law)
5. [Locality Exists in Graph Processing: Workload Characterization on an Ivy Bridge Serve](papers/Locality-Exists-In-Graph-Processing)
6. [Graphicionado A High Performance and Energy-Efficient Accelerator for Graph Analytics](papers/Graphicionado-A-High-Performance-and-Energy-Efficient-Accelerator-for-Graph-Analytics)
7. [Analysis and Optimization of the Memory Hierarchy for Graph Processing Workloads](papers/Analysis-and-Optimization-of-the-Memory-Hierarchy-for-Graph-Processing-Workloads)
8. [Alleviating Irregularity in Graph Analytics Acceleration: a Hardware/Software Design Approach](papers/Alleviating-Irregularity-in-Graph-Analytics-Acceleration)
9. [GNN Performance Optimization](papers/GNN-Performance-Optimization)
10. [Dissecting the Graphcore IPU Architecture](papers/Dissecting-the-Graphcore-IPU-Architecture)
11. [Using the Graphcore IPU for Traditional HPC Applications](papers/Using-the-Graphcore-IPU-for-Traditional-HPC-Applications)
12. [Roofline: An Insightful Visual Performance Model](papers/Roofline-An-Insightful-Visual-Performance-Model)
13. [CUDA New Features and Beyond](papers/CUDA-New-Features-and-Beyond)
14. [A Study of Persistent Threads Style GPU Programming for GPGPU Workloads](papers/A-Study-of-Persistent-Threads-Style-GPU-Programming-for-GPGPU-Workloads)
15. [BrainTorrent: A Peer to Peer Environment for Decentralized Federated Learning](papers/BrainTorrent-A-Peer-to-Peer-Environment-for-Decentralized-Federated-Learning)
16. [Whippletree: Task-based Scheduling of Dynamic Workloads on the GPU](papers/Whippletree:Task-based-Scheduling-of-Dynamic-Workloads-on-the-GPU)
17. [Groute: Asynchronous Multi-GPU Programming Model with Applications to Large-scale Graph Processing](papers/Groute-Asynchronous-Multi-GPU-Programming-Model-with-Applications-to-Large-scale-Graph-Processing)
18. [A Computational-Graph Partitioning Method for Training Memory-Constrained DNNs](papers/A-Computational-Graph-Partitioning-Method-for-Training-Memory-Constrained-DNNs)
19. [The Broker Queue: A Fast, Linearizable FIFO Queue for Fine-Granular Work Distribution on the GPU](papers/The-Broker-Queue-A-Fast-Linearizable-FIFO-Queue-for-Fine-Granular-Work-Distribution-on-the-GPU)
20. [Softshell: Dynamic Scheduling on GPUs](papers/Softshell-Dynamic-Scheduling-on-GPUs)
21. [Gravel: Fine-Grain GPU-Initiated Network Messages](papers/Gravel-Fine-Grain-GPU-Initiated-Network-Messages)
22. [SPIN:Seamless Operating System Integration of Peer to Peer DMA Between SSDs and GPUs](papers/SPIN-Seamless-Operating-System-Integration-of-Peer-to-Peer-DMA-Between-SSDs-and-GPUs)
23. [Automatic Graph Partitioning for Very Large-scale Deep Learning](papers/Automatic-Graph-Partitioning-for-Very-Large-scale-Deep-Learning)
24. [Stateful Dataflow Multigraphs: A data-centric
model for performance portability on heterogeneous architectures](papers/Stateful-Dataflow-Multigraphs-A-Data-Centric-Model-for-Performance-Portability-on-Heterogeneous-Architectures)
25. [Productivity, Portability, Performance: Data-Centric Python](papers/Productivity-Portability-Performance-Data-Centric-Python)
26. [Interferences between Communications and Computations in Distributed HPC Systems](papers/Interferences-between-Communications-and-Computations-in-Distributed-HPC-Systems)
27. [MVAPICH2-GPU: optimized GPU to GPU communication for InfiniBand clusters](papers/MVAPICH2-GPU-optimized-GPU-to-GPU-communication-for-InfiniBand-clusters)
28. [GGAS: Global GPU Address Spaces for Efficient Communication in Heterogeneous Clusters](papers/GGAS-Global-GPU-Address-Spaces-for-Efficient-Communication-in-Heterogeneous-Clusters)
29. [GPUnet: Networking Abstractions for GPU Programs](papers/GPUnet-Networking-Abstractions-for-GPU-Programs)
30. [GPUrdma: GPU-side library for high performance networking from GPU kernels](papers/GPUrdma-GPU-side-library-for-high-performance-networking-from-GPU-kernels)
31. [Trends in Data Locality Abstractions for HPC Systems](papers/Trends-in-Data-Locality-Abstractions-for-HPC-Systems)
32. [Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelines](papers/Chimera-Efficiently-Training-Large-Scale-Neural-Networks-with-Bidirectional-Pipelines)
33. [Benchmarking GPUs to Tune Dense Linear Algebra](papers/Benchmarking-GPUs-to-Tune-Dense-Linear-Algebra)
34. [Brook for GPUs: stream computing on graphics hardware](papers/Brook-for-GPUs-stream-computing-on-graphics-hardware)
35. [IPUG: Accelerating Breadth-First Graph Traversals using Manycore Graphcore IPUs](papers/IPUG-Accelerating-Breadth-First-Graph-Traversals-using-Manycore-Graphcore-IPUs)
36. [Supporting RISC-V Performance Counters through Performance analysis tools for Linux](papers/Supporting-RISC-V-Performance-Counters-through-Performance-analysis-tools-for-Linux)
37. [Merrimac: Supercomputing with Streams](papers/Merrimac-Supercomputing-with-Streams)
38. [Peta-scale Phase-Field Simulation for Dendritic Solidification on the TSUBAME 2.0 Supercomputer](papers/Peta-scale-Phase-Field-Simulation-for-Dendritic-Solidification-on-the-TSUBAME-2-Supercomputer)
39. [A-RISC-V-Simulator-and-Benchmark-Suite-for-Designing-and-Evaluating-Vector-Architectures](papers/A-RISC-V-Simulator-and-Benchmark-Suite-for-Designing-and-Evaluating-Vector-Architectures)
40. [PyTorch Distributed Experiences on Accelerating Data Parallel Training](papers/PyTorch-Distributed-Experiences-on-Accelerating-Data-Parallel-Training)
41. [An Oracle for Guiding Large-Scale Model/Hybrid Parallel Training of Convolutional Neural Networks](papers/An-Oracle-for-Guiding-Large-Scale-Model-Hybrid-Parallel-Training-of-Convolutional-Neural-Networks)
42. [ZeRO Memory Optimizations Toward Training Trillion Parameter Models](papers/ZeRO-Memory-Optimizations-Toward-Training-Trillion-Parameter-Models)
43. [Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM](papers/Efficient-Large-Scale-Language-Model-Training-on-GPU-Clusters-Using-Megatron-LM)
44. [XeFlow: Streamlining Inter-Processor Pipeline Execution for the Discrete CPU-GPU Platform](papers/XeFlow-Streamlining-Inter-Processor-Pipeline-Execution-for-the-Discrete-CPU-GPU-Platform)
45. [Architecture and Performance of Devito, a System for Automated Stencil Computation](papers/Architecture-and-Performance-of-Devito-a-System-for-Automated-Stencil-Computation)
46. [Distributed Training of Deep Learning Models A Taxonomic Perspective](papers/Distributed-Training-of-Deep-Learning-Models-A-Taxonomic-Perspective)
47. [Performance Trade-offs in GPU Communication A Study of Host and Device-initiated Approaches](papers/Performance-Trade-offs-in-GPU-Communication-A-Study-of-Host-and-Device-initiated-Approaches)
48. [Assessment of NVSHMEM for High Performance Computing](papers/Assessment-of-NVSHMEM-for-High-Performance-Computing)
49. [Sparse GPU Kernels for Deep Learning](papers/Sparse-GPU-Kernels-for-Deep-Learning)
50. [The State of Sparsity in Deep Neural Networks](papers/The-State-of-Sparsity-in-Deep-Neural-Networks)
51. [Pruning neural networks without any data by iteratively conserving synaptic flow](papers/Pruning-Neural-Networks-Without-Any-Data-by-Iteratively-Conserving-Synaptic-Flow)
52. [SNIP: Single-shot Network Pruning based on Connection Sensitivity](papers/SNIP-Single-shot-Network-Pruning-based-on-Connection-Sensitivity)
53. [Comparing Rewinding and Fine-tuning in Neural Network Pruning](papers/Comparing-Rewinding-and-Fine-tuning-in-Neural-Network-Pruning)
54. [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](papers/The-Lottery-Ticket-Hypothesis-Finding-Sparse-Trainable-Neural-Networks)
55. [Torch.fx: Practical Program Capture and Transformation for Deep Learning in Python](papers/torch.fx)
56. [An asynchronous message driven parallel framework for extreme scale deep learning](papers/An-asynchronous-message-driven-parallel-framework-for-extreme-scale-deep-learning)
57. [Bolt: Bridging The Gap Between Auto Tuners And Hardware Native Performance](papers/Bolt-Bridging-The-Gap-Between-Auto-Tuners-And-Hardware-Native-Performance)
58. [Efficient Tensor Core-Based GPU Kernels for Structured Sparsity under Reduced Precision](papers/Efficient-Tensor-Core-Based-GPU-Kernels-for-Structured-Sparsity-under-Reduced-Precision)
59. [Attention is All You Need](papers/Attention-Is-All-You-Need)
60. [Scaling Laws for Neural Language Models](papers/Scaling-Laws-for-Neural-Language-Models)
61. [Language Models are Few-Shot Learners](papers/Language-Models-are-Few-Shot-Learners)
62. [BERT Pre-training of Deep Bidirectional Transformers for Language Understanding](papers/BERT-Pre-training-of-Deep-Bidirectional-Transformers-for-Language-Understanding)
63. [RoBERTa: A Robustly Optimized BERT Pretraining Approach](papers/RoBERTa-A-Robustly-Optimized-BERT-Pretraining-Approach)
64. [Longformer: The Long-Document Transformer](papers/Longformer-The-Long-Document-Transformer)
65. [Linformer: Self-Attention with Linear Complexity](papers/Linformer-Self-Attention-with-Linear-Complexity)
66. [The Efficiency Misnomer](papers/The-Efficiency-Misnomer)
67. [A Survey of Transformers](papers/A-Survey-of-Transformers)
68. [PipeTransformer-Automated Elastic Pipelining for Distributed Training of Large-scale Models](papers/PipeTransformer-Automated-Elastic-Pipelining-for-Distributed-Training-of-Large-scale-Models)
69. [Training Compute-Optimal Large Language Models](papers/Training-Compute-Optimal-Large-Language-Models)
70. [WholeGraph: A Fast Graph Neural Network Training Framework with Multi-GPU Distributed Shared Memory Architecture](papers/WholeGraph/)
71. [Sparse-GPT-Massive-Language-Models-Can-Be-Accurately-Pruned-in-One-Shot](papers/Sparse-GPT-Massive-Language-Models-Can-Be-Accurately-Pruned-in-One-Shot/)

## Two Papers A Week Goal (Starting from 28.06.2021)

### 28.06.2021 - 04.07.2021

* [Whippletree: Task-based Scheduling of Dynamic Workloads on the GPU](papers/Whippletree-Task-based-Scheduling-of-Dynamic-Workloads-on-the-GPU)
* [Groute: Asynchronous Multi-GPU Programming Model with Applications to Large-scale Graph Processing](papers/Groute-Asynchronous-Multi-GPU-Programming-Model-with-Applications-to-Large-scale-Graph-Processing)

### 05.07.2021 - 11.07.2021

* [A Computational-Graph Partitioning Method for Training Memory-Constrained DNNs](papers/A-Computational-Graph-Partitioning-Method-for-Training-Memory-Constrained-DNNs)
* [The Broker Queue: A Fast, Linearizable FIFO Queue for Fine-Granular Work Distribution on the GPU](papers/The-Broker-Queue-A-Fast-Linearizable-FIFO-Queue-for-Fine-Granular-Work-Distribution-on-the-GPU)

### 12.07.2021 - 18.07.2021

* [Softshell: Dynamic Scheduling on GPUs](papers/Softshell-Dynamic-Scheduling-on-GPUs)
* [Gravel: Fine-Grain GPU-Initiated Network Messages](papers/Gravel-Fine-Grain-GPU-Initiated-Network-Messages)

### 09.08.2021 - 15.08.2021

* [SPIN:Seamless Operating System Integration of Peer to Peer DMA Between SSDs and GPUs](papers/SPIN-Seamless-Operating-System-Integration-of-Peer-to-Peer-DMA-Between-SSDs-and-GPUs)
* [GPU-to-CPU Callbacks](papers/https://link.springer.com/content/pdf/10.1007%2F978-3-642-21878-1_45.pdf)

### 16.08.2021 - 22.08.2021

* [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://arxiv.org/pdf/1912.01703.pdf) -> Zero technical depth. Please give my time back.
* [Automatic Graph Partitioning for Very Large-scale Deep Learning](papers/Automatic-Graph-Partitioning-for-Very-Large-scale-Deep-Learning)

### 23.08.2021 - 29.08.2021

* [Stateful Dataflow Multigraphs: A data-centric
model for performance portability on heterogeneous architectures](papers/Stateful-Dataflow-Multigraphs-A-Data-Centric-Model-for-Performance-Portability-on-Heterogeneous-Architectures)
* [Productivity, Portability, Performance: Data-Centric Python](papers/Productivity-Portability-Performance-Data-Centric-Python)

### 30.08.2021 - 05.09.2021

* [Analyzing Put/Get APIs for Thread-collaborative Processors](https://ieeexplore.ieee.org/document/7103479)
* [Analyzing Communication Models for Distributed Thread-Collaborative Processors in Terms of Energy and Time](https://ieeexplore.ieee.org/document/7095817)

### 06.09.2021 - 12.09.2021

* [Interferences between Communications and Computations in Distributed HPC Systems](papers/Interferences-between-Communications-and-Computations-in-Distributed-HPC-Systems)
* [Memory Bandwidth Contention: Communication vs Computation Tradeoffs in Supercomputers with Multicore Architectures](papers/Memory-Bandwidth-Contention-Communication-vs-Computation-Tradeoffs-in-Supercomputers-with-Multicore-Architectures)

### 13.09.2021 - 19.09.2021

* [MVAPICH2-GPU: optimized GPU to GPU communication for InfiniBand clusters](papers/MVAPICH2-GPU-optimized-GPU-to-GPU-communication-for-InfiniBand-clusters)
* [GGAS: Global GPU Address Spaces for Efficient Communication in Heterogeneous Clusters](papers/GGAS-Global-GPU-Address-Spaces-for-Efficient-Communication-in-Heterogeneous-Clusters)

### 20.09.2021 - 26.09.2021

* [GPUnet: Networking Abstractions for GPU Programs](papers/GPUnet-Networking-Abstractions-for-GPU-Programs)
* [GPUrdma: GPU-side library for high performance networking from GPU kernels](papers/GPUrdma-GPU-side-library-for-high-performance-networking-from-GPU-kernels)

### 27.09.2021 - 03.10.2021

* [Trends in Data Locality Abstractions for HPC Systems](papers/Trends-in-Data-Locality-Abstractions-for-HPC-Systems)
* [Moores Law is ending](https://ieeexplore.ieee.org/document/7368023)

### 04.10.2021 - 10.10.2021

* [Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelines](papers/Chimera-Efficiently-Training-Large-Scale-Neural-Networks-with-Bidirectional-Pipelines)
* [Deep Residual Learning for Image Recognition](papers/Deep-Residual-Learning-for-Image-Recognition)

### 11.10.2021 - 17.10.2021

* [Benchmarking GPUs to Tune Dense Linear Algebra](papers/Benchmarking-GPUs-to-Tune-Dense-Linear-Algebra)
* [Brook for GPUs: stream computing on graphics hardware](papers/Brook-for-GPUs-stream-computing-on-graphics-hardware)

### 18.10.2021 - 24.10.2021

* [IPUG: Accelerating Breadth-First Graph Traversals using Manycore Graphcore IPUs](papers/IPUG-Accelerating-Breadth-First-Graph-Traversals-using-Manycore-Graphcore-IPUs)
* [Supporting RISC-V Performance Counters through Performance analysis tools for Linux](papers/Supporting-RISC-V-Performance-Counters-through-Performance-analysis-tools-for-Linux)

### 25.10.2021 - 31.10.2021

* [Merrimac: Supercomputing with Streams](papers/Merrimac-Supercomputing-with-Streams) 
* [Peta-scale Phase-Field Simulation for Dendritic Solidification on the TSUBAME 2.0 Supercomputer](papers/Peta-scale-Phase-Field-Simulation-for-Dendritic-Solidification-on-the-TSUBAME-2-Supercomputer)

### 01.11.2021 - 07.11.2021

* [Toward a Scalable and Distributed Infrastructure for Deep Learning Applications](https://arxiv.org/abs/2010.03012)
* [A Data-centric Optimization Framework for Machine Learning](https://arxiv.org/pdf/2110.10802.pdf)

### 08.11.2021 - 14.11.2021

* [A RISC-V Simulator and Benchmark Suite for Designing and Evaluating Vector Architectures](papers/A-RISC-V-Simulator-and-Benchmark-Suite-for-Designing-and-Evaluating-Vector-Architectures)
* [PyTorch Distributed Experiences on Accelerating Data Parallel Training](papers/PyTorch-Distributed-Experiences-on-Accelerating-Data-Parallel-Training)

### 15.11.2021 - 21.11.2021
* [An Oracle for Guiding Large-Scale Model/Hybrid Parallel Training of Convolutional Neural Networks](papers/An-Oracle-for-Guiding-Large-Scale-Model-Hybrid-Parallel-Training-of-Convolutional-Neural-Networks)
* [ZeRO Memory Optimizations Toward Training Trillion Parameter Models](papers/ZeRO-Memory-Optimizations-Toward-Training-Trillion-Parameter-Models)

### 22.11.2021 - 28.11.2021
* [ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning](https://arxiv.org/pdf/2104.07857.pdf)
* [Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM](papers/Efficient-Large-Scale-Language-Model-Training-on-GPU-Clusters-Using-Megatron-LM)

### 29.11.2021 - 05.12.2021
* Unpublished paper (will update if it's accepted)
* [XeFlow: Streamlining Inter-Processor Pipeline Execution for the Discrete CPU-GPU Platform](papers/XeFlow-Streamlining-Inter-Processor-Pipeline-Execution-for-the-Discrete-CPU-GPU-Platform)

### 06.11.2021 - 12.12.2021
* [Architecture and Performance of Devito, a System for Automated Stencil Computation](papers/Architecture-and-Performance-of-Devito-a-System-for-Automated-Stencil-Computation)
* [Distributed Training of Deep Learning Models A Taxonomic Perspective](papers/Distributed-Training-of-Deep-Learning-Models-A-Taxonomic-Perspective)

### 13.12.2021 - 19.12.2021
* [Performance Trade-offs in GPU Communication A Study of Host and Device-initiated Approaches](papers/Performance-Trade-offs-in-GPU-Communication-A-Study-of-Host-and-Device-initiated-Approaches)
* [Assessment of NVSHMEM for High Performance Computing](papers/Assessment-of-NVSHMEM-for-High-Performance-Computing)

### 20.12.2021 - 26.12.2021
* [A Data-Centric Optimization Framework for Machine Learning](https://arxiv.org/pdf/2110.10802.pdf)
* [Scaling Distributed Deep Learning Workloads beyond the Memory Capacity with KARMA](https://arxiv.org/pdf/2008.11421.pdf)

## Essential Reading List in Parallel Computing (Including suggestions of my advisor (Didem Unat))

### Trends

* &#9989; [Moores Law is ending](https://ieeexplore.ieee.org/document/7368023)
* [A new golden age for computer architecture](https://dl.acm.org/doi/10.1145/3282307)
* [Abstract machine models and proxy architectures for exascale computing](https://scholar.google.ch/citations?view_op=view_citation&hl=fr&user=NR70RpYAAAAJ&citation_for_view=NR70RpYAAAAJ:IjCSPb-OGe4C)  
* &#9989; [Trends in Data Locality Abstractions for HPC Systems](https://ieeexplore.ieee.org/document/7924389)

### Architectures

* &#9989; [Merrimac: Supercomputing with Streams](https://ieeexplore.ieee.org/document/1592938)
* [Synergistic Processing in Cell's Multicore Architecture](https://ieeexplore.ieee.org/document/1624323)
* [Knights Landing: Second Generation Intel Xeon Phi Product](https://ieeexplore.ieee.org/document/7453080)

### Performance Models and Tools

* &#9989; [Roofline: an insightful visual performance model for multicore architectures](https://dl.acm.org/doi/fullHtml/10.1145/1498765.1498785)
* [ExaSAT: An exascale co-design tool for performance modeling](https://journals.sagepub.com/doi/full/10.1177/1094342014568690)
* [hwloc: A generic framework for managing hardware affinities in HPC applications](https://ieeexplore.ieee.org/abstract/document/5452445?casa_token=mT7X-a9jAzoAAAAA:PLvrDSzj4Ek_Jsp3NQhNY8t7CCnP9EXCBwa2XsgFZUH-JZ3-sRWPrhdpemlxSUxiUKy94n-p2w)

### Applications

* [Optimization of sparse matrix-vector multiplication on emerging multicore platforms](https://ieeexplore.ieee.org/abstract/document/5348797)
* [Stencil computation optimization and auto-tuning on state-of-the-art multicore architectures](https://ieeexplore.ieee.org/abstract/document/5222004)
* &#9989; [Benchmarking GPUs to tune dense linear algebra](https://ieeexplore.ieee.org/document/5214359)

### Programming Models

* &#9989; [Brook for GPUs: Stream Computing on Graphics Hardware](https://dl.acm.org/doi/10.1145/1015706.1015800)
* [OmpSs: A PROPOSAL FOR PROGRAMMING HETEROGENEOUS MULTI-CORE ARCHITECTURES](https://www.worldscientific.com/doi/abs/10.1142/S0129626411000151)
* [Productivity and performance using partitioned global address space languages](https://dl.acm.org/doi/10.1145/1278177.1278183)
* [Kokkos: Enabling manycore performance portability through polymorphic memory access patterns](https://www.osti.gov/servlets/purl/1106586)

### Compilers

* [Halide: A Language and Compiler for Optimizing Parallelism, Locality, and Recomputation in Image Processing Pipelines](https://people.csail.mit.edu/jrk/halide-pldi13.pdf)
* [Chill: A Framework for High Level Loop Transformations](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.214.8396&rep=rep1&type=pdf)
* [Pluto: A Practical and Automatic Polyhedral Program Optimization System](https://dl.acm.org/doi/10.1145/1375581.1375595)

### Runtime Systems

* [Cilk: An Efficient Multithreaded Runtime System](http://supertech.csail.mit.edu/papers/PPoPP95.pdf)
* [StarPU: a unified platform for task scheduling on heterogeneous multicore architectures](https://hal.inria.fr/inria-00550877/document)
* [Legion: expressing locality and independence with logical regions](https://dl.acm.org/doi/10.5555/2388996.2389086)
* [Charm++ A portable concurrent object oriented system based on C++](https://dl.acm.org/doi/pdf/10.1145/165854.165874)

## My Scalable Deep Learning List (Just a list of papers that I read recently. Not recommendations)

* [PyTorch Distributed Experiences on Accelerating Data Parallel Training](papers/PyTorch-Distributed-Experiences-on-Accelerating-Data-Parallel-Training)
* [An Oracle for Guiding Large-Scale Model/Hybrid Parallel Training of Convolutional Neural Networks](papers/An-Oracle-for-Guiding-Large-Scale-Model-Hybrid-Parallel-Training-of-Convolutional-Neural-Networks)
* [ZeRO Memory Optimizations Toward Training Trillion Parameter Models](papers/ZeRO-Memory-Optimizations-Toward-Training-Trillion-Parameter-Models)
* [Distributed Training of Deep Learning Models A Taxonomic Perspective](papers/Distributed-Training-of-Deep-Learning-Models-A-Taxonomic-Perspective)
* [Sparse GPU Kernels for Deep Learning](papers/Sparse-GPU-Kernels-for-Deep-Learning)
* [The State of Sparsity in Deep Neural Networks](papers/The-State-of-Sparsity-in-Deep-Neural-Networks)
* [Pruning neural networks without any data by iteratively conserving synaptic flow](papers/Pruning-Neural-Networks-Without-Any-Data-by-Iteratively-Conserving-Synaptic-Flow)
* [SNIP: Single-shot Network Pruning based on Connection Sensitivity](papers/SNIP-Single-shot-Network-Pruning-based-on-Connection-Sensitivity)
* [Comparing Rewinding and Fine-tuning in Neural Network Pruning](papers/Comparing-Rewinding-and-Fine-tuning-in-Neural-Network-Pruning)
* [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](papers/The-Lottery-Ticket-Hypothesis-Finding-Sparse-Trainable-Neural-Networks)
* [Torch.fx: Practical Program Capture and Transformation for Deep Learning in Python](papers/torch.fx)
* [An asynchronous message driven parallel framework for extreme scale deep learning](papers/An-asynchronous-message-driven-parallel-framework-for-extreme-scale-deep-learning)
* [Bolt: Bridging The Gap Between Auto Tuners And Hardware Native Performance](papers/Bolt-Bridging-The-Gap-Between-Auto-Tuners-And-Hardware-Native-Performance)
* [Efficient Tensor Core-Based GPU Kernels for Structured Sparsity under Reduced Precision](papers/Efficient-Tensor-Core-Based-GPU-Kernels-for-Structured-Sparsity-under-Reduced-Precision)
* [Attention is All You Need](papers/Attention-Is-All-You-Need)
* [Scaling Laws for Neural Language Models](papers/Scaling-Laws-for-Neural-Language-Models)
* [Language Models are Few-Shot Learners](papers/Language-Models-are-Few-Shot-Learners)
* [BERT Pre-training of Deep Bidirectional Transformers for Language Understanding](papers/BERT-Pre-training-of-Deep-Bidirectional-Transformers-for-Language-Understanding)
* [RoBERTa: A Robustly Optimized BERT Pretraining Approach](papers/RoBERTa-A-Robustly-Optimized-BERT-Pretraining-Approach)
* [Longformer: The Long-Document Transformer](papers/Longformer-The-Long-Document-Transformer)
* [Linformer: Self-Attention with Linear Complexity](papers/Linformer-Self-Attention-with-Linear-Complexity)
* [The Efficiency Misnomer](papers/The-Efficiency-Misnomer)
* [A Survey of Transformers](papers/A-Survey-of-Transformers)
* [PipeTransformer-Automated Elastic Pipelining for Distributed Training of Large-scale Models](papers/PipeTransformer-Automated-Elastic-Pipelining-for-Distributed-Training-of-Large-scale-Models)
* [Training Compute-Optimal Large Language Models](papers/Training-Compute-Optimal-Large-Language-Models)
* [Sparse-GPT-Massive-Language-Models-Can-Be-Accurately-Pruned-in-One-Shot](papers/Sparse-GPT-Massive-Language-Models-Can-Be-Accurately-Pruned-in-One-Shot/)

HPC-Paper-Notes

HPC-Paper-Notes Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

HPC-Paper-Notes

HPC-Paper-Notes Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README