deep-learning-foundations

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

deep-learning-foundations

★ 8 Open GitHub ↗

A list of the papers introducing the most popular advancement in deep learning (still some to be added over the next few days)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Deep Learning Foundations

A list of the papers introducing the most popular advancement in deep learning. A list that is curated to only present the essentials, the hit singles of deep learning as we know it if you will. I decided not to include any advanced papers as I consider that they are of little use for beginners and practionners with no interest in advanced research.
These are the papers that I consider paramount to read and understand as they present the challenges they tried to tackle and as a result gives some insight into neural networks practical uses.

## Architectures

#### Generative Adversarial Networks
**Generative adversarial nets** Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., ... & Bengio, Y. (2014) [[pdf]](http://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf)

#### LeNet 5
**Backpropagation applied to handwritten zip code recognition** Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. (1989) [[pdf]](http://yann.lecun.com/exdb/publis/pdf/lecun-89e.pdf)

#### AlexNet

**ImageNet classification with deep convolutional neural networks**, Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. In Proceedings of the 25th International Conference on Neural Information Processing Systems - Volume 1 (2012) [[pdf]](https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf).

#### VGGNet

**Very deep convolutional networks for large-scale image recognition**, K Simonyan, A Zisserman - arXiv preprint arXiv:1409.1556, (2014) [[pdf]](https://arxiv.org/pdf/1409.1556v6.pdf).

#### GoogLeNet (Inception V1)

**Going deeper with convolutions**, Szegedy, Christian, et al., Proceedings of the IEEE conference on computer vision and pattern recognition. (2015) [[pdf]](https://arxiv.org/pdf/1409.4842.pdf).

#### ResNet

**Deep residual learning for image recognition** He, Kaiming, Xiangyu Zhang, Shaoqing Ren, and Jian Sun (2016) [[pdf]](http://openaccess.thecvf.com/content_cvpr_2016/papers/He_Deep_Residual_Learning_CVPR_2016_paper.pdf)

#### Inception V2 & V3
**Rethinking the inception architecture for computer vision** Szegedy, Christian, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. (2016) [[pdf]](https://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/Szegedy_Rethinking_the_Inception_CVPR_2016_paper.pdf)

#### Inception V4
**Inception-v4, inception-resnet and the impact of residual connections on learning** Szegedy, Christian, Sergey Ioffe, Vincent Vanhoucke, and Alexander A. Alemi (2017) [[pdf]](https://arxiv.org/pdf/1512.00567)

## Activation Functions

#### Relu
**Deep sparse rectifier neural networks**, Glorot Xavier, Antoine Bordes, and Yoshua Bengio (2011) [[pdf]](http://proceedings.mlr.press/v15/glorot11a/glorot11a.pdf)

#### LeakyRelu
**Rectifier Nonlinearities Improve Neural Network Acoustic Models**  Andrew L. Maas Awni Y. Hannun Andrew Y. Ng (2012) [[pdf]](https://ai.stanford.edu/~amaas/papers/relu_hybrid_icml2013_final.pdf)

#### PRelu
**Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification** Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun (2015) [[pdf]](https://arxiv.org/pdf/1502.01852.pdf)

#### ELU
**Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)**, Djork-Arné Clevert, Thomas Unterthiner, Sepp Hochreiter (2016) [[pdf]](https://arxiv.org/pdf/1511.07289).

### Review
**An overview of gradient descent optimization algorithms**, Sebastian Ruder (2016) [[pdf]](https://arxiv.org/pdf/1609.04747.pdf).

## Gradient Descent

#### Best practices
**Efficient Backprop** LeCun, Yann A., Léon Bottou, Genevieve B. Orr, and Klaus-Robert Müller (2012) [[pdf]](http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf)

#### Adam
**Adam: A Method for Stochastic Optimization**, Diederik P. Kingma, Jimmy Ba (2014) [[pdf]](http://arxiv.org/abs/1412.6980)

#### Follow the regularized leader
**Ad Click Prediction: a View from the Trenches**, H. Brendan McMahan, Gary Holt, D. Sculley, Michael Young, Dietmar Ebner, Julian Grady, Lan Nie, Todd Phillips, Eugene Davydov, Daniel Golovin, Sharat Chikkerur, Dan Liu, Martin Wattenberg, Arnar Mar Hrafnkelsson, Tom Boulos, Jeremy Kubica (2013) [[pdf]](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/41159.pdf)

#### Xavier initialisation

**Understanding the difficulty of training deep feedforward neural networks**  Xavier Glorot, Yoshua Bengio (2010) [[pdf]](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf)

#### He et al. initialisation

**Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification** Kaiming He and Xiangyu Zhang and Shaoqing Ren and Jian Sun (2015) [[pdf]](https://arxiv.org/pdf/1502.01852.pdf)

## Regularization

**Dropout: A Simple Way to Prevent Neural Networks from Overfitting**, Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky
Ilya Sutskever, Ruslan Salakhutdinov (2014) [[pdf]](http://jmlr.org/papers/volume15/srivastava14a.old/srivastava14a.pdf).

**Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift**, Sergey Ioffe, Christian Szegedy (2015) [[pdf]](https://arxiv.org/pdf/1502.03167).

**Overtraining, regularization, and searching for minimum in neural networks**, Sjöberg Jonas, Lennart Ljung (1992) [[pdf]](https://pdfs.semanticscholar.org/6bb2/8b64d262b6aefae157be03399db08e41307c.pdf)

**Regularization of neural networks using dropconnect** Wan L, Zeiler M, Zhang S, Le Cun Y, Fergus R. (2013) [[pdf]](http://proceedings.mlr.press/v28/wan13.pdf)

**Layer normalization** Ba, Jimmy Lei, Jamie Ryan Kiros, and Geoffrey E. Hinton. (2013) [[pdf]](https://arxiv.org/pdf/1607.06450.pdf)

deep-learning-foundations

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README