tensorflow-mnist-MLP-batch_normalization-weight_initializers

KO | EN

gitlite — search

#react #ai #typescript #golang #machine-learning #deep-learning #linux #docker #macos #java #windows #awesome #cli #pytorch

tensorflow-mnist-MLP-batch_normalization-weight_initializers

(★ 57)

MNIST classification using Multi-Layer Perceptron (MLP) with 2 hidden layers. Some weight-initializers and batch-normalization are implemented.

tensorflow-mnist-MLP-batch_normalization-weight_initializers Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Various initializers and batch normalization

An implementation of weight/bias initializers and batch normalization in Tensorflow.

MNIST database is used to show performance-comparison

## Network architecture

In order to examine the effect of initializers and batch normalization, a simple network architecture called multilayer perceptrons (MLP) is employed.

MLP has following architecture.

+ input layer : 784 nodes (MNIST images size)
+ first hidden layer : 256 nodes
+ second hidden layer : 256 nodes
+ output layer : 10 nodes (number of class for MNIST)

## Various initializers

The following initializers for weights/biases of network are considered.

+ Weight Initializer
  * normal, trucated normal
  * xaiver : [Understanding the difficulty of training deep feedforward neural networks](http://jmlr.org/proceedings/papers/v9/glorot10a/glorot10a.pdf)
  * he : [Delving Deep into Rectifiers](http://arxiv.org/pdf/1502.01852v1.pdf)
+ Bias Initializer
  * normal, constant
  
### Simulation results
+ Weight Initializer : he > trauncated normal = xaiver > normal
+ Bias Initilaizer : zero > normal

Sample results are following.

![weight_init](https://raw.githubusercontent.com/hwalsuklee/tensorflow-mnist-MLP-batch_normalization-weight_initializers/master/fig/results_weight_initializers.jpg)

## Batch normalization
Batch normalization improves performance of network in terms of final accuracy and convergence rate.</br>
In this simulation, bias initalizer was zero-constant initializer and weight initializers were xavier / he.

Sample results are following.

![batch_norm](https://raw.githubusercontent.com/hwalsuklee/tensorflow-mnist-MLP-batch_normalization-weight_initializers/master/fig/results_batch_normalization.jpg)

|Index|Weight Initializer|Batch Normalization|Accuracy|
|:---:|:---:|:---:|:---:|
|xavier_woBN_0.9806|xavier|Unused|0.9806|
|xavier_withBN_0.9812|xavier|Used|0.9812|
|he_woBN_0.9811|he|Unused|0.9811|
|he_withBN_0.9837|he|Used|0.9837|

## Usage

`python run_main.py --weight-init <weight initializer> --bias-init <bias initializer> --batch-norm <True or False>`

`<weight initializer>` must be selected in [normal, truncated_normal, xavier, he].</br>
`<bias initializer>` must be selected in [normal, zero].

You may command like `python run_main.py --weight-init xavier --bias-init zero --batch-norm True`.

## Acknowledgement

This implementation has been tested on Tensorflow r0.12.

tensorflow-mnist-MLP-batch_normalization-weight_initializers

tensorflow-mnist-MLP-batch_normalization-weight_initializers Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

tensorflow-mnist-MLP-batch_normalization-weight_initializers

tensorflow-mnist-MLP-batch_normalization-weight_initializers Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README