Learned-Activation-Functions-Source

KO | EN

gitlite — search

#typescript #llm #go #rust #linux #docker #macos #java #nodejs #windows #cli #kubernetes #mcp #open-source

Learned-Activation-Functions-Source

(★ 19)

For the reproduction of research by Agostinelli et al. Learning Activation Functions to Improve Deep Neural Networks. http://arxiv.org/abs/1412.6830

Learned-Activation-Functions-Source Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

==========================================
Introduction
==========================================
This repository if for the reproduction of research by Agostinelli et al. Learning Activation Functions to Improve Deep Neural Networks. http://arxiv.org/abs/1412.6830

This is a learnable activation function for neural networks. Experiments have shown that it outperforms rectified linear units (ReLU) and Leaky ReLU.

==========================================
Memory
==========================================
The use of this layer will require more memory. There is a "save_mem" option that can be used. However, this will lead to slower performance and has not yet been thoroughly tested.

==========================================
In place computation
==========================================
In place computation can be done. However, due to implementation details, it does not conserve memory and tests show it will result in a slight decrease in speed.

==========================================
Solver Files
==========================================
We made custom changes to the solver files. The changes are reflected in src/caffe/solver.cpp, include/caffe.solver.hpp, and src/caffe/proto/caffe.proto

==========================================
Defining the Learned Activation Functions
==========================================
```
layer {
  name: "apl1"
  type: "APL"
  bottom: "blob_name"
  top: "blob_name"
  param {
    decay_mult: 1 # We set so weight decay is 0.001
  }
  param {
    decay_mult: 1 # We set so weight decay is 0.001
  }
  apl_param {
    sums: 1
    slope_filler {
      type: "uniform"
      min: -0.5
      max: 0.5
    }
    offset_filler {
      type: "gaussian"
      std: 0.5
    }
  }
}
```

Learned-Activation-Functions-Source

Learned-Activation-Functions-Source Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Learned-Activation-Functions-Source

Learned-Activation-Functions-Source Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README