KO
|
EN
gitlite — search
Search
#python
#java
#python3
#arduino
#golang
#machine-learning
#rust
#html
#flask
#javascript
#seismology
#nodejs
MACH
★ 10
Open GitHub ↗
Biological-Scale Neural Networks
Download README (.md)
Explore Similar Repositories
LittleWheelsESC
:
An open source arduino based ESC project for 12v to 24v kids ride-on toys
deepspeech-pt-br-scripts
:
DEPRECATED - Scripts and tools used to train pt-BR models for DeepSpeech
Unity_TCP
:
async/await TCP socket implementation for Unity3d.
cgsf2019
:
cgsf visual studio 2019 version
CVE-2017-17562
:
Standalone Python 3 exploit for CVE-2017-17562
// repository documentation
Was this content helpful?
★ 0
(0 ratings)
Select Rating:
★
★
★
★
★
Submit Feedback
Recent Feedback
×
Download README
Do you want to download the
README.md
file for
MACH
?
Download (.md)
## Biological Scale Neural Networks [](https://travis-ci.com/unconst/MACH) ### Sequential Distillation models. ``` ███████╗ ██████╗ ██████╗ █████╗ ██╗ ██╔════╝██╔═══██╗██╔══██╗ ██╔══██╗██║ █████╗ ██║ ██║██████╔╝ ███████║██║ ██╔══╝ ██║ ██║██╔══██╗ ██╔══██║██║ ██║ ╚██████╔╝██║ ██║██╗██║ ██║██║ ╚═╝ ╚═════╝ ╚═╝ ╚═╝╚═╝╚═╝ ╚═╝╚═╝ ``` "In reality, the law always contains less than the fact itself, because it does not reproduce the fact as a whole but only in that aspect of it which is important for us, the rest being intentionally or from necessity omitted." -- Ernst Mach ## TL;DR Increasing depth-wise scaling by training sequential distillation models. ## Motivation Depth is good, but deeper networks increasingly suffer from ’gradient locking’ if the network is required to update synchronously [1] [2] [4]. This issue can be avoided using depth-wise model parallelism, where sections of the network train independently. This, however, creates delayed gradients [5] which affect convergence when component depth exceed a certain size. This research investigates a new class of neural network architecture which is composed of many sequentially connected sub-components each training asynchronously and distilling knowledge from their child. The resulting model is, by its nature, a distilled versions of itself thus immediately usable in a production environment at reduced computational cost. Finally, the approach is well suited to an internet-wide environment making p2p training a potential avenue for future research. For a deeper description read the [research](https://www.overleaf.com/read/fvyqcmybsgfj) or join proj-mach on slack. --- ## Run ``` $ virtualenv env && source env/bin/activate && pip install -r requirements.txt $ python main.py ``` --- ## Resources Paper: https://www.overleaf.com/read/fvyqcmybsgfj </br> Code: https://www.github.com/unconst/Mach --- ## Pull Requests Use [Yapf](https://github.com/google/yapf) for code formatting ``` $ pip install yapf $ yapf --style google -r -vv -i . ``` --- ## References: References 1. Decoupled Neural Interfaces using Synthetic Gradients. </br> https://arxiv.org/pdf/1608.05343.pdf 2. Decoupled Parallel Backpropagation with Convergence Guarantee. </br> https://arxiv.org/pdf/1804.10574.pdf 3. Outrageously Large Neural Networks: Sparsely Gated Mixtures of Experts. </br> https://arxiv.org/abs/1701.06538 4. AMPNet: Asynchronous Model-Parallel Training for Dynamic Neural Networks. </br> https://www.microsoft.com/en-us/research/wp-content/uploads/2017/07/1705.09786.pdf 5. An analysis of delayed gradients problem in asynchronous SGD. </br> https://pdfs.semanticscholar.org/716b/a3d174006c19220c985acf132ffdfc6fc37b.pdf 6. Improved Knowledge Distillation via Teacher Assistant: Bridging the Gap Between Student and Teacher. </br> https://arxiv.org/abs/1902.03393
