AdaBound - GitLite

KO | EN

gitlite — search

#ai #typescript #llm #go #golang #android #macos #java #nodejs #windows #kubernetes #vue #pytorch #openai

AdaBound

(★ 2,902)

An optimizer that trains as fast as Adam and as good as SGD.

파일 탐색기

최종 버전 다운로드 (.zip)

- __init__.py
- adabound.py
- - .gitignore
  - main.py
  - README.md
  - requirements.txt
  - visualization.ipynb
- README.md
.gitignore
LICENSE
README.md
release.sh
setup.py

# 설치 가이드

1. 코드 내려받기

git clone https://github.com/Luolc/AdaBound

깃허브에서 프로젝트 코드 전체를 내 컴퓨터로 내려받습니다.

cd AdaBound

방금 내려받은 프로젝트 폴더 안으로 이동합니다.

2. 공식 설치 스크립트

쉬움 추천

사전 준비물

Python 3 pip 명령어를 쓰려면 Python이 필요합니다.

pip install adabound

PyPI에 배포된 패키지를 바로 설치합니다. 소스 클론이 필요 없습니다.

✅ 설치 후 새 터미널을 열고, 프로그램의 버전 확인 명령(예: --version)으로 정상 설치됐는지 확인하세요.

이 레포의 README에 적힌 실제 명령어를 그대로 가져왔습니다.

3. Python

쉬움

사전 준비물

Git GitHub에서 프로젝트 코드를 내려받으려면 필요합니다.
Python 3 설치 시 'Add Python to PATH' 옵션을 꼭 체크하세요 (Windows).

pip install adabound

PyPI에 배포된 패키지를 바로 설치합니다. 소스 클론이 필요 없습니다.

✅ 에러 메시지 없이 실행되고 터미널에 안내 문구가 출력되면 정상입니다.

이 레포의 README에 적힌 실제 명령어를 그대로 가져왔습니다.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# AdaBound
[![PyPI - Version](https://img.shields.io/pypi/v/adabound.svg?style=flat)](https://pypi.org/project/adabound/)
[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/adabound.svg)](https://pypi.org/project/adabound/)
[![PyPI - Wheel](https://img.shields.io/pypi/wheel/adabound.svg?style=flat)](https://pypi.org/project/adabound/)
[![GitHub - LICENSE](https://img.shields.io/github/license/Luolc/AdaBound.svg?style=flat)](https://raw.githubusercontent.com/Luolc/AdaBound/master/LICENSE)

An optimizer that trains as fast as Adam and as good as SGD, for developing state-of-the-art 
deep learning models on a wide variety of popular tasks in the field of CV, NLP, and etc.

Based on Luo et al. (2019). 
[Adaptive Gradient Methods with Dynamic Bound of Learning Rate](https://openreview.net/forum?id=Bkg3g2R9FX).
In *Proc. of ICLR 2019*.

<p align='center'>
  <img src='https://www.luolc.com/assets/research/adabound/adabound-banner.png' width="60%"/>
</p>

## Quick Links

- [Website](https://www.luolc.com/publications/adabound/)
- [Demos](./demos)

## Installation

AdaBound requires Python 3.6.0 or later.
We currently provide PyTorch version and AdaBound for TensorFlow is coming soon.

### Installing via pip

The preferred way to install AdaBound is via `pip` with a virtual environment.
Just run 
```bash
pip install adabound
```
in your Python environment and you are ready to go!

### Using source code

As AdaBound is a Python class with only 100+ lines, an alternative way is directly downloading
[adabound.py](./adabound/adabound.py) and copying it to your project.

## Usage

You can use AdaBound just like any other PyTorch optimizers.

```python3
optimizer = adabound.AdaBound(model.parameters(), lr=1e-3, final_lr=0.1)
```

As described in the paper, AdaBound is an optimizer that behaves like Adam at the beginning of
training, and gradually transforms to SGD at the end.
The `final_lr` parameter indicates AdaBound would transforms to an SGD with this learning rate.
In common cases, a default final learning rate of `0.1` can achieve relatively good and stable
results on unseen data.
It is not very sensitive to its hyperparameters.
See Appendix G of the paper for more details.

Despite of its robust performance, we still have to state that, **there is no silver bullet**.
It does not mean that you will be free from tuning hyperparameters once using AdaBound.
The performance of a model depends on so many things including the task, the model structure,
the distribution of data, and etc.
**You still need to decide what hyperparameters to use based on your specific situation,
but you may probably use much less time than before!**

## Demos

Thanks to the awesome work by the GitHub team and the Jupyter team, the Jupyter notebook (`.ipynb`)
files can render directly on GitHub.
We provide several notebooks (like [this one](./demos/cifar10/visualization.ipynb)) for better
visualization.
We hope to illustrate the robust performance of AdaBound through these examples.

For the full list of demos, please refer to [this page](./demos).

## Citing
If you use AdaBound in your research, please cite [Adaptive Gradient Methods with Dynamic Bound of Learning Rate](https://openreview.net/forum?id=Bkg3g2R9FX).
```text
@inproceedings{Luo2019AdaBound,
  author = {Luo, Liangchen and Xiong, Yuanhao and Liu, Yan and Sun, Xu},
  title = {Adaptive Gradient Methods with Dynamic Bound of Learning Rate},
  booktitle = {Proceedings of the 7th International Conference on Learning Representations},
  month = {May},
  year = {2019},
  address = {New Orleans, Louisiana}
}
```

## Contributors

[@kayuksel](https://github.com/kayuksel)

## License
[Apache 2.0](./LICENSE)