lookahead_pytorch

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #typescript #llm #android #linux #docker #nodejs #windows #awesome #kubernetes #ios

lookahead_pytorch

(★ 194)

pytorch implement of Lookahead Optimizer

파일 탐색기

최종 버전 다운로드 (.zip)

- loss.png
- valid_acc.png
- valid_loss.png
LICENSE
nn.py
optimizer.py
progressbar.py
README.md
run.py
tools.py
trainingmonitor.py

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

## Lookahead Pytorch

This repository contains a PyTorch implementation of the Lookahead Optimizer from the paper

[Lookahead Optimizer: k steps forward, 1 step back](https://arxiv.org/abs/1907.08610)

by Michael R. Zhang, James Lucas, Geoffrey Hinton and Jimmy Ba.

## Dependencies

* PyTorch
* torchvision
* matplotlib

## Usage

The code in this repository implements both Lookahead and Adam training, with examples on the CIFAR-10 datasets.

To use Lookahead use the following command.

```python
from optimizer import Lookahead
optimizer = optim.Adam(model.parameters(), lr=0.001)
optimizer = Lookahead(optimizer=optimizer,k=5,alpha=0.5)
```

We found that evaluation performance is typically better using the slow weights. This can be done in PyTorch with something like this in your eval loop:
```python
if args.lookahead:
    optimizer._backup_and_load_cache()
    val_loss = eval_func(model)
    optimizer._clear_and_load_backup()
```
## Example

To produce th result,we use CIFAR-10 dataset for ResNet18.

```python
# use adam
python run.py --optimizer=adam

# use lookahead 
python run.py --optimizer=lookahead
```
## Results

Train loss of adam and lookahead with ResNet18 on CIFAR-10.

![](https://raw.githubusercontent.com/lonePatient/lookahead_pytorch/master/png/loss.png)

Valid loss of adam and lookahead with ResNet18 on CIFAR-10.

![](https://raw.githubusercontent.com/lonePatient/lookahead_pytorch/master/png/valid_loss.png)

Valid accuracy of adam and lookahead with ResNet18 on CIFAR-10.

![](https://raw.githubusercontent.com/lonePatient/lookahead_pytorch/master/png/valid_acc.png)

lookahead_pytorch

파일 탐색기

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

lookahead_pytorch

파일 탐색기

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README