pause-transformer

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #llm #android #machine-learning #deep-learning #linux #macos #windows #awesome #cli #mcp

pause-transformer

(★ 53)

Yet another random morning idea to be quickly tried and architecture shared if it works; to allow the transformer to pause for any amount of time on any token

pause-transformer 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

## Pause Transformer (wip)

Yet another random morning idea to be quickly tried and architecture shared if it works; to allow the transformer to pause for any amount of time on any token.

Again, the idea relies on axial attention; one axis attends along the sequence length as in the usual transformer, the other along a `thinking` or `pause` dimension.

## Todo

- [x] allow for custom pause distributions across token
- [x] see if one can do a two pass, using the logit entropy as a way to decide how to shape the pause mask

- [ ] run experiments on enwik8, but if do not see anything, move onwards to something harder, say arithmetic

## Citations

```bibtex
@inproceedings{Goyal2023ThinkBY,
    title   = {Think before you speak: Training Language Models With Pause Tokens},
    author  = {Sachin Goyal and Ziwei Ji and Ankit Singh Rawat and Aditya Krishna Menon and Sanjiv Kumar and Vaishnavh Nagarajan},
    year    = {2023},
    url     = {https://api.semanticscholar.org/CorpusID:263608983}
}
```

pause-transformer

pause-transformer 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

pause-transformer

pause-transformer 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README