pause-transformer

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #golang #android #machine-learning #java #awesome-list #vue #pytorch #openai #mcp

pause-transformer

(★ 53)

Yet another random morning idea to be quickly tried and architecture shared if it works; to allow the transformer to pause for any amount of time on any token

pause-transformer Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

## Pause Transformer (wip)

Yet another random morning idea to be quickly tried and architecture shared if it works; to allow the transformer to pause for any amount of time on any token.

Again, the idea relies on axial attention; one axis attends along the sequence length as in the usual transformer, the other along a `thinking` or `pause` dimension.

## Todo

- [x] allow for custom pause distributions across token
- [x] see if one can do a two pass, using the logit entropy as a way to decide how to shape the pause mask

- [ ] run experiments on enwik8, but if do not see anything, move onwards to something harder, say arithmetic

## Citations

```bibtex
@inproceedings{Goyal2023ThinkBY,
    title   = {Think before you speak: Training Language Models With Pause Tokens},
    author  = {Sachin Goyal and Ziwei Ji and Ankit Singh Rawat and Aditya Krishna Menon and Sanjiv Kumar and Vaishnavh Nagarajan},
    year    = {2023},
    url     = {https://api.semanticscholar.org/CorpusID:263608983}
}
```

pause-transformer

pause-transformer Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

pause-transformer

pause-transformer Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README