transformer-lm-gan

KO | EN

gitlite — search

#hacktoberfest #llm #go #rust #docker #macos #windows #awesome #cli #kubernetes #vue #pytorch #security #ios

transformer-lm-gan

(★ 41)

Explorations into adversarial losses on top of autoregressive loss for language modeling

transformer-lm-gan Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

## Language model with adversarial loss

Explorations into adversarial losses on top of autoregressive loss for language modeling

I have tried this in the past, when GANs were still dominant. But at the time I was either too inexperienced or the research not there. Either way could not get it working. Will give it another shot in the next few weeks, mainly to see if an adversarial system could benefit [world modeling](https://github.com/lucidrains/improving-transformers-world-model-for-rl)

## Usage

```python
import torch

from transformer_lm_gan import (
    LanguageModelGenerator,
    Discriminator,
    GAN,
)

gan = GAN(
    strategy = 'gumbel_one_hot', # or 'rotate' for rotation trick, may try combination of two if both fails in experiments
    generator = dict(
        num_tokens = 256,
        dim = 512,
        depth = 6,
        dim_head = 64,
        heads = 8,
        max_seq_len = 1024
    ),
    discriminator = dict(
        num_tokens = 256,
        dim = 512,
        depth = 2,
        dim_head = 64,
        heads = 9,
        max_seq_len = 1024
    )
).cuda()

seq = torch.randint(0, 256, (2, 1024)).cuda()

discr_loss = gan.discriminate_forward(seq)
discr_loss.backward()

gen_loss = gan.generate_forward(seq)
gen_loss.backward()
```

## Citations

```bibtex
@inproceedings{Huang2025TheGI,
    title   = {The GAN is dead; long live the GAN! A Modern GAN Baseline},
    author  = {Yiwen Huang and Aaron Gokaslan and Volodymyr Kuleshov and James Tompkin},
    year    = {2025},
    url     = {https://api.semanticscholar.org/CorpusID:275405495}
}
```

```bibtex
@misc{sarkar2025evolutionstrategieshyperscale,
    title     = {Evolution Strategies at the Hyperscale}, 
    author    = {Bidipta Sarkar and Mattie Fellows and Juan Agustin Duque and Alistair Letcher and Antonio León Villares and Anya Sims and Dylan Cope and Jarek Liesen and Lukas Seier and Theo Wolf and Uljad Berdica and Alexander David Goldie and Aaron Courville and Karin Sevegnani and Shimon Whiteson and Jakob Nicolaus Foerster},
    year      = {2025},
    eprint    = {2511.16652},
    archivePrefix = {arXiv},
    primaryClass = {cs.LG},
    url       = {https://arxiv.org/abs/2511.16652}, 
}
```

transformer-lm-gan

transformer-lm-gan Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

transformer-lm-gan

transformer-lm-gan Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README