videoMultiGAN

KO | EN

gitlite — search

#php #java #javascript #ruby #python #hacktoberfest #wordpress #nodejs #shell #android #html #edger

★ 66 Open GitHub ↗

End to End learning for Video Generation from Text

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Video Multi GAN
Video Generation from Text using Tree like decision using GANs. The text annotation or statement is encoded using the LM into a embedding, which then is combined with random vector to generate relevant videos and images.

## Video Generation models
1. VAEGAN
2. VAEGAN with Latent Variable optimization
3. VAEGAN with anti reconstruction loss
4. VAEGAN + Anti reconstruction loss + Latent variable models
5. variants of above models with different Hyper parameters

## Model structure
* LSTM based model for next frame creation
* Wasserstein GAN setting discriminator 
* Word embedding based LM
* Attention based model for classification structure

## Training model
* The relevant models are in ``` Tensorflow >= v1.2 ```
* Experimentation with above mentioned models
* The training is done over self generated Bouncing MNIST with sentence based annotation
* The gensim pre trained fastText wikipedia work embeddings are used for embedding tokens as vectors
* Non attention based models are used initially to generate starting frames. 
* The GAN tree trains to look for discriminative features (unverified)

## Datasets
1. UCF101 : 3 channel image
2. Bouncing MNIST

## Documentation
1. We use Sync-DRAW to develop our datasets (https://github.com/syncdraw/Sync-DRAW)
2. UCF101 is available from University of Montreal
3. We use multiple GPU training (or a single K80 or Titan X)
4. Cluster traning is impossible for now

### Results will not be updated here since there might be related publications.

videoMultiGAN

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README