frame-predict-VAE-LSTM

KO | EN

gitlite — search

#python #react #llm #go #rust #linux #java #windows #awesome #cli #kubernetes #awesome-list #security #openai

frame-predict-VAE-LSTM

(★ 42)

Predicting image frames using autoencoder and LSTM

frame-predict-VAE-LSTM Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# frame-predict
This project idea is to try predict next n frames, by seeing only first few frames (3 in example)
I took UNet and removed skip connections, I used this architecture only to create encoder and decoder model.

Between encoder and decoder I am using LSTM which acts as a time encoder. 
Time encoder goal is to encoder information about the frames it has seen (like acceleration, position) so it can predict them later. 
LSTM's here are used simialr to seq2seq models.

Because sequence lenghts are variational, they are all stacked in a batch, im using indicies to later
split them apart and pick only the ones I need after using packed LSTM sequences to calculate loss.

Dataset is quite simple, so I wouldn't be suprised if it overfits

## Dataset is sequence of falling dot
![alt text](https://raw.githubusercontent.com/marisancans/frame-predict-VAE-LSTM/master/results/Figure_1.png)

## Architecure design
![alt text](https://raw.githubusercontent.com/marisancans/frame-predict-VAE-LSTM/master/results/Figure_2.png)

# Results:
How to interpret the results: 
![alt text](https://raw.githubusercontent.com/marisancans/frame-predict-VAE-LSTM/master/results/interpret.png)

## Vanilla autoencoder
- Batch size 32
- lr 0.001
- z_vector size 16
- beta 1
![alt text](https://raw.githubusercontent.com/marisancans/frame-predict-VAE-LSTM/master/results/vanilla.gif)

## Vanilla autoencoder z_vector size 2
- Batch size 32
- lr 0.001
- z_vector size 2
- beta 1
![alt text](https://raw.githubusercontent.com/marisancans/frame-predict-VAE-LSTM/master/results/vanilla_zsize2.gif)

## VAE autoencoder
- Batch size 32
- lr 0.001
- z_vector size 16
- beta 1
![alt text](https://raw.githubusercontent.com/marisancans/frame-predict-VAE-LSTM/master/results/vae.gif)

## Beta VAE autoencoder
- Batch size 32
- lr 0.001
- z_vector size 3
- beta 150
![alt text](https://raw.githubusercontent.com/marisancans/frame-predict-VAE-LSTM/master/results/vae_beta.gif)

frame-predict-VAE-LSTM

frame-predict-VAE-LSTM Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

frame-predict-VAE-LSTM

frame-predict-VAE-LSTM Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README