KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#dsh #dsh-plugin #ai-agents #developer-tools #typescript #codex #cli #claude-code #mcp #rust #deepseek-harness #ai-agent

Image-text-video-I2VGENXL

★ 14 Open GitHub ↗

A PyTorch implementation of a text-image to video diffussion model with a cascaded architecture I2VGEN-XL

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# I2VGEN-XL: Image + Text to Video Diffusion Model

I2VGEN-XL is a Cascaded Diffussion model designed for generating videos from image and text inputs. It employs a cascaded architecture, leveraging diffusion processes to produce high-quality and coherent video outputs. This repository provides an easy-to-use Jupyter Notebook to get started with I2VGEN-XL.

<a href='https://i2vgen-xl.github.io/'><img src='https://img.shields.io/badge/Project-Page-Green'></a> <a href='https://arxiv.org/abs/2311.04145'><img src='https://img.shields.io/badge/Paper-Arxiv-red'></a> [![Paper page](https://huggingface.co/datasets/huggingface/badges/resolve/main/paper-page-sm-dark.svg)](https://huggingface.co/papers/2311.04145) 
[![Replicate](https://replicate.com/cjwbw/i2vgen-xl/badge)](https://replicate.com/cjwbw/i2vgen-xl/)

## Features

- **Multimodal Inputs**: Combines images and text to create dynamic video outputs.
- **Cascaded Architecture**: Utilizes a hierarchical approach to ensure progressively refined video quality.
- **Lightweight Setup**: Run directly in Jupyter Notebook with minimal setup requirements.
  
  
  | **Model Architecture** |
  | -------------- |
  | ![Modelfigure](https://raw.githubusercontent.com/MOSTAFA1172m/Image-text-video-I2VGENXL/main/results/i2vgenxl-model-figure.gif) |
  | **Description**:The I2VGen-XL framework consists of two main stages. In the base stage, hierarchical encoders work together to extract both high-level semantics and fine-grained details from input images, ensuring realistic motion while maintaining the content and structure of the images. In the refinement stage, a dedicated diffusion model enhances resolution and significantly improves temporal consistency by refining finer details. The "D.Enc." and "G.Enc." components represent the detail encoder and global encoder, respectively.|

---

## Getting Started

### Requirements

- A Kaggle account for running the notebook on Kaggle’s GPU-enabled environment.
- Basic familiarity with Python and Jupyter Notebooks.

### Steps

1. **Download the Notebook**:

- Clone the repository or download the `IVGEN-XL.ipynb` file directly.
     ```bash
     git clone https://github.com/MOSTAFA1172m/Image-text-video-IVGENXL.git
     cd IVGEN-XL
     ```

2. **Upload the Notebook to Kaggle**:

- Go to [Kaggle](https://www.kaggle.com/).
   - Create a new notebook and upload the `IVGEN-XL.ipynb` file.

3. **Enable GPU**:

- In your Kaggle notebook, navigate to **Settings** > **Accelerator** and select **GPU**.

4. **Install Dependencies**:

- Run the following command in a code cell:
     ```python
     !pip install -r requirements.txt
     ```

5. **Run the Notebook**:

- Follow the step-by-step instructions provided in the notebook to input images and text, and generate videos.

## Results

### Example Outputs

Here are some example outputs generated by the model. The results showcase how the model processes various inputs:

#### Image + Text Input Examples:

| **Example 1** | **Example 2** |
| -------------- | -------------- |
| ![Newton portrait](https://raw.githubusercontent.com/MOSTAFA1172m/Image-text-video-I2VGENXL/main/results/Newton.gif) | ![Mona Lisa painting by Leonardo da Vinci](https://raw.githubusercontent.com/MOSTAFA1172m/Image-text-video-I2VGENXL/main/results/monalisa.gif) |
| **Description**: Newton smiling and waving. | **Description**: Monalisa laughing. |

| **Example 3** | **Example 4** |
| -------------- | -------------- |
| ![car](https://raw.githubusercontent.com/MOSTAFA1172m/Image-text-video-I2VGENXL/main/results/car.gif) | ![Sunset Sea](https://raw.githubusercontent.com/MOSTAFA1172m/Image-text-video-I2VGENXL/main/results/sea.gif) |
| **Description**: Car driving on the road.  | **Description**: Sunset over the sea. |

| **Example 5** |
| -------------- |
| ![Skyward clouds](https://raw.githubusercontent.com/MOSTAFA1172m/Image-text-video-I2VGENXL/main/results/sky.gif) |
| **Description**: Clouds moving across the sky over the mountains.|

---

### Generation Speed

On Kaggle, you can use GPU for faster results.

Feel free to experiment with different inputs and see how the model generates videos in response.

## License

This project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.

## Contact

For any questions or feedback, feel free to reach out to [my linkedin](https://www.linkedin.com/in/mostafa-hazem-961931294/).

---

*Note: Ensure the GPU runtime is enabled before running the notebook to avoid performance issues.*

Image-text-video-I2VGENXL

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README