EmbodiedGPT_Pytorch - GitLite

KO | EN

gitlite — search

#javascript #python #ai #go #golang #android #machine-learning #rust #deep-learning #kubernetes #vue #security #pytorch #ios

EmbodiedGPT_Pytorch

(★ 347)

No description available.

File Explorer

Download Latest Version (.zip)

- imgs
- main_features_embodiedgpt.png
- overall_frame_embodiedgpt.png
- datasets_share.zip
- .DS_Store
- inference.py
- script.py
- test.py
- - __init__.py
  - compression.py
  - configuration_husky.py
  - configuration_husky_ori.py
  - modeling_husky.py
  - modeling_husky_embody2.py
  - modeling_husky_embody2_ori.py
  - processing_husky.py
- - .DS_Store
  - llama_flash_attn_monkey_patch.py
  - llama_rmsnorm_monkey_patch.py
  - train.py
  - train_uni.py
- .DS_Store
- base_dataset.py
- base_dataset_uni.py
- compression.py
- configuration_husky.py
- constants.py
- conversation.py
- convert_fp16.py
- convert_husky_fp16.py
- convert_reward_fp16.py
- dist_utils.py
- llama2_flash_attn_monkey_patch.py
- utils.py
- video_transformers.py
- example_train_file.json
align.sh
INSTALLATION.md
pyproject.toml
README.md
requirements.txt
zero_stage0_config.json
zero_stage1_config.json
zero_stage2_config.json
zero_stage3_config.json

# Installation Guide

1. Get the code

git clone https://github.com/EmbodiedGPT/EmbodiedGPT_Pytorch

Downloads the entire project code from GitHub to your computer.

cd EmbodiedGPT_Pytorch

Moves into the project folder you just downloaded.

2. Python

Easy Recommended

Prerequisites

Git Needed to download the project code from GitHub.
Python 3 On Windows, be sure to check 'Add Python to PATH' during installation.

pip install -r requirements.txt

Installs the Python libraries listed in requirements.txt (or similar).

python <실행할 파일명>.py # README에서 정확한 실행 파일명을 확인하세요

Runs the Python script (or module).

✅ If it runs without errors and prints output in the terminal, it worked.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Embodied Family Code Base

We will update the instructions for this codebase as soon as possible.

## Installation

See [INSTALLATION.md](https://github.com/EmbodiedGPT/EmbodiedGPT_Pytorch/blob/main/INSTALLATION.md)

## Data Preparation

1. Download the [EgoCOT dataset](https://github.com/EmbodiedGPT/EgoCOT_Dataset).
2. Download the [COCO-2017 dataset](https://www.kaggle.com/datasets/awsaf49/coco-2017-dataset).

## Download the Pretrained Model

Download the testing
model [Embodied_family_7btiny](https://huggingface.co/Liang-ZX/Embodied_family_7b/).

## Prepare the Text Data Paired with Video and Image

- Unzip `datasets_share.zip`, which contains the text part of the multi-modal dataset, to the `./datasets/` directory.

## 🏠 Overview

<img width="800" alt="image" src="https://github.com/EmbodiedGPT/EmbodiedGPT_Pytorch/blob/main/assest/overall_frame_embodiedgpt.png">

## 🎁 Major Features

<img width="800" alt="image" src="https://github.com/EmbodiedGPT/EmbodiedGPT_Pytorch/blob/main/assest/main_features_embodiedgpt.png">

## Usage

This repo can be used in conjunction with PyTorch's `Dataset` and `DataLoader` for training models on heterogeneous
data. Here's a brief overview of the classes and their functionalities:

### BaseDataset

The `BaseDataset` class extends PyTorch's `Dataset` and is designed to handle different media types (images, videos, and
text). It includes a transformation process to standardize the input data and a processor to handle the data specific to
the task.

#### Example

```python
from robohusky.base_dataset_uni import BaseDataset

# Initialize the dataset with the required parameters
dataset = BaseDataset(
    dataset,  # Your dataset here
    processor,  # Your processor here
    image_path="path/to/images",
    input_size=224,
    num_segments=8,
    norm_type="openai",
    media_type="image"
)

# Use the dataset with a PyTorch DataLoader
from torch.utils.data import DataLoader

data_loader = DataLoader(dataset, batch_size=32, shuffle=True)
```

### WeightedConcatDataset

The `WeightedConcatDataset` class extends PyTorch's `ConcatDataset` and allows for the creation of a unified dataset by
concatenating multiple datasets with specified weights.

#### Example

```python
from robohusky.base_dataset_uni import WeightedConcatDataset

# Assume we have multiple datasets for different tasks
dataset1 = BaseDataset(...)
dataset2 = BaseDataset(...)
dataset3 = BaseDataset(...)

# Define the weights for each dataset
weights = [0.5, 0.3, 0.2]

# Create a weighted concatenated dataset
weighted_dataset = WeightedConcatDataset([dataset1, dataset2, dataset3], weights=weights)

# Use the weighted dataset with a PyTorch DataLoader
data_loader = DataLoader(weighted_dataset, batch_size=32, shuffle=True)
```

## Customization

The package is designed to be flexible and customizable. You can implement your own transformation and processing logic
by subclassing `BaseDataset` and overriding the necessary methods.

## 🎫 License

This project is released under the [Apache 2.0 license](LICENSE).

## 🖊️ Citation

If you find this project useful in your research, please consider cite:
```bibtex
@article{mu2024embodiedgpt,
  title={Embodiedgpt: Vision-language pre-training via embodied chain of thought},
  author={Mu, Yao and Zhang, Qinglong and Hu, Mengkang and Wang, Wenhai and Ding, Mingyu and Jin, Jun and Wang, Bin and Dai, Jifeng and Qiao, Yu and Luo, Ping},
  journal={Advances in Neural Information Processing Systems},
  volume={36},
  year={2024}
}
```