MDI-Benchmark

KO | EN

gitlite — search

#python #hacktoberfest #react #llm #go #machine-learning #rust #docker #java #nodejs #vue #awesome-list #pytorch #mcp

(★ 14)

No description available.

MDI-Benchmark Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Multi-Dimensional Insights:  Benchmarking Real-World Personalization in Large Multimodal Models

![Dataset](https://img.shields.io/badge/Dataset-MDI--Benchmark-green)
![Multi-Modal](https://img.shields.io/badge/Task-Multi--Modal-red)
![Mathematical Reasoning](https://img.shields.io/badge/Task-Real_world_Reasoning-red)

## Outlines
- [💥 News 💥](README.md#-news-)
- [👀 About MDI-Benchmark]()
- [🏆 Leaderboard on MDI-Benchmark 🏆](https://github.com/MDI-Benchmark/MDI-Benchmark/blob/main/README.md#-leaderboard-on-MDI-Benchmark-)

- [📊 MDI-Benchmark Dataset](https://github.com/MDI-Benchmark/MDI-Benchmark/blob/main/README.md#-MDI-Benchmark-dataset)
- [📜 License](https://github.com/MDI-Benchmark/MDI-Benchmark/blob/main/README.md#-license)
- [🤝 Contributors](https://github.com/MDI-Benchmark/MDI-Benchmark/blob/main/README.md#-contributors)

## 💥 News 💥
  **[2023.12.18]** Our paper is now accessible at https://arxiv.org/abs/2412.12606.

**[2024.12.18]** Our project homepage can be accessed at https://MDI-Benchmark.github.io/.

**[2024.12.18]** Our dataset will be released as open source soon.

## 👀 About MDI-Benchmark
To align with the actual needs of humans for Large Multimodal Models, we propose a multi-modal benchmark for providing a thorough assessment of the capacities of LMMs in practical, real-world scenarios.

<img src="./assets/fig1.png" alt="Overview of MDI-Benchmark" style="width: 95%;"> 
 The overview of the MDI-Benchmark six real-world multimodal scenarios.

The MDI-Benchmark includes over 500 real-world images and 1.2k human-posed questions, spanning six real-world multimodal scenarios. Each scenario is divided into 3 sub-domains with 2 levels of complexity. Additionally, we incorporate age factors into the evaluation to guide LMMs in personalizing their responses for different demographic groups.

<img src="./assets/fig2.png" alt="MDI-Benchmark question" style="width: 95%;"> 
 The MDI-Benchmark includes real needs of different age groups in six major real-world scenarios.

With the MDI-Benchmark, we conduct a comprehensive evaluation of several mainstream LMMs. Specifically, GPT-4o achieved the best results across all indicators, but there is still significant room for improvement in addressing the needs of different age groups. Further analysis across dimensions such as Scenario, Complexity and Age provides valuable insights for developing reliable, personalized human assistants.

<img src="./assets/leaderboard.png" alt="leaderboard" style="width: 95%;"> 
 Performance of the model at different difficulty levels and the overall performance results of the model under the score metric.

We hope our research will advance the application of multimodal large models in real-world scenarios and pave the way for the development of multi-dimensional personalization.

## 🏆 Leaderboard on MDI-Benchmark 🏆
🚨🚨 The [Leaderboard](https://MDI-Benchmark.github.io/#leaderboard) is continuously being updated. We welcome the results of your model!
To submit your results to the leaderboard on the **testmini** subset, please send to [this email](mailto:zhangyifan_ai@bupt.edu.cn) with your result JSON file and score CSV file.

<!-- ## 📝 Evaluation Piplines on MDI-Benchmark

### Response Generation 
The models generate responses based on the given questions and images. Examples for generating responses from some LMMs are provided in the [evaluation](./evaluation). Our prompt specifies the format of answer generation to facilitate subsequent extraction of the answer using string matching. Please refer to the following template to prepare your result JSON files for subsequent evaluation.
```json
{
 "question number": 1,
 "split": "testmini",
 "key": "level1-architecture-old",
 "image_path": "architecture/houseplaning_0",
 "question": "Looking at this house plan, how many bedrooms does this house have?",
 "option": "A.1; B.2; C.3; D.4",
 "answer": "C",
 "response": "<Thought process>: ... <Answer>: ..."
}

```
### Score Calculation
Due to the multiple-choice question format of our dataset and the specific answer generation prompt, we use string matching to directly extract answers, which eliminates the high cost of using additional models for further answer extraction.  The extracted answer is normalized to an option letter and calculate scores on our proposed four-dimensional metrics in [accuracy.py](https://github.com/MDI-Benchmark/MDI-Benchmark/blob/main/evaluation/accuracy.py).

```sh
cd evaluation

python accuracy.py \
--model_name GPT-4o \
--output_json ../output/GPT-4o.json  \
--main_results_csv_path ../result/res.csv
``` -->

## 📊 MDI-Benchmark Dataset

In contrast to existing work, MDI-Benchmark emphasizes the model's performance on real-world problems across various ages and complexities within specific task scenarios, it is structured along three different dimensions: scenario, age, and problem complexity.

### Exmaples
<details>
<summary>🔍Examples of question samples.</summary>

 <img src="./assets/trans.png" width="90%"> 
 Examples of Transport Scenario.

<img src="./assets/edu_level.png" width="90%"> 
	Examples of Education Scenario Level Questions.

<img src="./assets/social_age.png" width="90%"> 
	Example of Social Service Scenario Age Questions.

</details>

## 📜 License

Our dataset are distributed under the [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/) license.

## :white_check_mark: Cite

If you find **MDI-Benchmark** useful for your your research and applications, please kindly cite using this BibTeX:

```bibtex
@misc{zhang2024multi,
      title={Multi-Dimensional Insights: Benchmarking Real-World Personalization in Large Multimodal Models}, 
      author={YiFan Zhang and Shanglin Lei and Runqi Qiao and Zhuoma GongQue and Xiaoshuai Song and Guanting Dong and Qiuna Tan and Zhe Wei and Peiqing Yang and Ye Tian and Yadong Xue and Xiaofei Wang and Honggang Zhang},
      year={2024},
      eprint={2412.12606},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2412.12606}, 
}
```

## 🤝 Contributors
Here are the key contributors to this project:

Yifan Zhang, Shanglin Lei, Runqi Qiao, Zhuoma GongQue, Xiaoshuai Song, Guanting Dong, Qiuna Tan, Zhe Wei, Peiqing Yang, Ye Tian, Xiaofei Wang, Honggang Zhang

MDI-Benchmark

MDI-Benchmark Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

MDI-Benchmark

MDI-Benchmark Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README