MoSEAR

KO | EN

gitlite — search

#python #hacktoberfest #react #typescript #go #golang #machine-learning #nodejs #java #awesome #cli #kubernetes #security #openai

(★ 40)

Benchmarking and Bridging Emotion Conflicts for Multimodal Emotion Reasoning (ACM MM 2025 Oral)

MoSEAR 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Benchmarking and Bridging Emotion Conflicts for Multimodal Emotion Reasoning

[![Project Page](https://img.shields.io/badge/Project-Page-green?logo=googlechrome)](https://zhiyuanhan-aaron.github.io/MoSEAR-page/)
[![arXiv](https://img.shields.io/badge/arXiv-2508.01181-b31b1b.svg)](https://arxiv.org/abs/2508.01181)
[![Conference](https://img.shields.io/badge/ACM%20MM-2025-blue)](https://2025.acmmm.org/)
[![Model](https://img.shields.io/badge/🤗%20HuggingFace-Model-yellow)](https://huggingface.co/AaronHan/MoSEAR)
[![YouTube](https://img.shields.io/badge/YouTube-Video-red?logo=youtube)](https://www.youtube.com/watch?v=zkI9cYC5v08)

</div>

---

## 📑 Table of Contents

- [News](#-news)
- [Overview](#-overview)
- [CA-MER Benchmark](#-ca-mer-benchmark)
- [MoSEAR Framework](#-mosear-framework)
- [Setup](#%EF%B8%8F-setup)
- [Training](#-training)
- [Inference & Evaluation](#-inference--evaluation)
- [Acknowledgements](#-acknowledgements)
- [Citation](#%EF%B8%8F-citation)
- [License](#-license)
- [Contact](#-contact)

---

## 📢 News
- **[2024.10]** 🚀 Code and CA-MER benchmark released!
- **[2025.06]** 🎉 Our paper has been accepted by **ACM MM 2025** and recommended for **Oral Presentation**!

## 🚀 Overview

Despite strong performance in multimodal emotion reasoning, existing Multimodal Large Language Models (MLLMs) often overlook scenarios involving **emotion conflicts**, where emotional cues from different modalities are inconsistent. This limitation is problematic because humans naturally express emotions inconsistently, and evaluations reveal that current models systematically **over-rely on the audio signal** during these conflicts, neglecting critical cues from the visual modality.

To fill this gap, we:
1. Introduce **CA-MER**, a new benchmark designed to examine MLLMs under realistic emotion conflicts, consisting of video-aligned, audio-aligned, and consistent subsets.
2. Propose **MoSEAR**, a parameter-efficient framework that promotes balanced modality integration through:
   - **MoSE** (Modality-Specific Experts): Reduces modality bias in fine-tuning heads with a regularized gating mechanism
   - **AR** (Attention Reallocation): Rebalances modality contributions in frozen backbones during inference without trade-offs

Experiments on multiple benchmarks demonstrate that **MoSEAR achieves state-of-the-art performance**, particularly under modality conflict conditions.

For more details, please refer to our [paper](https://arxiv.org/abs/2508.01181).

---

## 📊 CA-MER Benchmark

CA-MER (Conflict-Aware Multimodal Emotion Reasoning) is a novel benchmark specifically designed to evaluate MLLMs under emotion conflicts.

### Dataset Structure
- **Annotation Files**: Available in `CA-MER/`
- **Video Samples**: Constructed from MER2023 dataset (videos can be retrieved using sample names)

### Sample Format
```json
{
    "file_name": "sample_00000002",
    "video_pseudo_label": "worried",
    "video_analysis": "The main character exhibits worried and sad emotions...",
    "audio_pseudo_label": "angry",
    "audio_analysis": "The speaker's tone is raised and forceful...",
    "emotionllama_pseudo_label": "angry",
    "final_label": "angry",
    "subtitle": "I grew up, and it wasn't easy for me either...",
    "reasoning": "The visual clue reflects sadness and worry..."
}
```

### Benchmark Construct Pipeline

---

## 🧠 MoSEAR Framework

### Analyses of modality bias in emotion conflicts
<p align="center">
  <img src="./images/img_attn_analysis_00.png" width="90%">
</p>

### Modality-Specific Experts (MoSE)
MoSE introduces modality-aware parameter-efficient fine-tuning to reduce audio bias:
- Shared projection matrices for efficient training
- Expert-gating mechanism for modality-specific learning
- Regularization to prevent over-specialization

**Implementation**: [`minigpt4/peft/tuners/MoSE.py`](minigpt4/peft/tuners/MoSE.py)

### Attention Reallocation (AR)

AR dynamically rebalances attention weights during inference:
- Layer-wise balance ratio computation
- Adaptive head selection based on modality dominance
- Zero-parameter inference-time optimization

**Implementation**: [`minigpt4/models/attention_reallocation.py`](minigpt4/models/attention_reallocation.py)

---

## 🛠️ Setup

Our work builds upon [Emotion-LLaMA](https://github.com/ZebangCheng/Emotion-LLaMA). Thanks to their detailed open-source contributions!

---

## 💡 Training

### 1. Download Dataset
Due to copyright restrictions, we cannot provide raw videos. Please visit the official MER2023 website to apply for dataset access:
> http://merchallenge.cn/datasets

Then specify the dataset path in the [config file](minigpt4/configs/datasets/firstface/featureface.yaml):

```yaml
# Set dataset video path
image_path: /path/to/datasets/Emotion/MER2023/video
```

### 2. Prepare Multimodal Features
To save GPU memory, we use pre-extracted features from multiple encoders:
- **Audio**: HuBERT-large
- **Global Visual**: EVA-CLIP
- **Local Visual**: MAE
- **Temporal**: VideoMAE

Download pre-extracted features from [Google Drive](https://drive.google.com/drive/folders/1DqGSBgpRo7TuGNqMJo9BYg6smJE20MG4) and update the feature paths in [`minigpt4/datasets/datasets/first_face.py`](minigpt4/datasets/datasets/first_face.py).

### 3. Configure Dataset
In [`minigpt4/configs/datasets/firstface/featureface.yaml`](minigpt4/configs/datasets/firstface/featureface.yaml), select `MERR_coarse_grained.txt` (28,618 samples for Stage 1).

### 4. Multi-Task Learning Setup
Our framework supports multi-task learning for emotion recognition and reasoning.

Configure task types in [`minigpt4/datasets/datasets/first_face.py`](minigpt4/datasets/datasets/first_face.py):

```python
self.task_pool = [
    "emotion",      # Multimodal emotion recognition
    "reason",       # Emotion reasoning (MERR-coarse)
    # "reason_v2",  # Emotion reasoning (MERR-fine) - for Stage 2
]
```

**Training Stages**:
- **Stage 1**: Use `emotion` + `reason`
- **Stage 2**: Use `emotion` + `reason_v2`

Retrieve multimodal descriptions:
```python
caption = self.fine_grained_dict[video_name]['smp_reason_caption']
```

### 5. Enable MoSE
Configure MoSE in [`train_configs/stage1_finetune.yaml`](train_configs/stage1_finetune.yaml):

```yaml
choose_lora_config: 'SM3LoraConfig'  # Enable MoSE
```

### 6. Run Training
Execute training scripts for both stages:

```bash
# Stage 1: Coarse-grained training
bash scripts/train_stage1.sh

# Stage 2: Fine-grained fine-tuning
bash scripts/train_stage2.sh
```

**Note**: Use the checkpoint from the last epoch of each stage.

---

## 🧪 Inference & Evaluation

### Model Checkpoints
Download our trained models: [🤗 Hugging Face](https://huggingface.co/AaronHan/MoSEAR)

### MER2023 Evaluation
Evaluate emotion recognition performance on MER2023:

1. Configure checkpoint path in [`eval_configs/eval_emotion_MER23.yaml`](eval_configs/eval_emotion_MER23.yaml)
2. Run evaluation:
```bash
bash scripts/mer2023_test.sh
```

This will compute **Accuracy** and **F1 Score**.

### CA-MER & EMER Evaluation

#### Step 1: Configure Dataset
Select dataset subset in [`eval_configs/eval_emotion_CAMER.yaml`](eval_configs/eval_emotion_CAMER.yaml):
- CA-MER subsets (video-aligned, audio-aligned, consistent)
- EMER dataset

#### Step 2: Enable/Disable Attention Reallocation (AR)
Configure AR in [`eval_emotion_EMER.py`](eval_emotion_EMER.py):

```python
# Enable AR (recommended for conflict scenarios)
llama_modify(
    model.llama_model.base_model.model,
    start_layer=0,
    end_layer=32,
    enable_balance=True,  # Set to False to disable AR
    alpha=0.4,
    use_cfg=False,
    img_start_idx=8,
    img_end_idx=267,
)
```

#### Step 3: Generate Predictions
Run emotion reasoning evaluation:
```bash
bash scripts/emer_test.sh
```

This generates a CSV file with multimodal emotion reasoning results.

#### Step 4: Extract Open-Ended Emotion Vocabulary
Use LLM to generate open-ended emotion vocabulary:
```bash
python emer_evaluation/get_ov1.py
```

#### Step 5: Compute Metrics
Calculate set-level accuracy and recall:
```bash
python emer_evaluation/emer_ov.py
```

**Note**: Ground truth emotion vocabulary can also be processed using `get_ov.py`.

---

## 🙏 Acknowledgements

We thank the following projects for their excellent open-source contributions:

- [Emotion-LLaMA](https://arxiv.org/abs/2406.11161): Multimodal Emotion Large Language Model
- [MiniGPT-v2](https://arxiv.org/abs/2310.09478): Unified Vision-Language Interface
- [AffectGPT](https://arxiv.org/abs/2306.15401): Explainable Multimodal Emotion Recognition

---

## ✒️ Citation

If you find our work helpful, please consider citing:

```bibtex
@article{han2025benchmarking,
  title={Benchmarking and Bridging Emotion Conflicts for Multimodal Emotion Reasoning},
  author={Han, Zhiyuan and Zhu, Beier and Xu, Yanlong and Song, Peipei and Yang, Xun},
  journal={arXiv preprint arXiv:2508.01181},
  year={2025}
}
```

---

## 📜 License

This project is released under the **BSD 3-Clause License**. See [LICENSE.md](./LICENSE.md) for details.

---

## 📧 Contact

For questions or collaboration, please feel free to contact:
- Zhiyuan Han: [aaronhan@mail.ustc.edu.cn]

MoSEAR

MoSEAR 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

MoSEAR

MoSEAR 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README