KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#typescript #ai-agents #developer-tools #claude-code #codex #cli #agent-skills #ai #dsh-plugin #mcp #windows #agent

AWESOME-MER

★ 126 Open GitHub ↗

🔆 📝 A reading list focused on Multimodal Emotion Recognition (MER) 👂👄 👀 💬

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# AWESOME-MER

:memo: A reading list focused on Multimodal Emotion Recognition (MER)  :ear: :lips: :eyes: :speech_balloon:

（:white_small_square: indicates a specific modality）

***

:high_brightness: [Datasets](#datasets)

:high_brightness: [Challenges](#challenges)

:high_brightness: [Projects](#projects)

:high_brightness: [Related Reviews](#related-reviews)

:high_brightness: [Multimodal Emotion Recognition (MER)](#multimodal-emotion-recognition)

## Datasets

- (2018) [CMU-MOSEI](https://github.com/A2Zadeh/CMU-MultimodalSDK)[:white_small_square:Visual:white_small_square:Audio:white_small_square:Language]
- (2018) [ASCERTAIN Dataset](http://mhug.disi.unitn.it/wp-content/ASCERTAIN/ascertain.html)[:white_small_square:Facial activity data:white_small_square:Physiological data]
- (2017) [EMOTIC Dataset](http://sunai.uoc.edu/emotic/)[:white_small_square:Face:white_small_square:Context]
- (2016) [Multimodal Spontaneous Emotion Database (BP4D+)](http://www.cs.binghamton.edu/~lijun/Research/3DFE/3DFE_Analysis.html)[:white_small_square:Face:white_small_square:Thermal data:white_small_square:Physiological data]
- (2016) [EmotiW Database](https://sites.google.com/view/emotiw2020)[:white_small_square:Visual:white_small_square:Audio]
- (2015) [LIRIS-ACCEDE Database](https://liris-accede.ec-lyon.fr/)[:white_small_square:Visual:white_small_square:Audio]
- (2014) [CREMA-D](https://github.com/CheyneyComputerScience/CREMA-D)[:white_small_square:Visual:white_small_square:Audio]
- (2013) [SEMAINE Database](https://ibug.doc.ic.ac.uk/resources/semaine-database2/)[:white_small_square:Visual:white_small_square:Audio:white_small_square:Conversation transcripts]
- (2011) [MAHNOB-HCI](https://mahnob-db.eu/hci-tagging/)[:white_small_square:Visual:white_small_square:Eye gaze:white_small_square:Physiological data]
- (2008) [IEMOCAP Database](https://sail.usc.edu/iemocap/)[:white_small_square:Visual:white_small_square:Audio:white_small_square:Text transcripts]
- (2005) [eNTERFACE Dataset](http://enterface.net/)[:white_small_square:Visual:white_small_square:Audio]

## Challenges

- [Multimodal (Audio, Facial and Gesture) based Emotion Recognition Challenge (MMER) @ FG](https://icv.tuit.ut.ee/challenge/)
- [Emotion Recognition in the Wild Challenge (EmotiW) @ ICMI](https://sites.google.com/view/emotiw2018)
- [Audio/Visual Emotion Challenge (AVEC) @ ACM MM](https://sites.google.com/view/avec2019/home?authuser=0)
- [One-Minute Gradual-Emotion Behavior Challenge @ IJCNN](https://www2.informatik.uni-hamburg.de/wtm/OMG-EmotionChallenge/)
- [Multimodal Emotion Recognition Challenge (MEC) @ ACII](http://www.chineseldc.org/htdocsEn/emotion.html) 
- [Multimodal Pain Recognition (Face and Body) Challenge (EmoPain) @ FG](https://mvrjustid.github.io/EmoPainChallenge2020/)

## Projects

- [CMU Multimodal SDK](https://github.com/A2Zadeh/CMU-MultimodalSDK)
- [Real-Time Multimodal Emotion Recognition](https://github.com/maelfabien/Multimodal-Emotion-Recognition) 
- [MixedEmotions Toolbox](https://github.com/MixedEmotions/MixedEmotions)
- [End-to-End Multimodal Emotion Recognition](https://github.com/tzirakis/Multimodal-Emotion-Recognition)

## Related Reviews

- (IEEE Journal of Selected Topics in Signal Processing20) Multimodal Intelligence: Representation Learning,
Information Fusion, and Applications [[paper](https://arxiv.org/pdf/1911.03977.pdf)]
- (Information Fusion20) A snapshot research and implementation of multimodal information fusion for data-driven emotion recognition [[paper](https://www.sciencedirect.com/science/article/pii/S1566253519301381)]
- (Information Fusion17) A review of affective computing: From unimodal analysis to multimodal fusion [[paper](https://ww.sentic.net/affective-computing-review.pdf)] 
- (Image and Vision Computing17) A survey of multimodal sentiment analysis [[paper](https://ibug.doc.ic.ac.uk/media/uploads/documents/multi_modal.pdf)] 
- (ACM Computing Surveys15) A Review and Meta-Analysis of Multimodal Affect Detection Systems [[paper](https://dl.acm.org/doi/10.1145/2682899)]

## Multimodal Emotion Recognition

### :small_orange_diamond: CVPR

- (2020) EmotiCon: Context-Aware Multimodal Emotion Recognition using Frege’s Principle [[paper](https://openaccess.thecvf.com/content_CVPR_2020/papers/Mittal_EmotiCon_Context-Aware_Multimodal_Emotion_Recognition_Using_Freges_Principle_CVPR_2020_paper.pdf)]

[:white_small_square:Faces/Gaits :white_small_square:Background :white_small_square:Social interactions]

- (2017) Emotion Recognition in Context [[paper](https://openaccess.thecvf.com/content_cvpr_2017/papers/Kosti_Emotion_Recognition_in_CVPR_2017_paper.pdf)]

[:white_small_square:Face :white_small_square:Context]

### :small_orange_diamond: ICCV

- (2019) Context-Aware Emotion Recognition Networks [[paper](https://openaccess.thecvf.com/content_ICCV_2019/papers/Lee_Context-Aware_Emotion_Recognition_Networks_ICCV_2019_paper.pdf)]

[:white_small_square:Faces :white_small_square:Context]

- (2017) A Multimodal Deep Regression Bayesian Network for Affective Video Content Analyses [[paper](https://openaccess.thecvf.com/content_ICCV_2017/papers/Gan_A_Multimodal_Deep_ICCV_2017_paper.pdf)]

[:white_small_square:Visual :white_small_square:Audio]

### :small_orange_diamond: AAAI

- (2020) M3ER: Multiplicative Multimodal Emotion Recognition Using Facial, Textual, and Speech Cues [[paper](https://arxiv.org/pdf/1911.05659.pdf)]

[:white_small_square:Face :white_small_square:Speech :white_small_square:Text ]

- (2020) An End-to-End Visual-Audio Attention Network for Emotion Recognition in User-Generated Videos [[paper](https://aaai.org/Papers/AAAI/2020GB/AAAI-ZhaoS.7155.pdf)]

[:white_small_square:Visual :white_small_square:Audio ]

- (2019) Multi-Interactive Memory Network for Aspect Based Multimodal Sentiment Analysis [[paper](https://www.aaai.org/ojs/index.php/AAAI/article/view/3807)]

[:white_small_square:Visual :white_small_square:Text ]

- (2019) VistaNet: Visual Aspect Attention Network for Multimodal Sentiment Analysis [[paper](https://www.aaai.org/ojs/index.php/AAAI/article/view/3799)]

[:white_small_square:Visual :white_small_square:Text ]

- (2019) Cooperative Multimodal Approach to Depression Detection in Twitter [[paper](https://www.aaai.org/ojs/index.php/AAAI/article/view/3775)]

[:white_small_square:Visual :white_small_square:Text ]

- (2014) Predicting Emotions in User-Generated Videos [[paper](http://www.yugangjiang.info/publication/aaai14-emotions.pdf)]

[:white_small_square:Visual :white_small_square:Audio :white_small_square:Attribute ]

### :small_orange_diamond: IJCAI

- (2019) DeepCU: Integrating both Common and Unique Latent Information for Multimodal Sentiment Analysis [[paper](https://arxiv.org/pdf/1911.05659.pdf)]

[:white_small_square:Face :white_small_square:Audio :white_small_square:Text ]

- (2019) Adapting BERT for Target-Oriented Multimodal Sentiment Classification [[paper](https://www.ijcai.org/Proceedings/2019/0751.pdf)]

[:white_small_square:Image :white_small_square:Text ]

- (2018) Personality-Aware Personalized Emotion Recognition from Physiological Signals [[paper](https://www.ijcai.org/Proceedings/2018/0230.pdf)]

[:white_small_square:Personality :white_small_square: Physiological signals ]

- (2015) Combining Eye Movements and EEG to Enhance Emotion Recognition  [[paper](https://www.ijcai.org/Proceedings/15/Papers/169.pdf)]

[:white_small_square:EEG :white_small_square:Eye movements ]

### :small_orange_diamond: ACM MM

- (2019) Emotion Recognition using Multimodal Residual LSTM Network  [[paper](https://haotang1995.github.io/files/ACM-MM-19.pdf)]

[:white_small_square:EEG :white_small_square:Other physiological signals ]

- (2019) Mutual Correlation Attentive Factors in Dyadic Fusion Networks for Speech Emotion Recognition [[paper](https://dl.acm.org/doi/10.1145/3343031.3351039)]

[:white_small_square:Audio:white_small_square: Text]

- (2019) Multimodal Deep Denoise Framework for Affective Video Content Analysis [[paper](https://dl.acm.org/doi/10.1145/3343031.3350997)]

[:white_small_square:Face :white_small_square:Body gesture:white_small_square:Voice:white_small_square: Physiological signals]

### :small_orange_diamond: WACV

- (2016) Multimodal emotion recognition using deep learning architectures [[paper](https://ieeexplore.ieee.org/document/7477679)]

[:white_small_square:Visual :white_small_square:Audio]

### :small_orange_diamond: FG

- (2020) Multimodal Deep Learning Framework for Mental Disorder Recognition  [[paper](https://www.cl.cam.ac.uk/~mmam3/pub/FG2020_Multimodal_Deep_Learning_Framework_for_Mental_Disorder_Recognition.pdf)]

[:white_small_square:Visual :white_small_square:Audio :white_small_square:Text]

- (2019) Multi-Attention Fusion Network for Video-based Emotion Recognition [[paper](https://dl.acm.org/doi/abs/10.1145/3340555.3355720?download=true)]

[:white_small_square:Visual :white_small_square:Audio]

- (2019) Audio-Visual Emotion Forecasting: Characterizing and Predicting Future Emotion Using Deep Learning  [[paper](https://ieeexplore.ieee.org/document/8756599)]

[:white_small_square:Face :white_small_square:Speech]

### :small_orange_diamond: ICMI

- (2018) Multimodal Local-Global Ranking Fusion for Emotion Recognition [[paper](https://arxiv.org/abs/1809.04931)]

[:white_small_square:Visual :white_small_square:Audio ]

- (2017) Emotion recognition with multimodal features and temporal models [[paper](https://dl.acm.org/doi/10.1145/3136755.3143016)]

[:white_small_square:Visual :white_small_square:Audio ]

- (2017) Modeling Multimodal Cues in a Deep Learning-Based Framework for Emotion Recognition in the Wild [[paper](https://dl.acm.org/doi/abs/10.1145/3136755.3143006)]

[:white_small_square:Visual :white_small_square:Audio ]

### :small_orange_diamond: IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)

- (2020) Context Based Emotion Recognition using EMOTIC Dataset  [[paper](https://arxiv.org/abs/2003.13401)]

[:white_small_square:Face :white_small_square:Context]

### IEEE Transactions on Circuits and Systems for Video Technology

- (2018) Learning Affective Features With a Hybrid Deep Model for Audio–Visual Emotion Recognition [[paper](https://ieeexplore.ieee.org/document/7956190)]

[:white_small_square:Visual :white_small_square:Audio ]

### :small_orange_diamond:  IEEE Transactions on Cybernetics

- (2020) Emotion Recognition From Multimodal Physiological Signals Using a Regularized Deep Fusion of Kernel Machine [[paper](https://ieeexplore.ieee.org/document/9093122)]

[:white_small_square:EEG :white_small_square:Other physiological signals ]

- (2019) EmotionMeter: A Multimodal Framework for Recognizing Human Emotions  [[paper](https://ieeexplore.ieee.org/document/8283814)]

[:white_small_square:EEG :white_small_square:Eye movements]

- (2015) Temporal Bayesian Fusion for Affect Sensing: Combining Video, Audio, and Lexical Modalities [[paper](https://ieeexplore.ieee.org/document/6930787)]

[:white_small_square:Face :white_small_square:Audio:white_small_square:Lexical features]

### :small_orange_diamond: IEEE Transactions on Multimedia

- (2020) Visual-Texual Emotion Analysis With Deep Coupled Video and Danmu Neural Networks [[paper](https://arxiv.org/abs/1811.07485)]

[:white_small_square:Visual:white_small_square:Text]

- (2020) Locally Confined Modality Fusion Network With a Global Perspective for Multimodal Human Affective Computing [[paper](https://ieeexplore.ieee.org/document/8752006)]

[:white_small_square:Visual:white_small_square:Audio:white_small_square:Language]

- (2019) Metric Learning-Based Multimodal Audio-Visual Emotion Recognition [[paper](https://ieeexplore.ieee.org/document/8935376)]

[:white_small_square:Visual:white_small_square:Audio]

- (2019) Knowledge-Augmented Multimodal Deep Regression Bayesian Networks for Emotion Video Tagging [[paper](https://ieeexplore.ieee.org/document/8794750)]

[:white_small_square:Visual:white_small_square:Audio:white_small_square:Attribute]

- (2018) Multimodal Framework for Analyzing the Affect of a Group of People  [[paper](https://ieeexplore.ieee.org/document/8323249)]

[:white_small_square:Face:white_small_square:Upper body:white_small_square: Scene]

- (2012) Kernel Cross-Modal Factor Analysis for Information Fusion With Application to Bimodal Emotion Recognition [[paper](https://ieeexplore.ieee.org/document/6161652)]

[:white_small_square:Visual:white_small_square:Audio]

### :small_orange_diamond: IEEE Transactions on Affective Computing

- (2019) Audio-Visual Emotion Recognition in Video Clips  [[paper](https://ieeexplore.ieee.org/document/7945502)]

[:white_small_square:Visual :white_small_square:Audio]

- (2019) Recognizing Induced Emotions of Movie Audiences From Multimodal Information [[paper]()]

[:white_small_square:Visual :white_small_square:Audio :white_small_square:Dialogue:white_small_square:Attribute]

- (2019) EmoBed: Strengthening Monomodal Emotion Recognition via Training with Crossmodal Emotion Embeddings [[paper](https://arxiv.org/abs/1907.10428)]

[:white_small_square:Face :white_small_square:Audio]

- (2018) Combining Facial Expression and Touch for Perceiving Emotional Valence [[paper](https://ieeexplore.ieee.org/document/7752812)]

[:white_small_square:Face :white_small_square:Touch stimuli]

- (2018) A Combined Rule-Based & Machine Learning Audio-Visual Emotion Recognition Approach [[paper](https://www.computer.org/csdl/journal/ta/2018/01/07506248/13rRUxBa5lU)]

[:white_small_square:Visual :white_small_square:Audio]

- (2016) Analysis of EEG Signals and Facial Expressions for Continuous Emotion Detection [[paper](https://ieeexplore-ieee-org.eproxy.lib.hku.hk/document/7112127)]

[:white_small_square:Face :white_small_square:EEG signals]

- (2013) Exploring Cross-Modality Affective Reactions for Audiovisual Emotion Recognition [[paper](https://ieeexplore.ieee.org/document/6507534)]

[:white_small_square:Face :white_small_square:Audio]

- (2012) Multimodal Emotion Recognition in Response to Videos  [[paper](https://ieeexplore.ieee.org/document/6095505)]

[:white_small_square:Eye gaze :white_small_square:EEG signals]

- (2012) Context-Sensitive Learning for Enhanced Audiovisual Emotion Classification [[paper](https://ieeexplore.ieee.org/document/7344611)]

[:white_small_square:Visual :white_small_square:Audio :white_small_square:Utterance]

- (2011) Continuous Prediction of Spontaneous Affect from Multiple Cues and Modalities in Valence-Arousal Space  [[paper](https://ieeexplore.ieee.org/document/5740839)]

[:white_small_square:Face :white_small_square: Shoulder gesture:white_small_square:Audio]

### :small_orange_diamond: Neurocomputing

- (2020) Joint low rank embedded multiple features learning for audio–visual emotion recognition [[paper](https://www.sciencedirect.com/science/article/abs/pii/S092523122030045X)]

[:white_small_square:Visual :white_small_square:Audio]

- (2018) Multi-cue fusion for emotion recognition in the wild [[paper](https://www.sciencedirect.com/science/article/abs/pii/S0925231218304867)]

[:white_small_square:Visual :white_small_square:Audio]

- (2018) Multi-modality weakly labeled sentiment learning based on Explicit Emotion Signal for Chinese microblog [[paper](https://www.sciencedirect.com/science/article/abs/pii/S0925231217312298)]

[:white_small_square:Visual :white_small_square:Text]

- (2016) Fusing audio, visual and textual clues for sentiment analysis from multimodal content [[paper](https://www.sciencedirect.com/science/article/abs/pii/S0925231215011297)]

[:white_small_square:Visual :white_small_square:Audio:white_small_square: Text]

### :small_orange_diamond: Information Fusion

- (2019) Affective video content analysis based on multimodal data fusion in heterogeneous networks [[paper](https://www.sciencedirect.com/science/article/abs/pii/S1566253518307309)]

[:white_small_square:Visual :white_small_square:Audio]

- (2019) Audio-visual emotion fusion (AVEF): A deep efficient weighted approach [[paper](https://www.sciencedirect.com/science/article/abs/pii/S1566253518300733)]

[:white_small_square:Visual :white_small_square:Audio]

### :small_orange_diamond: Neural Networks

- (2015) Towards an intelligent framework for multimodal affective data analysis [[paper](https://www.sciencedirect.com/science/article/abs/pii/S0893608014002342)]

[:white_small_square:Visual :white_small_square:Audio:white_small_square: Text]

- (2015) Multimodal emotional state recognition using sequence-dependent deep hierarchical features [[paper](sciencedirect.com/science/article/pii/S0893608015001847)]

[:white_small_square:Face :white_small_square:Upper-body]

### :small_orange_diamond: Others

- (Knowledge-Based Systems 2018) Multimodal sentiment analysis using hierarchical fusion with context modeling [[paper](https://arxiv.org/abs/1806.06228)]

[:white_small_square:Visual :white_small_square:Audio:white_small_square: Text]

- (IEEE Journal of Selected Topics in Signal Processing 2017) End-to-End Multimodal Emotion Recognition Using Deep Neural Networks [[paper](https://arxiv.org/pdf/1704.08619.pdf)]

[:white_small_square:Visual :white_small_square:Audio]

- (Computer Vision and Image Understanding 2016) Multi-modal emotion analysis from facial expressions and electroencephalogram [[paper](https://www.sciencedirect.com/science/article/abs/pii/S1077314215002106)]

[:white_small_square:Face :white_small_square:EEG]

AWESOME-MER

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README