Multi-modal Fusion for Continuous Emotion Recognition by Using Auto-Encoders - CEA - Commissariat à l’énergie atomique et aux énergies alternatives Access content directly
Conference Papers Year : 2021

Multi-modal Fusion for Continuous Emotion Recognition by Using Auto-Encoders

Salam Hamieh
  • Function : Correspondent author
  • PersonId : 1122569

Connectez-vous pour contacter l'auteur
Vincent Heiries
  • Function : Author
  • PersonId : 948855
Christelle Godin
  • Function : Author
  • PersonId : 859057


Human stress detection is of great importance for monitoring mental health. The Multimodal Sentiment Analysis Challenge (MuSe) 2021 focuses on emotion, physiological-emotion, and stress recognition as well as sentiment classification by exploiting several modalities. In this paper, we present our solution for the Muse-Stress sub-challenge. The target of this sub-challenge is continuous prediction of arousal and valence for people under stressful conditions where text transcripts and audio and video recordings are provided. To this end, we utilize bidirectional Long Short-Term Memory (LSTM) and Gated Recurrent Unit networks (GRU) to explore high-level and low-level features from different modalities. We employ Concordance Correlation Coefficient (CCC) as a loss function and evaluation metric for our model. To improve the unimodal predictions, we add difficulty indicators of the data obtained by using Auto-Encoders. Finally, we perform late fusion on our unimodal predictions in addition to the difficulty indicators to obtain our final predictions. With this approach, we achieve CCC of 0.4278 and 0.5951 for arousal and valence respectively, our submission to MuSe 2021 ranks in the top three for arousal and fourth for valence.
Fichier principal
Vignette du fichier
Hamieh et al. - 2021 - Multi-modal Fusion for Continuous Emotion Recognit.pdf (1.28 Mo) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

cea-03517175 , version 1 (07-01-2022)




Salam Hamieh, Vincent Heiries, Hussein Al Osman, Christelle Godin. Multi-modal Fusion for Continuous Emotion Recognition by Using Auto-Encoders. MM '21: ACM Multimedia Conference, Oct 2021, Virtual Event China, France. pp.21-27, ⟨10.1145/3475957.3484455⟩. ⟨cea-03517175⟩
92 View
103 Download



Gmail Mastodon Facebook X LinkedIn More