Multimodal Aspect Based Sentiment Analysis with Emotion Fusion in Social Discourse

Aspect Based Sentiment Analysis (ABSA) offers fine-grained sentiment detection by identifying opinions tied to specific aspects within text. However, most existing ABSA models are unimodal, relying solely on textual data and struggling with subtle expressions like sarcasm or weak opinions, which is common in real-world discourse. This limitation highlights the value of incorporating non-verbal cues, such as facial emotions, to better capture the emotional context.

This work presents a novel multimodal aspect based sentiment analysis model that integrates textual embeddings from MaskedABSA with facial emotion features extracted using EMO-AffectNet, which are then passed through a temporal model. This enhances sentiment understanding by aligning semantic and affective cues. A custom-curated dataset centered on the Black Lives Matter vs. All Lives Matter discourse is introduced, annotated using a structured codebook to provide stance- specific supervision.

Compared to unimodal and a state of the art vision language baseline, the pro- posed model demonstrates consistent improvements in both accuracy and F1 Score, achieving a relative increase of approximately 11% in accuracy and over 10% in F1 score compared to a text only ABSA model. These results underscore the effective- ness of multimodal integration for accurate stance detection and its potential for promoting ethical, context aware recommendation systems