Emotion-Driven Music Recommendation System Using Multimodal Data

Recommending music that matches or regulates a listener's emotional state requires knowing that state first, and most commercial recommenders skip this step entirely, relying instead on listening history and collaborative filtering that say nothing about how the listener feels right now. This paper presents a multimodal emotion-driven recommendation system that infers a listener's affective state from three signals captured during a session: facial expression from the device camera, sentiment extracted from any text the listener enters (search queries, chat, or captions), and, where available, vocal tone from a short voice sample. Each modality is scored independently by a lightweight classifier and the three scores are combined through a confidence-weighted late fusion step, so a modality that is unavailable or low-confidence in a given session, such as a blank camera feed, does not silently distort the result. The fused output is a position on a two-dimensional valence-arousal plane rather than a single discrete label, which is then matched against a track catalogue annotated with the same valence-arousal coordinates derived from audio features such as tempo, energy, and mode. We describe the fusion architecture, the mapping from emotional state to track features, and a feedback loop that adjusts future recommendations when a listener skips a suggested track quickly. We evaluate the system on a constructed listening-session dataset for classification accuracy per modality, fusion accuracy against self-reported mood, and recommendation acceptance rate compared to a collaborative-filtering baseline. Index Terms - Emotion Recognition, Multimodal Fusion, Music Recommendation, Valence-Arousal Model, Affective Computing, Session-Based Feedback

Keywords - Emotion Recognition, Multimodal Fusion, Music Recommendation, Valence-Arousal Model, Affective Computing, Session-Based Feedback