Abstract:Sentiment analysis can be used to study an individual or a group’s emotions and attitudes towards other people and entities like products, services, or social events. With the advancements in the field of deep learning, the enormity of available information on internet, chiefly on social media, combined with powerful computing machines, it’s just a matter of time before artificial intelligence (AI) systems make their presence in every aspect of human life, making our lives more introspective. In this paper, we propose to implement a multimodal sentiment prediction system that can analyze the emotions predicted from different modal sources such as video, audio and text and integrate them to recognize the group emotions of the students in a classroom. Our experimental setup involves a digital video camera with microphones to capture the live video and audio feeds of the students during a lecture. The students are advised to provide their digital feedback on the lecture as ‘tweets’ on their twitter account addressed to the lecturer’s official twitter account. The audio and video frames are separated from the live streaming video using tools such as lame and ffmpeg. A twitter API was used to access and extract messages from twitter platform. The audio and video features are extracted using Mel-Frequency Cepstral Co-efficients (MFCC) and Haar Cascades classifier respectively. The extracted features are then passed to the Convolutional Neural Network (CNN) model trained on the FER2013 facial images database to generate the feature vector for classification of video-based emotions. A Recurrent Neural Network (RNN) with Long Short-Term Memory (LSTM), trained on speech emotion corpus database was used to train on the audio features. A lexicon-based approach with senti-word dictionary and learning based approach with custom dataset trained by Support Vector Machines (SVM) was used in the twitter-texts based approach. A decision-level fusion algorithm was applied on these three different modal schemes to integrate the classification results and deduce the overall group emotions of the students. The use-case of this proposed system will be in student emotion recognition, employee performance feedback, monitoring or surveillance-based systems. The implemented system framework was tested in a classroom environment during a live lecture and the predicted emotions demonstrated the classification accuracy of our approach.

Multiclass mood classification on Twitter using lexicon dictionary and machine learning algorithms

An Emotion based Sentiment Analysis on Twitter Dataset

Multi-class Emotion AI by reconstructing linguistic context of words

Twitter Sentiment Analysis: Lexicon Method, Machine Learning Method and Their Combination

A lexicon-based approach for sentiment analysis of multimodal content in tweets

Lexicon-Based Sentiment Analysis on Text Polarities with Evaluation of Classification Models

Emotion Detection and Analysis on Social Media

Machine Learning Technique to Detect and Classify Mental Illness on Social Media Using Lexicon-Based Recommender System

Multimodal Decision-level Group Sentiment Prediction of Students in Classrooms

A reliable sentiment analysis for classification of tweets in social networks

Harnessing Twitter for Automatic Sentiment Identification Using Machine Learning Techniques

Sentiment Analysis on Twitter through Machine Learning: A Comprehensive Approach with User-Centric Visualisations

Tweets Sentiment Analysis via Word Embeddings and Machine Learning Techniques

Twitter Sentiment Analysis Using Textual Information and Diffusion Patterns

Twitter sentiment analysis using various classification algorithms

Twitter Sentiment Analysis System

Multi-level aspect based sentiment classification of Twitter data: using hybrid approach in deep learning

Fine-Tuning BERT Based Approach for Multi-Class Sentiment Analysis on Twitter Emotion Data

Evaluation of adjective and adverb types for effective Twitter sentiment classification

Lexicon-Based Approach to Sentiment Analysis of Tweets Using R Language

Climate Change Sentiment Analysis Using Lexicon, Machine Learning and Hybrid Approaches