Abstract:In the evolving landscape of computer vision (CV) technologies, the automatic detection and interpretation of gender and emotion in images is a critical area of study. This paper investigates social biases in CV models, emphasizing the limitations of traditional evaluation metrics such as precision, recall, and accuracy. These metrics often fall short in capturing the complexities of gender and emotion, which are fluid and culturally nuanced constructs. Our study proposes a sociotechnical framework for evaluating CV models, incorporating both technical performance measures and considerations of social fairness. Using a dataset of 5,570 images related to vaccination and climate change, we empirically compared the performance of various CV models, including traditional models like DeepFace and FER, and generative models like GPT-4 Vision. Our analysis involved manually validating the gender and emotional expressions in a subset of images to serve as benchmarks. Our findings reveal that while GPT-4 Vision outperforms other models in technical accuracy for gender classification, it exhibits discriminatory biases, particularly in response to transgender and non-binary personas. Furthermore, the model's emotion detection skew heavily towards positive emotions, with a notable bias towards associating female images with happiness, especially when prompted by male personas. These findings underscore the necessity of developing more comprehensive evaluation criteria that address both validity and discriminatory biases in CV models. Our proposed framework provides guidelines for researchers to critically assess CV tools, ensuring their application in communication research is both ethical and effective. The significant contribution of this study lies in its emphasis on a sociotechnical approach, advocating for CV technologies that support social good and mitigate biases rather than perpetuate them.

Ranking Computer Vision Service Issues using Emotion

Real-Time Svm-Based Emotion Recognition Algorithm

Performance Analysis and Evaluation of Cloud Vision Emotion APIs

Interpreting Cloud Computer Vision Pain-Points: A Mining Study of Stack Overflow

Emotion Classification: How Does an Automated System Compare to Naive Human Coders?

Emotion Classification In Software Engineering Texts: A Comparative Analysis of Pre-trained Transformers Language Models

Emotion detection of social data: APIs comparative study

Computer Vision Estimation of Emotion Reaction Intensity in the Wild

Uncovering the Causes of Emotions in Software Developer Communication Using Zero-shot LLMs

SVM Point-Based Real-Time Emotion Detection

Study of Impact of Computer Vision in Detecting Human Emotions

EmotioNet Challenge: Recognition of facial expressions of emotion in the wild

A Gold Standard for Emotion Annotation in Stack Overflow

Human Face Emotion in 3D Using Machine Learning

Perspective on Emotion Detection for Automotive Applications: Performance Evaluation of Two Emotion AI SDKs

Online Learning for Wearable EEG-Based Emotion Classification

A Sociotechnical Lens for Evaluating Computer Vision Models: A Case Study on Detecting and Reasoning about Gender and Emotion

Emotional Images: Assessing Emotions in Images and Potential Biases in Generative Models

Automatic Facial Expression Recognition in Standardized and Non-standardized Emotional Expressions

Emotion Detection in Reddit: Comparative Study of Machine Learning and Deep Learning Techniques

Emotion Recognition Using WT-SVM in Human-Computer Interaction