Safety and Fairness for Content Moderation in Generative Models

Susan Hao,Piyush Kumar,Sarah Laszlo,Shivani Poddar,Bhaktipriya Radharapu,Renee Shelby
DOI: https://doi.org/10.48550/arXiv.2306.06135
IF: 5.414
2023-06-09
Machine Learning
Abstract:With significant advances in generative AI, new technologies are rapidly being deployed with generative components. Generative models are typically trained on large datasets, resulting in model behaviors that can mimic the worst of the content in the training data. Responsible deployment of generative technologies requires content moderation strategies, such as safety input and output filters. Here, we provide a theoretical framework for conceptualizing responsible content moderation of text-to-image generative technologies, including a demonstration of how to empirically measure the constructs we enumerate. We define and distinguish the concepts of safety, fairness, and metric equity, and enumerate example harms that can come in each domain. We then provide a demonstration of how the defined harms can be quantified. We conclude with a summary of how the style of harms quantification we demonstrate enables data-driven content moderation decisions.
What problem does this paper attempt to address?