Abstract:Social media platforms curate access to information and opportunities, and so play a critical role in shaping public discourse today. The opaque nature of the algorithms these platforms use to curate content raises societal questions. Prior studies have used black-box methods led by experts or collaborative audits driven by everyday users to show that these algorithms can lead to biased or discriminatory outcomes. However, existing auditing methods face fundamental limitations because they function independent of the platforms. Concerns of potential harmful outcomes have prompted proposal of legislation in both the U.S. and the E.U. to mandate a new form of auditing where vetted external researchers get privileged access to social media platforms. Unfortunately, to date there have been no concrete technical proposals to provide such auditing, because auditing at scale risks disclosure of users' private data and platforms' proprietary algorithms. We propose a new method for platform-supported auditing that can meet the goals of the proposed legislation. The first contribution of our work is to enumerate the challenges and the limitations of existing auditing methods to implement these policies at scale. Second, we suggest that limited, privileged access to relevance estimators is the key to enabling generalizable platform-supported auditing of social media platforms by external researchers. Third, we show platform-supported auditing need not risk user privacy nor disclosure of platforms' business interests by proposing an auditing framework that protects against these risks. For a particular fairness metric, we show that ensuring privacy imposes only a small constant factor increase (6.34x as an upper bound, and 4× for typical parameters) in the number of samples required for accurate auditing. Our technical contributions, combined with ongoing legal and policy efforts, can enable public oversight into how social media platforms affect individuals and society by moving past the privacy-vs-transparency hurdle.

Trustless Audits without Revealing Data or Models

Auditing and Generating Synthetic Data with Controllable Trust Trade-offs

Under manipulations, are some AI models harder to audit?

Reclaiming the Digital Commons: A Public Data Trust for Training Data

Towards Trustworthy LLMs for Code: A Data-Centric Synergistic Auditing Framework

Theoretical Guarantees for Model Auditing with Finite Adversaries

Building Trust: Foundations of Security, Safety and Transparency in AI

Mathematical Algorithm Design for Deep Learning under Societal and Judicial Constraints: The Algorithmic Transparency Requirement

Auditing Visualizations: Transparency Methods Struggle to Detect Anomalous Behavior

A General Framework for Data-Use Auditing of ML Models

Insights into Data through Model Behaviour: An Explainability-driven Strategy for Data Auditing for Responsible Computer Vision Applications

Balancing Transparency and Risk: The Security and Privacy Risks of Open-Source Machine Learning Models

Black-Box Access is Insufficient for Rigorous AI Audits

Optimized Scoring Systems: Toward Trust in Machine Learning for Healthcare and Criminal Justice

Towards algorithm auditing: managing legal, ethical and technological risks of AI, ML and associated algorithms

Having your Privacy Cake and Eating it Too: Platform-supported Auditing of Social Media Algorithms for Public Interest

Transparency and the Black Box Problem: Why We Do Not Trust AI

TRUCE: Private Benchmarking to Prevent Contamination and Improve Comparative Evaluation of LLMs

Blockchain Enabled Trustless API Marketplace

The Right Tool for the Job: Open-Source Auditing Tools in Machine Learning

Towards algorithms and models that we can trust: A theoretical perspective