Abstract:Social media platforms curate access to information and opportunities, and so play a critical role in shaping public discourse today. The opaque nature of the algorithms these platforms use to curate content raises societal questions. Prior studies have used black-box methods led by experts or collaborative audits driven by everyday users to show that these algorithms can lead to biased or discriminatory outcomes. However, existing auditing methods face fundamental limitations because they function independent of the platforms. Concerns of potential harmful outcomes have prompted proposal of legislation in both the U.S. and the E.U. to mandate a new form of auditing where vetted external researchers get privileged access to social media platforms. Unfortunately, to date there have been no concrete technical proposals to provide such auditing, because auditing at scale risks disclosure of users' private data and platforms' proprietary algorithms. We propose a new method for platform-supported auditing that can meet the goals of the proposed legislation. The first contribution of our work is to enumerate the challenges and the limitations of existing auditing methods to implement these policies at scale. Second, we suggest that limited, privileged access to relevance estimators is the key to enabling generalizable platform-supported auditing of social media platforms by external researchers. Third, we show platform-supported auditing need not risk user privacy nor disclosure of platforms' business interests by proposing an auditing framework that protects against these risks. For a particular fairness metric, we show that ensuring privacy imposes only a small constant factor increase (6.34x as an upper bound, and 4× for typical parameters) in the number of samples required for accurate auditing. Our technical contributions, combined with ongoing legal and policy efforts, can enable public oversight into how social media platforms affect individuals and society by moving past the privacy-vs-transparency hurdle.

A CI-based Auditing Framework for Data Collection Practices

Privacy Checklist: Privacy Violation Detection Grounding on Contextual Integrity Theory

Ethical Considerations Related to Personal Data Collection and Reuse: Trust and Transparency in Language and Speech Technologies

CI-Bench: Benchmarking Contextual Integrity of AI Assistants on Synthetic Data

Operationalizing Contextual Integrity in Privacy-Conscious Assistants

Auditing AI for Whom? A Community-Centric Approach to Rebuilding Public Trust in Singapore

DiffAudit: Auditing Privacy Practices of Online Services for Children and Adolescents

Discovering Smart Home Internet of Things Privacy Norms Using Contextual Integrity

Auditing of AI: Legal, Ethical and Technical Approaches

A Framework for Assurance Audits of Algorithmic Systems

Cognitive Tracing Data Trails: Auditing Data Provenance in Discriminative Language Models Using Accumulated Discrepancy Score

From Transparency to Accountability and Back: A Discussion of Access and Evidence in AI Auditing

FIAT: Fine-grained Information Audit for Trustless Transborder Data Flow

Implanting Life-Cycle Privacy Policies in a Context Database

Security Clouds to Improve Privacy and Conduct Continuous Audits in 6G Networked Environments Smart Cities

Consortium Blockchain-Based Public Integrity Verification in Cloud Storage for IoT

Towards Compliance and Accountability: a Framework for Privacy Online.

Ethics-Based Auditing to Develop Trustworthy AI

CIPPPA: Conditional Identity Privacy-Preserving Public Auditing for Cloud-Based WBANs Against Malicious Auditors

A Multi-Disciplinary Perspective for Conducting Artificial Intelligence-enabled Privacy Analytics

Having your Privacy Cake and Eating it Too: Platform-supported Auditing of Social Media Algorithms for Public Interest