Mapping the Field of Algorithm Auditing: A Systematic Literature Review Identifying Research Trends, Linguistic and Geographical Disparities

Aleksandra Urman,Mykola Makhortykh,Aniko Hannak

2024-01-20

Abstract:The increasing reliance on complex algorithmic systems by online platforms has sparked a growing need for algorithm auditing, a research methodology evaluating these systems' functionality and societal impact. In this paper, we systematically review algorithm auditing studies and identify trends in their methodological approaches, the geographic distribution of authors, and the selection of platforms, languages, geographies, and group-based attributes in the focus of auditing research. We present evidence of a significant skew of research focus toward Western contexts, particularly the US, and a disproportionate reliance on English language data. Additionally, our analysis indicates a tendency in algorithm auditing studies to focus on a narrow set of group-based attributes, often operationalized in simplified ways, which might obscure more nuanced aspects of algorithmic bias and discrimination. By conducting this review, we aim to provide a clearer understanding of the current state of the algorithm auditing field and identify gaps that need to be addressed for a more inclusive and representative research landscape.

Human-Computer Interaction

What problem does this paper attempt to address?

The issues this paper attempts to address are as follows:

1. **Systematic Literature Review**: The paper updates and expands the understanding of the field of algorithm auditing through a systematic literature review. Specifically, it reviews 128 related research papers and points out imbalances in previous literature, such as the choice of problem domains and platforms.

2. **Geographic and Language Diversity**: The paper finds that algorithm auditing research is heavily biased towards Western countries (especially the United States) and English data. Additionally, the group attributes focused on in the research (such as gender and race) are often overly simplified, potentially obscuring the complexity of algorithmic bias and discrimination.

3. **Researcher Distribution**: The paper also explores the geographic distribution of algorithm auditing researchers, finding that these researchers are primarily from the United States or a few European countries, which does not fully align with the geographic bias of the research focus.

4. **Attention to Specific Issues**: The paper further analyzes the attention to specific issues in algorithm auditing research, finding that most studies focus on distortion issues, followed by discrimination. The search domain is the most frequently audited area, while some popular platforms (such as Instagram and TikTok) are under-researched.

In summary, this paper aims to reveal the current research trends, geographic and language disparities in the field of algorithm auditing, and points out areas for improvement to achieve a more inclusive and representative research landscape.