Algorithmic bias and the Value Sensitive Design approach

Judith Simon,Pak-Hang Wong,Gernot Rieder

DOI: https://doi.org/10.14763/2020.4.1534

2020-12-18

Internet Policy Review

Abstract:This article provides an overview of the Value Sensitive Design (VSD) methodology and explores how it can enrich the current debates on algorithmic bias and fairness in machine learning.

What problem does this paper attempt to address?

Algorithmic bias: review, synthesis, and future research directions

Nima Kordzadeh,Maryam Ghasemaghaei

DOI: https://doi.org/10.1080/0960085X.2021.1927212

2021-06-08

European Journal of Information Systems

Abstract:As firms are moving towards data-driven decision making, they are facing an emerging problem, namely, algorithmic bias. Accordingly, algorithmic systems can yield socially-biased outcomes, thereby compounding inequalities in the workplace and in society. This paper reviews, summarises, and synthesises the current literature related to algorithmic bias and makes recommendations for future information systems research. Our literature analysis shows that most studies have conceptually discussed the ethical, legal, and design implications of algorithmic bias, whereas only a limited number have empirically examined them. Moreover, the mechanisms through which technology-driven biases translate into decisions and behaviours have been largely overlooked. Based on the reviewed papers and drawing on theories such as the stimulus-organism-response theory and organisational justice theory, we identify and explicate eight important theoretical concepts and develop a research model depicting the relations between those concepts. The model proposes that algorithmic bias can affect fairness perceptions and technology-related behaviours such as machine-generated recommendation acceptance, algorithm appreciation, and system adoption. The model also proposes that contextual dimensions (i.e., individual, task, technology, organisational, and environmental) can influence the perceptual and behavioural manifestations of algorithmic bias. These propositions highlight the significant gap in the literature and provide a roadmap for future studies.

information science & library science,management,computer science, information systems
Situated algorithms: a sociotechnical systemic approach to bias

Claude Draude,Goda Klumbyte,Phillip Lücking,Pat Treusch

DOI: https://doi.org/10.1108/oir-10-2018-0332

2019-11-12

Online Information Review

Abstract:Purpose The purpose of this paper is to propose that in order to tackle the question of bias in algorithms, a systemic, sociotechnical and holistic perspective is needed. With reference to the term “algorithmic culture,” the interconnectedness and mutual shaping of society and technology are postulated. A sociotechnical approach requires translational work between and across disciplines. This conceptual paper undertakes such translational work. It exemplifies how gender and diversity studies, by bringing in expertise on addressing bias and structural inequalities, provide a crucial source for analyzing and mitigating bias in algorithmic systems. Design/methodology/approach After introducing the sociotechnical context, an overview is provided regarding the contemporary discourse around bias in algorithms, debates around algorithmic culture, knowledge production and bias identification as well as common solutions. The key concepts of gender studies (situated knowledges and strong objectivity) and concrete examples of gender bias then serve as a backdrop for revisiting contemporary debates. Findings The key concepts reframe the discourse on bias and concepts such as algorithmic fairness and transparency by contextualizing and situating them. The paper includes specific suggestions for researchers and practitioners on how to account for social inequalities in the design of algorithmic systems. Originality/value A systemic, gender-informed approach for addressing the issue is provided, and a concrete, applicable methodology toward a situated understanding of algorithmic bias is laid out, providing an important contribution for an urgent multidisciplinary dialogue.

information science & library science,computer science, information systems
Mapping value sensitive design onto AI for social good principles

Steven Umbrello,Ibo van de Poel

DOI: https://doi.org/10.1007/s43681-021-00038-3

2021-02-01

AI and Ethics

Abstract:Abstract Value sensitive design (VSD) is an established method for integrating values into technical design. It has been applied to different technologies and, more recently, to artificial intelligence (AI). We argue that AI poses a number of challenges specific to VSD that require a somewhat modified VSD approach. Machine learning (ML), in particular, poses two challenges. First, humans may not understand how an AI system learns certain things. This requires paying attention to values such as transparency, explicability, and accountability. Second, ML may lead to AI systems adapting in ways that ‘disembody’ the values embedded in them. To address this, we propose a threefold modified VSD approach: (1) integrating a known set of VSD principles (AI4SG) as design norms from which more specific design requirements can be derived; (2) distinguishing between values that are promoted and respected by the design to ensure outcomes that not only do no harm but also contribute to good, and (3) extending the VSD process to encompass the whole life cycle of an AI technology to monitor unintended value consequences and redesign as needed. We illustrate our VSD for AI approach with an example use case of a SARS-CoV-2 contact tracing app.
Guidelines for Integrating Value Sensitive Design in Responsible AI Toolkits

Malak Sadek,Marios Constantinides,Daniele Quercia,Céline Mougenot

2024-03-01

Abstract:Value Sensitive Design (VSD) is a framework for integrating human values throughout the technology design process. In parallel, Responsible AI (RAI) advocates for the development of systems aligning with ethical values, such as fairness and transparency. In this study, we posit that a VSD approach is not only compatible, but also advantageous to the development of RAI toolkits. To empirically assess this hypothesis, we conducted four workshops involving 17 early-career AI researchers. Our aim was to establish links between VSD and RAI values while examining how existing toolkits incorporate VSD principles in their design. Our findings show that collaborative and educational design features within these toolkits, including illustrative examples and open-ended cues, facilitate an understanding of human and ethical values, and empower researchers to incorporate values into AI systems. Drawing on these insights, we formulated six design guidelines for integrating VSD values into the development of RAI toolkits.

Human-Computer Interaction
LUCID: Exposing Algorithmic Bias through Inverse Design

Carmen Mazijn,Carina Prunkl,Andres Algaba,Jan Danckaert,Vincent Ginis

DOI: https://doi.org/10.48550/arXiv.2208.12786

2022-08-27

Abstract:AI systems can create, propagate, support, and automate bias in decision-making processes. To mitigate biased decisions, we both need to understand the origin of the bias and define what it means for an algorithm to make fair decisions. Most group fairness notions assess a model's equality of outcome by computing statistical metrics on the outputs. We argue that these output metrics encounter intrinsic obstacles and present a complementary approach that aligns with the increasing focus on equality of treatment. By Locating Unfairness through Canonical Inverse Design (LUCID), we generate a canonical set that shows the desired inputs for a model given a preferred output. The canonical set reveals the model's internal logic and exposes potential unethical biases by repeatedly interrogating the decision-making process. We evaluate LUCID on the UCI Adult and COMPAS data sets and find that some biases detected by a canonical set differ from those of output metrics. The results show that by shifting the focus towards equality of treatment and looking into the algorithm's internal workings, the canonical sets are a valuable addition to the toolbox of algorithmic fairness evaluation.

Machine Learning,Data Structures and Algorithms
Algorithmic bias: Looking beyond data bias to ensure algorithmic accountability and equity

Savannah Thais,Hannah Shumway,Austin Saragih

DOI: https://doi.org/10.38105/spr.5lwvw66ssy

2023-08-31

MIT Science Policy Review

Abstract:As algorithms increasingly aid public sector decision making in the United States, it becomes important to understand how to effectively tackle algorithmic bias in systems that local, state, and federal government entities use and procure, including what kinds of policies are currently in place or proposed. There is a prevalent belief that algorithmic bias arises primarily from statistical biases present in the data used to train or develop the algorithm, but bias may also arise during data collection, problem specification, how and where algorithms are deployed, and within the broader societal contextualization of algorithms. So far, enacted policies in in the US center on temporary bans of particular types of algorithms, transparency, and post-hoc bias audits, as well as more wide-ranging (but non-binding) policy frameworks; all largely focus on quantitative notions of fairness when they assess bias, leaving room for more comprehensive legislation to meaningfully address this issue going forward.
Bias, Fairness and Accountability with Artificial Intelligence and Machine Learning Algorithms

Nengfeng Zhou,Zach Zhang,Vijayan N. Nair,Harsh Singhal,Jie Chen

DOI: https://doi.org/10.1111/insr.12492

2022-04-10

International Statistical Review

Abstract:SummaryThe advent of artificial intelligence (AI) and machine learning algorithms has led to opportunities as well as challenges in their use. In this overview paper, we begin with a discussion of bias and fairness issues that arise with the use of AI techniques, with a focus on supervised machine learning algorithms. We then describe the types and sources of data bias and discuss the nature of algorithmic unfairness. In addition, we provide a review of fairness metrics in the literature, discuss their limitations, and describe de‐biasing (or mitigation) techniques in the model life cycle.

statistics & probability
Fairness and Bias in Algorithmic Hiring: a Multidisciplinary Survey

Alessandro Fabris,Nina Baranowska,Matthew J. Dennis,David Graus,Philipp Hacker,Jorge Saldivar,Frederik Zuiderveen Borgesius,Asia J. Biega

DOI: https://doi.org/10.1145/3696457

2024-09-25

Abstract:Employers are adopting algorithmic hiring technology throughout the recruitment pipeline. Algorithmic fairness is especially applicable in this domain due to its high stakes and structural inequalities. Unfortunately, most work in this space provides partial treatment, often constrained by two competing narratives, optimistically focused on replacing biased recruiter decisions or pessimistically pointing to the automation of discrimination. Whether, and more importantly what types of, algorithmic hiring can be less biased and more beneficial to society than low-tech alternatives currently remains unanswered, to the detriment of trustworthiness. This multidisciplinary survey caters to practitioners and researchers with a balanced and integrated coverage of systems, biases, measures, mitigation strategies, datasets, and legal aspects of algorithmic hiring and fairness. Our work supports a contextualized understanding and governance of this technology by highlighting current opportunities and limitations, providing recommendations for future work to ensure shared benefits for all stakeholders.

Computers and Society,Artificial Intelligence
Fair and Unbiased Algorithmic Decision Making: Current State and Future Challenges

Songül Tolan

DOI: https://doi.org/10.48550/arXiv.1901.04730

2019-01-15

Abstract:Machine learning algorithms are now frequently used in sensitive contexts that substantially affect the course of human lives, such as credit lending or criminal justice. This is driven by the idea that `objective' machines base their decisions solely on facts and remain unaffected by human cognitive biases, discriminatory tendencies or emotions. Yet, there is overwhelming evidence showing that algorithms can inherit or even perpetuate human biases in their decision making when they are based on data that contains biased human decisions. This has led to a call for fairness-aware machine learning. However, fairness is a complex concept which is also reflected in the attempts to formalize fairness for algorithmic decision making. Statistical formalizations of fairness lead to a long list of criteria that are each flawed (or harmful even) in different contexts. Moreover, inherent tradeoffs in these criteria make it impossible to unify them in one general framework. Thus, fairness constraints in algorithms have to be specific to the domains to which the algorithms are applied. In the future, research in algorithmic decision making systems should be aware of data and developer biases and add a focus on transparency to facilitate regular fairness audits.

Machine Learning,Computers and Society
Bias, Fairness, and Accountability with AI and ML Algorithms

Nengfeng Zhou,Zach Zhang,Vijayan N. Nair,Harsh Singhal,Jie Chen,Agus Sudjianto

DOI: https://doi.org/10.48550/arXiv.2105.06558

2021-05-14

Abstract:The advent of AI and ML algorithms has led to opportunities as well as challenges. In this paper, we provide an overview of bias and fairness issues that arise with the use of ML algorithms. We describe the types and sources of data bias, and discuss the nature of algorithmic unfairness. This is followed by a review of fairness metrics in the literature, discussion of their limitations, and a description of de-biasing (or mitigation) techniques in the model life cycle.

Machine Learning
Mitigating Bias in Algorithmic Hiring: Evaluating Claims and Practices

Manish Raghavan,Solon Barocas,Jon Kleinberg,Karen Levy

DOI: https://doi.org/10.1145/3351095.3372828

2019-12-07

Abstract:There has been rapidly growing interest in the use of algorithms in hiring, especially as a means to address or mitigate bias. Yet, to date, little is known about how these methods are used in practice. How are algorithmic assessments built, validated, and examined for bias? In this work, we document and analyze the claims and practices of companies offering algorithms for employment assessment. In particular, we identify vendors of algorithmic pre-employment assessments (i.e., algorithms to screen candidates), document what they have disclosed about their development and validation procedures, and evaluate their practices, focusing particularly on efforts to detect and mitigate bias. Our analysis considers both technical and legal perspectives. Technically, we consider the various choices vendors make regarding data collection and prediction targets, and explore the risks and trade-offs that these choices pose. We also discuss how algorithmic de-biasing techniques interface with, and create challenges for, antidiscrimination law.

Computers and Society,Artificial Intelligence,Machine Learning
Designing value-sensitive AI: a critical review and recommendations for socio-technical design processes

Malak Sadek,Rafael A. Calvo,Céline Mougenot

DOI: https://doi.org/10.1007/s43681-023-00373-7

2023-11-21

AI and Ethics

Abstract:Abstract This paper presents a critical review of how different socio-technical design processes for AI-based systems, from scholarly works and industry, support the creation of value-sensitive AI (VSAI). The review contributes to the emerging field of human-centred AI, and the even more embryonic space of VSAI in four ways: (i) it introduces three criteria for the review of VSAI based on their contribution to design processes’ overall value-sensitivity, and as a response to criticisms that current interventions are lacking in these aspects: comprehensiveness, level of guidance offered, and methodological value-sensitivity, (ii) it provides a novel review of socio-technical design processes for AI-based systems, (iii) it assesses each process based on the mentioned criteria and synthesises the results into broader trends, and (iv) it offers a resulting set of recommendations for the design of VSAI. The objective of the paper is to help creators and followers of design processes—whether scholarly or industry-based—to understand the level of value-sensitivity offered by different socio-technical design processes and act accordingly based on their needs: to adopt or adapt existing processes or to create new ones.
Algorithmic bias: on the implicit biases of social technology

Gabbrielle M. Johnson

DOI: https://doi.org/10.1007/s11229-020-02696-y

IF: 1.595

2020-06-20

Synthese

Abstract:Often machine learning programs inherit social patterns reflected in their training data without any directed effort by programmers to include such biases. Computer scientists call this algorithmic bias. This paper explores the relationship between machine bias and human cognitive bias. In it, I argue similarities between algorithmic and cognitive biases indicate a disconcerting sense in which sources of bias emerge out of seemingly innocuous patterns of information processing. The emergent nature of this bias obscures the existence of the bias itself, making it difficult to identify, mitigate, or evaluate using standard resources in epistemology and ethics. I demonstrate these points in the case of mitigation techniques by presenting what I call 'the Proxy Problem'. One reason biases resist revision is that they rely on proxy attributes, seemingly innocuous attributes that correlate with socially-sensitive attributes, serving as proxies for the socially-sensitive attributes themselves. I argue that in both human and algorithmic domains, this problem presents a common dilemma for mitigation: attempts to discourage reliance on proxy attributes risk a tradeoff with judgement accuracy. This problem, I contend, admits of no purely algorithmic solution.

history & philosophy of science
Value Sensitive Design for autonomous weapon systems – a primer

Boshuijzen-van Burken, Christine

DOI: https://doi.org/10.1007/s10676-023-09687-w

IF: 3.633

2023-02-11

Ethics and Information Technology

Abstract:Value Sensitive Design (VSD) is a design methodology developed by Batya Friedman and Peter Kahn (2003) that brings in moral deliberations in an early stage of a design process. It assumes that neither technology itself is value neutral, nor shifts the value-ladennes to the sole usage of technology. This paper adds to emerging literature onVSD for autonomous weapons systems development and discusses extant literature on values in autonomous systems development in general and in autonomous weapons development in particular. I identify opportunities, such as public debates, and threats, such as the classified nature of the design process, for VSD in autonomous weapons development. This paper contributes to academic debates about the non-neutrality of technology by pointing out that values have been and can be explicitly designed into autonomous systems. It is informative for policy makers and designers who are tasked with developing actual autonomous weapons or policies around such systems, as they learn about an established design methodology that is sensitive to societal concerns and industry needs and that can be applied to autonomous weapons systems.

information science & library science,ethics
Designing AI for Explainability and Verifiability: A Value Sensitive Design Approach to Avoid Artificial Stupidity in Autonomous Vehicles

Steven Umbrello,Roman V. Yampolskiy

DOI: https://doi.org/10.1007/s12369-021-00790-w

IF: 3.802

2021-05-15

International Journal of Social Robotics

Abstract:Abstract One of the primary, if not most critical, difficulties in the design and implementation of autonomous systems is the black-boxed nature of the decision-making structures and logical pathways. How human values are embodied and actualised in situ may ultimately prove to be harmful if not outright recalcitrant. For this reason, the values of stakeholders become of particular significance given the risks posed by opaque structures of intelligent agents. This paper explores how decision matrix algorithms, via the belief-desire-intention model for autonomous vehicles, can be designed to minimize the risks of opaque architectures. Primarily through an explicit orientation towards designing for the values of explainability and verifiability. In doing so, this research adopts the Value Sensitive Design (VSD) approach as a principled framework for the incorporation of such values within design. VSD is recognized as a potential starting point that offers a systematic way for engineering teams to formally incorporate existing technical solutions within ethical design, while simultaneously remaining pliable to emerging issues and needs. It is concluded that the VSD methodology offers at least a strong enough foundation from which designers can begin to anticipate design needs and formulate salient design flows that can be adapted to the changing ethical landscapes required for utilisation in autonomous vehicles.

robotics
Keeping Designers in the Loop: Communicating Inherent Algorithmic Trade-offs Across Multiple Objectives

Bowen Yu,Ye Yuan,Loren Terveen,Zhiwei Steven Wu,Jodi Forlizzi,Haiyi Zhu

DOI: https://doi.org/10.48550/arXiv.1910.03061

2020-07-06

Abstract:Artificial intelligence algorithms have been used to enhance a wide variety of products and services, including assisting human decision making in high-stakes contexts. However, these algorithms are complex and have trade-offs, notably between prediction accuracy and fairness to population subgroups. This makes it hard for designers to understand algorithms and design products or services in a way that respects users' goals, values, and needs. We proposed a method to help designers and users explore algorithms, visualize their trade-offs, and select algorithms with trade-offs consistent with their goals and needs. We evaluated our method on the problem of predicting criminal defendants' likelihood to re-offend through (i) a large-scale Amazon Mechanical Turk experiment, and (ii) in-depth interviews with domain experts. Our evaluations show that our method can help designers and users of these systems better understand and navigate algorithmic trade-offs. This paper contributes a new way of providing designers the ability to understand and control the outcomes of algorithmic systems they are creating.

Human-Computer Interaction
Twenty years of value sensitive design: a review of methodological practices in VSD projects

Till Winkler,Sarah Spiekermann

DOI: https://doi.org/10.1007/s10676-018-9476-2

IF: 3.633

2018-08-21

Ethics and Information Technology

Abstract:Abstract This article reviews the academic literature (1996–2016) that emerged under value sensitive design (VSD). It investigates those VSD projects that employed the tripartite methodology, examining the use of VSD methodological elements, and illustrating common practices and identifying shortcomings. The article provides advice for VSD researchers on how to complete and enhance their methodological approach as the research community moves forward.

information science & library science,ethics
Algorithmic Fairness: A Tolerance Perspective

Renqiang Luo,Tao Tang,Feng Xia,Jiaying Liu,Chengpei Xu,Leo Yu Zhang,Wei Xiang,Chengqi Zhang

2024-04-26

Abstract:Recent advancements in machine learning and deep learning have brought algorithmic fairness into sharp focus, illuminating concerns over discriminatory decision making that negatively impacts certain individuals or groups. These concerns have manifested in legal, ethical, and societal challenges, including the erosion of trust in intelligent systems. In response, this survey delves into the existing literature on algorithmic fairness, specifically highlighting its multifaceted social consequences. We introduce a novel taxonomy based on 'tolerance', a term we define as the degree to which variations in fairness outcomes are acceptable, providing a structured approach to understanding the subtleties of fairness within algorithmic decisions. Our systematic review covers diverse industries, revealing critical insights into the balance between algorithmic decision making and social equity. By synthesizing these insights, we outline a series of emerging challenges and propose strategic directions for future research and policy making, with the goal of advancing the field towards more equitable algorithmic systems.

Computers and Society,Artificial Intelligence,Information Retrieval,Machine Learning
Algorithmic bias and racial inequality: a critical review

Maximilian Kasy

DOI: https://doi.org/10.1093/oxrep/grae031

2024-11-27

Oxford Review of Economic Policy

Abstract:Most definitions of algorithmic bias and fairness encode decision-maker interests, such as profits, rather than the interests of disadvantaged groups (e.g. racial minorities): bias is defined as a deviation from profit maximization. Future research should instead focus on the causal effect of automated decisions on the distribution of welfare, both across and within groups. The literature emphasizes some apparent contradictions between different notions of fairness, and between fairness and profits. These contradictions vanish, however, when profits are maximized. Existing work involves conceptual slippages between statistical notions of bias and misclassification errors, economic notions of profit, and normative notions of bias and fairness. Notions of bias nonetheless carry some interest within the welfare paradigm that I advocate for, if we understand bias and discrimination as mechanisms and potential points of intervention.

economics
Algorithmic discrimination: examining its types and regulatory measures with emphasis on US legal practices

Xukang Wang,Ying Cheng Wu,Xueliang Ji,Hongpeng Fu

DOI: https://doi.org/10.3389/frai.2024.1320277

2024-05-21

Abstract:Introduction: Algorithmic decision-making systems are widely used in various sectors, including criminal justice, employment, and education. While these systems are celebrated for their potential to enhance efficiency and objectivity, they also pose risks of perpetuating and amplifying societal biases and discrimination. This paper aims to provide an indepth analysis of the types of algorithmic discrimination, exploring both the challenges and potential solutions. Methods: The methodology includes a systematic literature review, analysis of legal documents, and comparative case studies across different geographic regions and sectors. This multifaceted approach allows for a thorough exploration of the complexity of algorithmic bias and its regulation. Results: We identify five primary types of algorithmic bias: bias by algorithmic agents, discrimination based on feature selection, proxy discrimination, disparate impact, and targeted advertising. The analysis of the U.S. legal and regulatory framework reveals a landscape of principled regulations, preventive controls, consequential liability, self-regulation, and heteronomy regulation. A comparative perspective is also provided by examining the status of algorithmic fairness in the EU, Canada, Australia, and Asia. Conclusion: Real-world impacts are demonstrated through case studies focusing on criminal risk assessments and hiring algorithms, illustrating the tangible effects of algorithmic discrimination. The paper concludes with recommendations for interdisciplinary research, proactive policy development, public awareness, and ongoing monitoring to promote fairness and accountability in algorithmic decision-making. As the use of AI and automated systems expands globally, this work highlights the importance of developing comprehensive, adaptive approaches to combat algorithmic discrimination and ensure the socially responsible deployment of these powerful technologies.

Algorithmic bias and the Value Sensitive Design approach

Algorithmic bias: review, synthesis, and future research directions

Situated algorithms: a sociotechnical systemic approach to bias

Mapping value sensitive design onto AI for social good principles

Guidelines for Integrating Value Sensitive Design in Responsible AI Toolkits

LUCID: Exposing Algorithmic Bias through Inverse Design

Algorithmic bias: Looking beyond data bias to ensure algorithmic accountability and equity

Bias, Fairness and Accountability with Artificial Intelligence and Machine Learning Algorithms

Fairness and Bias in Algorithmic Hiring: a Multidisciplinary Survey

Fair and Unbiased Algorithmic Decision Making: Current State and Future Challenges

Bias, Fairness, and Accountability with AI and ML Algorithms

Mitigating Bias in Algorithmic Hiring: Evaluating Claims and Practices

Designing value-sensitive AI: a critical review and recommendations for socio-technical design processes

Algorithmic bias: on the implicit biases of social technology

Value Sensitive Design for autonomous weapon systems – a primer

Designing AI for Explainability and Verifiability: A Value Sensitive Design Approach to Avoid Artificial Stupidity in Autonomous Vehicles

Keeping Designers in the Loop: Communicating Inherent Algorithmic Trade-offs Across Multiple Objectives

Twenty years of value sensitive design: a review of methodological practices in VSD projects

Algorithmic Fairness: A Tolerance Perspective

Algorithmic bias and racial inequality: a critical review

Algorithmic discrimination: examining its types and regulatory measures with emphasis on US legal practices