Trust Is Not Enough: Accuracy, Error, Randomness, and Accountability in an Algorithmic Society

Emanuel Moss

DOI: https://doi.org/10.1145/3565479

IF: 22.7

2023-05-24

Communications of the ACM

Abstract:Searching for improved algorithmic accountability.

computer science, theory & methods, software engineering, hardware & architecture

What problem does this paper attempt to address?

Accountability in an Algorithmic Society: Relationality, Responsibility, and Robustness in Machine Learning

A. Feder Cooper,Emanuel Moss,Benjamin Laufer,Helen Nissenbaum

DOI: https://doi.org/10.1145/3531146.3533150

2022-05-14

Abstract:In 1996, Accountability in a Computerized Society [95] issued a clarion call concerning the erosion of accountability in society due to the ubiquitous delegation of consequential functions to computerized systems. Nissenbaum [95] described four barriers to accountability that computerization presented, which we revisit in relation to the ascendance of data-driven algorithmic systems--i.e., machine learning or artificial intelligence--to uncover new challenges for accountability that these systems present. Nissenbaum's original paper grounded discussion of the barriers in moral philosophy; we bring this analysis together with recent scholarship on relational accountability frameworks and discuss how the barriers present difficulties for instantiating a unified moral, relational framework in practice for data-driven algorithmic systems. We conclude by discussing ways of weakening the barriers in order to do so.

Computers and Society,Artificial Intelligence,Machine Learning
A Model for Evaluating Algorithmic Systems Accountability

Yiannis Kanellopoulos

DOI: https://doi.org/10.48550/arXiv.1807.06083

2018-07-13

Abstract:Algorithmic systems make decisions that have a great impact in our lives. As our dependency on them is growing so does the need for transparency and holding them accountable. This paper presents a model for evaluating how transparent these systems are by focusing on their algorithmic part as well as the maturity of the organizations that utilize them. We applied this model on a classification algorithm created and utilized by a large financial institution. The results of our analysis indicated that the organization was only partially in control of their algorithm and they lacked the necessary benchmark to interpret the deducted results and assess the validity of its inferencing.

Computers and Society,Machine Learning
Algorithmic policing accountability: eight sociotechnical challenges

Martijn Wessels

DOI: https://doi.org/10.1080/10439463.2023.2241965

2023-08-02

Abstract:Law enforcement agencies increasingly adopt algorithms for their policing practices, leading to new accountability challenges. However, academic literature tends to regard algorithmic applications as a single entity, resulting in broad and general statements on these risks. By introducing a sociotechnical model of algorithmic policing, this paper identifies eight accountability challenges that originate from technical, organisational and human components. It is illuminated how algorithmic policing accountability is a process that consists of multiple steps that should be considered separately to scrutinise the accountability challenges. The present study showcases how the sociotechnical context determines whether and how accountability issues might arise in algorithmic policing practices. The accountability challenges are presented as propositions that can direct future research in assessing the accountability risks of algorithmic policing applications.

criminology & penology
Towards algorithms and models that we can trust: A theoretical perspective

Luca Oneto,Sandro Ridella,Davide Anguita

DOI: https://doi.org/10.1016/j.neucom.2024.127798

IF: 6

2024-05-09

Neurocomputing

Abstract:In the last decade it became increasingly apparent the inability of technical metrics such as accuracy, sustainability, and non-regressiveness to well characterize the behavior of intelligent systems. In fact, they are nowadays requested to meet also ethical requirements such as explainability, fairness, robustness, and privacy increasing our trust in their use in the wild. Of course often technical and ethical metrics are in tension between each other but the final goal is to be able to develop a new generation of more responsible and trustworthy machine learning. In this paper, we focus our attention on machine learning algorithms and associated predictive models, questioning for the first time, from a theoretical perspective, if it is possible to simultaneously guarantee their performance in terms of both technical and ethical metrics towards machine learning algorithms that we can trust. In particular, we will investigate for the first time both theory and practice of deterministic and randomized algorithms and associated predictive models showing the advantages and disadvantages of the different approaches. For this purpose we will leverage the most recent advances coming from the statistical learning theory: Complexity-Based Methods, Distribution Stability, PAC-Bayes, and Differential Privacy. Results will show that it is possible to develop consistent algorithms which generate predictive models with guarantees on multiple trustworthiness metrics.

computer science, artificial intelligence
Research on the Role of Algorithm Transparency in Algorithm Accountability

Weimin Ouyang

DOI: https://doi.org/10.2991/assehr.k.191221.055

2019-01-01

Abstract:Algorithm transparency as an algorithm accountability method has been widely praised by the community. Algorithm transparency is considered to be an effective way to crack the algorithm black box. The algorithm transparency refers to the disclosure of the source code of the algorithm system and the data used in the algorithm system. This paper first...
Automated news recommendation in front of adversarial examples and the technical limits of transparency in algorithmic accountability

Antonin Descampe,Clément Massart,Simon Poelman,François-Xavier Standaert,Olivier Standaert

DOI: https://doi.org/10.1007/s00146-021-01159-3

2021-03-13

Abstract:Algorithmic decision making is used in an increasing number of fields. Letting automated processes take decisions raises the question of their accountability. In the field of computational journalism, the algorithmic accountability framework proposed by Diakopoulos formalizes this challenge by considering algorithms as objects of human creation, with the goal of revealing the intent embedded into their implementation. A consequence of this definition is that ensuring accountability essentially boils down to a transparency question: given the appropriate reverse-engineering tools, it should be feasible to extract design criteria and to identify intentional biases. General limitations of this transparency ideal have been discussed by Ananny and Crawford (New Media Soc 20(3):973–989, 2018). We further focus on its technical limitations. We show that even if reverse-engineering concludes that the criteria embedded into an algorithm correspond to its publicized intent, it may be that adversarial behaviors make the algorithm deviate from its expected operation. We illustrate this issue with an automated news recommendation system, and show how the classification algorithms used in such systems can be fooled with hard-to-notice modifications of the articles to classify. We therefore suggest that robustness against adversarial behaviors should be taken into account in the definition of algorithmic accountability, to better capture the risks inherent to algorithmic decision making. We finally discuss the various challenges that this new technical limitation raises for journalism practice.
A relationship and not a thing: A relational approach to algorithmic accountability and assessment documentation

Jacob Metcalf,Emanuel Moss,Ranjit Singh,Emnet Tafese,Elizabeth Anne Watkins

DOI: https://doi.org/10.48550/arXiv.2203.01455

2022-03-03

Abstract:Central to a number of scholarly, regulatory, and public conversations about algorithmic accountability is the question of who should have access to documentation that reveals the inner workings, intended function, and anticipated consequences of algorithmic systems, potentially establishing new routes for impacted publics to contest the operations of these systems. Currently, developers largely have a monopoly on information about how their systems actually work and are incentivized to maintain their own ignorance about aspects of how their systems affect the world. Increasingly, legislators, regulators and advocates have turned to assessment documentation in order to address the gap between the public's experience of algorithmic harms and the obligations of developers to document and justify their design decisions. However, issues of standing and expertise currently prevent publics from cohering around shared interests in preventing and redressing algorithmic harms; as we demonstrate with multiple cases, courts often find computational harms non-cognizable and rarely require developers to address material claims of harm. Constructed with a triadic accountability relationship, algorithmic impact assessment regimes could alter this situation by establishing procedural rights around public access to reporting and documentation. Developing a relational approach to accountability, we argue that robust accountability regimes must establish opportunities for publics to cohere around shared experiences and interests, and to contest the outcomes of algorithmic systems that affect their lives. Furthermore, algorithmic accountability policies currently under consideration in many jurisdictions must provide the public with adequate standing and opportunities to access and contest the documentation provided by the actors and the judgments passed by the forum.

Computers and Society
Designing Accountable Systems

Severin Kacianka,Alexander Pretschner

DOI: https://doi.org/10.1145/3442188.3445905

2021-01-20

Abstract:Accountability is an often called for property of technical systems. It is a requirement for algorithmic decision systems, autonomous cyber-physical systems, and for software systems in general. As a concept, accountability goes back to the early history of Liberalism and is suggested as a tool to limit the use of power. This long history has also given us many, often slightly differing, definitions of accountability. The problem that software developers now face is to understand what accountability means for their systems and how to reflect it in a system's design. To enable the rigorous study of accountability in a system, we need models that are suitable for capturing such a varied concept. In this paper, we present a method to express and compare different definitions of accountability using Structural Causal Models. We show how these models can be used to evaluate a system's design and present a small use case based on an autonomous car.

Software Engineering
Accountable Artificial Intelligence: Holding Algorithms to Account

Madalina Busuioc

DOI: https://doi.org/10.1111/puar.13293

IF: 8.144

2020-11-11

Public Administration Review

Abstract:<p>Artificial intelligence algorithms govern in subtle, yet fundamental ways, the way we live and are transforming our societies. The promise of efficient, low‐cost or 'neutral' solutions harnessing the potential of big data has led public bodies to adopt algorithmic systems in the provision of public services. As AI algorithms have permeated high‐stakes aspects of our public existence – from hiring and education decisions, to the governmental use of enforcement powers (policing) or liberty‐restricting decisions (bail and sentencing), this necessarily raises important accountability questions: What accountability challenges do AI algorithmic systems bring with them, and how can we safeguard accountability in algorithmic decision‐making? Drawing on a decidedly public administration perspective, and given the current challenges that have thus far become manifest in the field, we critically reflect on and map out in a conceptually‐guided manner, the implications of these systems, and the limitations they pose, for public accountability.</p><p>This article is protected by copyright. All rights reserved.</p>

public administration
Fairness, Accuracy, and Unreliable Data

Kevin Stangl

2024-08-29

Abstract:This thesis investigates three areas targeted at improving the reliability of machine learning; fairness in machine learning, strategic classification, and algorithmic robustness. Each of these domains has special properties or structure that can complicate learning. A theme throughout this thesis is thinking about ways in which a `plain' empirical risk minimization algorithm will be misleading or ineffective because of a mis-match between classical learning theory assumptions and specific properties of some data distribution in the wild. Theoretical understanding in eachof these domains can help guide best practices and allow for the design of effective, reliable, and robust systems.

Machine Learning
The Algorithmic Leviathan: Arbitrariness, Fairness, and Opportunity in Algorithmic Decision Making Systems

Kathleen A. Creel,Deborah Hellman

DOI: https://doi.org/10.1017/can.2022.3

2021-03-03

Canadian Journal of Philosophy

Abstract:Automated decision-making systems implemented in public life are typically standardized. One algorithmic decision-making system can replace thousands of human deciders. Each of the humans so replaced had her own decision-making criteria: some good, some bad, and some arbitrary. Is such arbitrariness of moral concern? We argue that an isolated arbitrary decision need not morally wrong the individual whom it misclassifies. However, if the same algorithms are applied across a public sphere, such as hiring or lending, a person could be excluded from a large number of opportunities. This harm persists even when the automated decision-making systems are "fair" on standard metrics of fairness. We argue that such arbitrariness at scale is morally problematic and propose technically informed solutions that can lessen the impact of algorithms at scale and so mitigate or avoid the moral harms we identify.

Philosophy,Computer Science,Law
Mathematical Algorithm Design for Deep Learning under Societal and Judicial Constraints: The Algorithmic Transparency Requirement

Holger Boche,Adalbert Fono,Gitta Kutyniok

2024-01-18

Abstract:Deep learning still has drawbacks in terms of trustworthiness, which describes a comprehensible, fair, safe, and reliable method. To mitigate the potential risk of AI, clear obligations associated to trustworthiness have been proposed via regulatory guidelines, e.g., in the European AI Act. Therefore, a central question is to what extent trustworthy deep learning can be realized. Establishing the described properties constituting trustworthiness requires that the factors influencing an algorithmic computation can be retraced, i.e., the algorithmic implementation is transparent. Motivated by the observation that the current evolution of deep learning models necessitates a change in computing technology, we derive a mathematical framework which enables us to analyze whether a transparent implementation in a computing model is feasible. We exemplarily apply our trustworthiness framework to analyze deep learning approaches for inverse problems in digital and analog computing models represented by Turing and Blum-Shub-Smale Machines, respectively. Based on previous results, we find that Blum-Shub-Smale Machines have the potential to establish trustworthy solvers for inverse problems under fairly general conditions, whereas Turing machines cannot guarantee trustworthiness to the same degree.

Machine Learning,Artificial Intelligence,Computational Complexity
Algorithmic Accountability in Context. Socio-Technical Perspectives on Structural Causal Models

Nikolaus Poechhacker,Severin Kacianka

DOI: https://doi.org/10.3389/fdata.2020.519957

2021-01-29

Frontiers in Big Data

Abstract:The increasing use of automated decision making (ADM) and machine learning sparked an ongoing discussion about algorithmic accountability. Within computer science, a new form of producing accountability has been discussed recently: causality as an expression of algorithmic accountability, formalized using structural causal models (SCMs). However, causality itself is a concept that needs further exploration. Therefore, in this contribution we confront ideas of SCMs with insights from social theory, more explicitly pragmatism, and argue that formal expressions of causality must always be seen in the context of the social system in which they are applied. This results in the formulation of further research questions and directions.
How Much Information?: Effects of Transparency on Trust in an Algorithmic Interface

René F. Kizilcec

DOI: https://doi.org/10.1145/2858036.2858402

2016-05-07

Abstract:The rising prevalence of algorithmic interfaces, such as curated feeds in online news, raises new questions for designers, scholars, and critics of media. This work focuses on how transparent design of algorithmic interfaces can promote awareness and foster trust. A two-stage process of how transparency affects trust was hypothesized drawing on theories of information processing and procedural justice. In an online field experiment, three levels of system transparency were tested in the high-stakes context of peer assessment. Individuals whose expectations were violated (by receiving a lower grade than expected) trusted the system less, unless the grading algorithm was made more transparent through explanation. However, providing too much information eroded this trust. Attitudes of individuals whose expectations were met did not vary with transparency. Results are discussed in terms of a dual process model of attitude change and the depth of justification of perceived inconsistency. Designing for trust requires balanced interface transparency - not too little and not too much.

Computer Science,Psychology
International Human Rights Law As A Framework For Algorithmic Accountability

Lorna McGregor,Daragh Murray,Vivian Ng

DOI: https://doi.org/10.1017/S0020589319000046

2019-10-07

International and Comparative Law Quarterly

Abstract:Existing approaches to 'algorithmic accountability', such as transparency, provide an important baseline, but are insufficient to address the (potential) harm to human rights caused by the use of algorithms in decision-making. In order to effectively address the impact on human rights, we argue that a framework that sets out a shared understanding and means of assessing harm; is capable of dealing with multiple actors and different forms of responsibility; and applies across the full algorithmic life cycle, from conception to deployment, is needed. While generally overlooked in debates on algorithmic accountability, in this article, we suggest that international human rights law already provides this framework. We apply this framework to illustrate the effect it has on the choices to employ algorithms in decision-making in the first place and the safeguards required. While our analysis indicates that in some circumstances, the use of algorithms may be restricted, we argue that these findings are not 'anti-innovation' but rather appropriate checks and balances to ensure that algorithms contribute to society, while safeguarding against risks.

law
Towards algorithm auditing: managing legal, ethical and technological risks of AI, ML and associated algorithms

Adriano Koshiyama,Emre Kazim,Philip Treleaven,Pete Rai,Lukasz Szpruch,Giles Pavey,Ghazi Ahamat,Franziska Leutner,Randy Goebel,Andrew Knight,Janet Adams,Christina Hitrova,Jeremy Barnett,Parashkev Nachev,David Barber,Tomas Chamorro-Premuzic,Konstantin Klemmer,Miro Gregorovic,Shakeel Khan,Elizabeth Lomas,Airlie Hilliard,Siddhant Chatterjee

DOI: https://doi.org/10.1098/rsos.230859

2024-05-15

Abstract:-Business reliance on algorithms is becoming ubiquitous, and companies are increasingly concerned about their algorithms causing major financial or reputational damage. High-profile cases include Google's AI algorithm for photo classification mistakenly labelling a black couple as gorillas in 2015 (Gebru 2020 In The Oxford handbook of ethics of AI, pp. 251-269), Microsoft's AI chatbot Tay that spread racist, sexist and antisemitic speech on Twitter (now X) (Wolf et al. 2017 ACM Sigcas Comput. Soc. 47, 54-64 (doi:10.1145/3144592.3144598)), and Amazon's AI recruiting tool being scrapped after showing bias against women. In response, governments are legislating and imposing bans, regulators fining companies and the judiciary discussing potentially making algorithms artificial 'persons' in law. As with financial audits, governments, business and society will require algorithm audits; formal assurance that algorithms are legal, ethical and safe. A new industry is envisaged: Auditing and Assurance of Algorithms (cf. data privacy), with the remit to professionalize and industrialize AI, ML and associated algorithms. The stakeholders range from those working on policy/regulation to industry practitioners and developers. We also anticipate the nature and scope of the auditing levels and framework presented will inform those interested in systems of governance and compliance with regulation/standards. Our goal in this article is to survey the key areas necessary to perform auditing and assurance and instigate the debate in this novel area of research and practice.
"It would be pretty immoral to choose a random algorithm"

Helena Webb,Menisha Patel,Michael Rovatsos,Alan Davoust,Sofia Ceppi,Ansgar Koene,Liz Dowthwaite,Virginia Portillo,Marina Jirotka,Monica Cano

DOI: https://doi.org/10.1108/JICES-11-2018-0092

2019-10-07

Journal of Information, Communication and Ethics in Society

Abstract:The purpose of this paper is to report on empirical work conducted to open up algorithmic interpretability and transparency. In recent years, significant concerns have arisen regarding the increasing pervasiveness of algorithms and the impact of automated decision-making in our lives. Particularly problematic is the lack of transparency surrounding the development of these algorithmic systems and their use. It is often suggested that to make algorithms more fair, they should be made more transparent, but exactly how this can be achieved remains unclear.
Algorithm aversion? On the influence of advice accuracy on trust in algorithmic advice

Stefan Daschner,Robert Obermaier

DOI: https://doi.org/10.1080/12460125.2022.2070951

2022-05-08

Journal of Decision Systems

Abstract:There is empirical evidence that decision makers show negative behaviours towards algorithmic advice compared to human advice, termed as algorithm aversion. Taking a trust theoretical perspective, this study broadens the quite monolithic view on behaviour to its cognitive antecedent: cognitive trust, i.e. trusting beliefs and trusting intentions. We examine initial trust (cognitive trust and behaviour) as well as its development after performance feedback by conducting an online experiment that asked participants to forecast the expected demand for a product. Advice accuracy was manipulated by ± 5 % relative to the participant's initial forecasting accuracy determined in a pre-test. Results show that initial behaviour towards algorithmic advice is not influenced by cognitive trust. Furthermore, the decision maker's initial forecasting accuracy indicates a threshold between near-perfect and bad advice. When advice accuracy is at this threshold, we observe behavioural algorithm appreciation, particularly due to higher trusting integrity beliefs in algorithmic advice.
Trustless Audits without Revealing Data or Models

Suppakit Waiwitlikhit,Ion Stoica,Yi Sun,Tatsunori Hashimoto,Daniel Kang

2024-04-06

Abstract:There is an increasing conflict between business incentives to hide models and data as trade secrets, and the societal need for algorithmic transparency. For example, a rightsholder wishing to know whether their copyrighted works have been used during training must convince the model provider to allow a third party to audit the model and data. Finding a mutually agreeable third party is difficult, and the associated costs often make this approach impractical.

Cryptography and Security,Artificial Intelligence,Computers and Society,Machine Learning
Discrimination in the Age of Algorithms

Jon Kleinberg,Jens Ludwig,Sendhil Mullainathan,Cass R. Sunstein

DOI: https://doi.org/10.48550/arXiv.1902.03731

2019-02-11

Abstract:The law forbids discrimination. But the ambiguity of human decision-making often makes it extraordinarily hard for the legal system to know whether anyone has actually discriminated. To understand how algorithms affect discrimination, we must therefore also understand how they affect the problem of detecting discrimination. By one measure, algorithms are fundamentally opaque, not just cognitively but even mathematically. Yet for the task of proving discrimination, processes involving algorithms can provide crucial forms of transparency that are otherwise unavailable. These benefits do not happen automatically. But with appropriate requirements in place, the use of algorithms will make it possible to more easily examine and interrogate the entire decision process, thereby making it far easier to know whether discrimination has occurred. By forcing a new level of specificity, the use of algorithms also highlights, and makes transparent, central tradeoffs among competing values. Algorithms are not only a threat to be regulated; with the right safeguards in place, they have the potential to be a positive force for equity.

Computers and Society,Artificial Intelligence,Machine Learning

Trust Is Not Enough: Accuracy, Error, Randomness, and Accountability in an Algorithmic Society

Accountability in an Algorithmic Society: Relationality, Responsibility, and Robustness in Machine Learning

A Model for Evaluating Algorithmic Systems Accountability

Algorithmic policing accountability: eight sociotechnical challenges

Towards algorithms and models that we can trust: A theoretical perspective

Research on the Role of Algorithm Transparency in Algorithm Accountability

Automated news recommendation in front of adversarial examples and the technical limits of transparency in algorithmic accountability

A relationship and not a thing: A relational approach to algorithmic accountability and assessment documentation

Designing Accountable Systems

Accountable Artificial Intelligence: Holding Algorithms to Account

Fairness, Accuracy, and Unreliable Data

The Algorithmic Leviathan: Arbitrariness, Fairness, and Opportunity in Algorithmic Decision Making Systems

Mathematical Algorithm Design for Deep Learning under Societal and Judicial Constraints: The Algorithmic Transparency Requirement

Algorithmic Accountability in Context. Socio-Technical Perspectives on Structural Causal Models

How Much Information?: Effects of Transparency on Trust in an Algorithmic Interface

International Human Rights Law As A Framework For Algorithmic Accountability

Towards algorithm auditing: managing legal, ethical and technological risks of AI, ML and associated algorithms

"It would be pretty immoral to choose a random algorithm"

Algorithm aversion? On the influence of advice accuracy on trust in algorithmic advice

Trustless Audits without Revealing Data or Models

Discrimination in the Age of Algorithms