The Problem with the Trolley Problem and the Need for Systems Thinking
Marc Steen
DOI: https://doi.org/10.1145/3635302
IF: 22.7
2024-05-23
Communications of the ACM
Abstract:In 1967, English philosopher Philippa Foot wanted to discuss the doctrine of double effect 2 : The difficulty of evaluating acting from good intentions whilst also bringing about harm as a side effect. She invited readers to imagine being "the driver of a runaway tram which he can only steer from one narrow track onto another; five men are working on one track and one man on the other; anyone on the track he enters is bound to be killed." To explore moral reasoning, people have created variations of what is now known as the Trolley Problem. For instance, a situation in which you have a lever that you can use to divert the trolley, so that one person is killed instead of five, or a situation in which you stand on a bridge above the track, with a person whom you can throw in front of the trolley to stop it. (Fewer people will choose to throw this person because it feels like killing that person, whereas pulling the lever feels like having the trolley do the killing.) Meanwhile, the Trolley Problem has become a trope, appearing in a comedy series, a and in many conversations about technology and ethics. The Problem The Trolley Problem has inspired scores of psychology experiments, including MIT's Moral Machine, 1 an online survey where people had to decide what a self-driving car should do in case of an impending accident. Participants were given a series of pairs of scenarios, presented as map-like diagrams, with various numbers and types of pedestrians and passengers. For each pair of scenarios, they had to choose between options such as driving ahead and killing pedestrians, or veering into an obstacle and killing passengers. Based on 40 million responses from more than 200 countries, they found general preferences, such as sparing humans over animals. They also found differences between cultures. People from countries with collectivistic cultures prefer sparing lives of older people instead of the lives of younger people. And people from poorer countries with weaker institutions tend to spare pedestrians who cross illegally. Clearly, Foot did not intend to present a practical problem that we can solve through computer science or software engineering. However, maybe such experiments can provide some clues for programming self-driving cars? Maybe indeed. Maybe not. There are at least two reasons why a study like the Moral Machine is inadequate in guiding the development of self-driving cars. First, it probed people's preferences . Scottish philosopher David Hume famously reminded us to not confuse is and ought 3 : You cannot derive moral statements from empirical findings. If people do X or say that they would do X, it does not automatically follow that X is morally acceptable or recommendable. This is called the naturalistic fallacy : the incorrect belief that what is the case also ought to be . Furthermore, the survey used scenarios that depict different types of pedestrians, rather stereotypically: homeless people, with a baggy coat; business people, with a suitcase; medical personnel, with a first-aid kit; and criminal people, with a mask and loot. Participants probably took these differences into account in the survey. In practice, however, people do not carry such identifiers, and it would be undesirable or unrealistic to require them to do that, so self-driving cars cannot take into account their identities. The problem with invoking the Trolley Problem in discussions of ethics and technology is that people might believe that surveying people's preferences can "solve" ethical questions and that computer scientists and software engineers can "solve" ethics through calculation and optimization. (An interesting view on the Trolley Problem, from a virtue ethics perspective, is from Liezl van Zyl. 10 ) In this Communications Opinion column, I present an alternative approach, more aligned with the original function of the Trolley Problem as a thought experiment: To promote ethical reflection and deliberation regarding the development and deployment of technologies 7 , 8 and ultimately result in more ethical design decisions. Context First, we can put the trolley, the workers, and the moral agent in a larger context to better understand the problem and to envision multiple approaches to enhancing the safety of emerging technologies. The original description is parsimonious on purpose: a trolley with a broken brake, two tracks with people, and you with a lever. Real life, however, is more complex. We can turn to systems thinking 4 to understand and appreciate how seemingly separate elements relate within a larger sociotechnical system aroun -Abstract Truncated-
computer science, theory & methods, software engineering, hardware & architecture