Predicting a Malicious Stop Sign: Knowledge, Exposure, Trust in AI
Scott Mishler,Katherine Garcia,Erin Fuller-Jakaitis,Cong Wang,Bin Hu,Jeremiah Still,Jing Chen
DOI: https://doi.org/10.1177/1071181321651239
2021-01-01
Proceedings of the Human Factors and Ergonomics Society Annual Meeting
Abstract:The detection of road signs and other objects in the environment are critical for automated vehicles (AVs). Artificial intelligence (AI) is responsible for ensuring that the sensors capture accurate data and process that information into proper decisions. No systems are 100% perfect, and any errors could compromise safety. Currently, the human driver is still required for AVs in case of potential system failures (SAE, 2016). However, we still lack understanding of users' perception and trust while partnering with AVs especially when the AV is experiencing cyberattacks. This study measured users’ AI knowledge, trust, and examined their understanding of the AI’s recognition of maliciously manipulated stop signs. For this study, we ran 205 participants using an online survey through Qualtrics containing 62 questions. The questions were focused on general AI understanding, exposure to AI, general understanding of AI in the context of AVs, applications of AI in AVs, future AI use for AVs, and trust and opinions on AI. Additionally, we tested their understanding of images of a normal stop sign and a maliciously manipulated stop sign. Participants were asked to rate how much they thought the each of the images was of a stop sign and whether the current AI technology would classify them as a stop sign. The testing of these stop signs used a factorial within-subjects design of agent type (user vs. AI) and image type (original vs. malicious). For the user agent, the statements asked if they believed that the image showed a stop sign or not. For the AI agent, the statements asked if they believed that the AI technology would think that the image showed a stop sign. For the original image, the image was a standard stop sign. For the malicious image, the image was the same stop sign image, but it had been tampered with. The malicious stop image used in the study was generated from the algorithms proposed in (Eykholt et al., 2017), which focused on how intentional perturbations on the images can lead to misclassification by an AI. The malicious stop sign was an intentionally perturbed image meant to prevent AI from the proper classification of the sign. The knowledge ratings showed participants were quite knowledgeable about AI and had a high amount of exposure to AI. These two ratings were correlated, likely because the more time spent with a system leads to a greater understanding of what processes are involved in the system. We also found that participants had an accurate perception of stop signs for both the standard and malicious conditions. Both images were recognizable as a stop sign even though the malicious sign was altered. Participants thought that they were better at judging stop signs than the AI, giving lower ratings for AI than their own ability. This result is similar to prior research that shows people are often likely to rely more on their own abilities than that of an AV or have higher selfconfidence than confidence in automation (Hoff & Bashir, 2015; Kyriakidis et al., 2015; Tenhundfeld et al., 2020). Participants rated AI’s ability to detect the malicious image as a stop sign lower than the original stop sign, which was expected. The perturbation noise added onto the sign was designed to be highly effective for confusing the AI. Participants showed high confidence in the AI (with an average rating of 4.15 out of 5). This result demonstrates a potential mismatch between an AI's capabilities and the beliefs that a potential user might have about that system. Participants accurately rated the AI’s ability to detect the original image significantly higher than the malicious image; however, most participants were fairly certain that the malicious image was identifiable by AI. This result could be a sign of overtrust in the AI (Parasuraman & Riley, 1997) because they are unaware of how the AI system works or its true capabilities.