Determining Novice and Expert Status in Human–Automation Interaction Through Hidden Markov Models

Anne French Mary L. Cummings Haibei Zhu Miroslav Pajic a Department of Electrical and Computer Engineering,Edmund T. Pratt Jr. School of Engineering,Duke University,Durham,North Carolina,USAb College of Engineering and Computing,George Mason University,Fairfax,Virginia,USA
DOI: https://doi.org/10.1080/08839514.2024.2402174
IF: 2.777
2024-09-11
Applied Artificial Intelligence
Abstract:Detecting when operators achieve expert proficiency is critical for organizations that employ human–automation interaction (HAI) in operations, particularly in safety-critical settings. Training operators for complex systems demand substantial time and resources, necessitated by safety considerations and the expansive scale of these systems. Recognizing operator expertise becomes instrumental in resource optimization and training efficiency. This study explores a modeling framework for real-time analysis of HAI operator behavior and strategies. Proposing a departure from traditional assessments, the research advocates for leveraging hidden Markov models (HMMs) to provide a comprehensive portrayal of operator performance, facilitating a nuanced comparison of expert and novice strategies. Using data from a real-time strategy game, the paper details the development of HMMs and elucidates training and interface design implications. Results affirm the hypothesis that experts formulate more efficient strategies, reflected in HMMs with fewer hidden states compared to those describing novice behavior. This aligns with prior research emphasizing the organized nature of experts' strategies. In-depth analysis delves into specific states, frequencies, and predominant strategies, revealing distinctions between experts' offensive focus and novices' emphasis on initial setup aspects of gameplay.
computer science, artificial intelligence,engineering, electrical & electronic
What problem does this paper attempt to address?