Using Multimodal Methods and Machine Learning to Recognize Mental Workload: Distinguishing Between Underload, Moderate Load, and Overload
Zebin JiangXinyan LiLiezhong GeJie XuYandi LuYijing ZhangMing Maoa Center for Psychological Sciences,Zhejiang University,Hangzhou,Chinab China North Vehicle Research Institute,Beijing,ChinaZebin Jiang is a PhD candidate at the Center for Psychological Sciences at Zhejiang University. He obtained his bachelor's degree in Computer Science and Technology from Zhengzhou University in 2021. His research interests include status recognition and human-machine interaction.Xinyan Li is a researcher fellow at the China North Vehicle Research Institute. She has long been engaged in the research of intelligent human-machine interaction-related technologies for special vehicles. She is committed to exploring methods of human-machine interaction and application modes suitable for special vehicles.Liezhong Ge is a professor at the Center for Psychological Sciences at Zhejiang University. He has been engaged in teaching and research in engineering psychology and cognitive psychology for more than three decades. His research interests include human-machine interaction,user experience and product usability,and facial recognition research.Jie Xu is an Assistant Professor at the Center for Psychological Sciences at Zhejiang University. He obtained his PhD in industrial and systems engineering from the University of Wisconsin–Madison in 2016. His research interests include human-agent interaction,physiological computing,and system safety.Yandi Lu is a PhD candidate at the Center for Psychological Sciences at Zhejiang University. He has an interdisciplinary background in design and psychology. His research interests include human–computer interaction,user experience,and human-centered AI.Yijing Zhang is currently a postdoctoral fellow at the Center for Psychological Sciences at Zhejiang University. He employs cognitive neuroscience approaches to delve into various aspects of research. His areas of focus include the investigation and regulation of mental workload,human-machine interaction,and special vehicle driver state recognition.Ming Mao is an academician of the Chinese Academy of Sciences. He has long been engaged in theoretical research and engineering practice on the overall design of armored weapons,drive systems,action systems,and other technologies.
DOI: https://doi.org/10.1080/10447318.2024.2352936
IF: 4.92
2024-05-22
International Journal of Human-Computer Interaction
Abstract:Mental workload recognition is of great significance in preventing human errors and accidents. This study constructed a multimodal recognition scheme to recognize three mental workload states: underload, moderate load, and overload. Based on driving scenarios, these three states were induced in this study by changing the driving modes and situations. Multimodal recognition of underload, moderate load, and overload was performed using electroencephalography (EEG), electrocardiography (ECG), and pupillometry. In addition, various machine learning methods were used to evaluate the recognition performance of different feature combinations. The results showed that the random forest method, trained using spectral power, pupil diameter, and heart rate variability, achieved the highest recognition accuracy of 83.13% for the three mental workload states. This study provides valuable reference information for multimodal recognition of mental workload states.
computer science, cybernetics,ergonomics