Interference inhibition of multimodal information in digital interfaces and its rule of cognitive processing

Junkai Shao,Wenzhe Tang,Jing Ji,Chengqi Xue,Feng Lu

DOI: https://doi.org/10.1002/hfm.21054

2024-09-27

Human Factors and Ergonomics in Manufacturing & Service Industries

Abstract:Human Factors and Ergonomics in Manufacturing &Service Industries, EarlyView.

engineering, manufacturing,ergonomics

What problem does this paper attempt to address?

The Attention Aware System: A Human-Computer Interaction Interface in the Multi-Task Context

Wen-jing YIN,Bai-hua XU,Xiang-yu CUI

DOI: https://doi.org/10.3969/j.issn.1006-6020.2006.03.015

2006-01-01

Abstract:The attention aware system is such a human-computer interaction system that aims to support users in better maintaining or switching their attention focus in the multi-task context.Attention aware systems are designed based on the theories of cognitive psychology and cognitive ergonomics.This paper offers a review of the current approach to attention aware systems,which is composed of three parts:to detect user's current attentional state,to make an evaluation of user's interruptibility and to optimize the style of notifications.Finally,the paper discusses the most promising research direction for development of attention aware systems
Effect of human–machine interface infotainment systems and automated vehicles on driver distraction

Elahe Abbasi,Yueqing Li,Yi Liu,Ruobing Zhao

DOI: https://doi.org/10.1002/hfm.21049

2024-09-04

Human Factors and Ergonomics in Manufacturing & Service Industries

Abstract:Human Factors and Ergonomics in Manufacturing &Service Industries, EarlyView.

engineering, manufacturing,ergonomics
Human-Machine Plan Conflict and Conflict Resolution in a Visual Search Task

Yunxian Pan,Jie Xu

DOI: https://doi.org/10.1016/j.ijhcs.2024.103377

2025-01-01

Abstract:With rapid technological development, humans are more likely to cooperatively work with intelligence systems in everyday life and work. Similar to interpersonal teamwork, the effectiveness of human-machine teams is affected by conflicts. Some human-machine conflict scenarios occur when neither the human nor the system was at fault, for example, when the human and the system formulated different but equally effective plans to achieve the same goal. In this study, we conducted two experiments to explore the effects of human-machine plan conflict and the different conflict resolution approaches (human adapting to the system, system adapting to the human, and transparency design) in a computer-aided visual search task. The results of the first experiment showed that when conflicts occurred, the participants reported higher mental load during the task, performed worse, and provided lower subjective evaluations towards the aid. The second experiment showed that all three conflict resolution approaches were effective in maintaining task performance, however, only the transparency design and the human adapting to the system approaches were effective in reducing mental load and improving subjective evaluations. The results highlighted the need to design appropriate human-machine conflict resolution strategies to optimize system performance and user experience.
Design of Human-Machine Interaction Interface for Autonomous Vehicles Based on Multidimensional Perceptual Context

Jin Lu,Jun Ma,Zaiyan Gong

DOI: https://doi.org/10.1155/2021/5859800

2021-11-25

Scientific Programming

Abstract:The objective of the study is to provide guidance for automobile human-computer interface design through the research, practice, and evaluation of intelligent driving automobile human-computer interfaces. In this study, the methods of intelligent vehicle automatic cruise control function, theoretical models of the situation analysis, and a three-level information architecture are proposed and designed. During the study, interface space layout is combined with the interactive interface design. The results obtained on the basis of the proposed three levels of perception and prediction of situation awareness are combined with typical application scenarios. The information on AR-HUD and W-HUD in anterior cingulate cortex (or ACC) function is analyzed. The feasibility of the theory is verified through the design practice of information architecture design, key prototype, and typical effect diagram. The human-computer interaction interface based on situational awareness can effectively clarify the display content and process of information and improve the usability of the interface.

computer science, software engineering
Human-machine Interface for a Master-Slave Hydraulic Manipulator with Vision Enhancement and Auditory Feedback

Ruqi Ding,Min Cheng,Zenan Han,Feng Wang,Bing Xu

DOI: https://doi.org/10.1016/j.autcon.2022.104145

IF: 10.3

2022-01-01

Automation in Construction

Abstract:This paper proposes a novel human-machine interface (HMI) for addressing the operational inefficiency problem owing to the asymmetrical workspace mapping in master-slave hydraulic manipulators. An augmented reality based head-mounted display system combining binocular vision with real-time virtual modeling was designed for demonstrating the commanded motion, and a hidden-appearance visualization rule was established for reducing visual interference, caused by the occlusion effect. An auditory feedback interface was designed for providing an additional cue for the master-slave position error. Eighteen participants were required for executing two typical tasks for validation. The test results indicated that the task completion time was reduced using the proposed HMI, and the number of collisions with surrounding items was reduced. Although the subjective feeling evaluation showed no statistical improvement owing to the visual fatigue and limited definition of the augmented reality scene, the proposed HMI is a practical method for improving the operational efficiency of hydraulic manipulators.
Review of Sensory Feedback Simulation Methods in Multi-modal Human-computer Interaction

SUN Ling-yun,WANG Ke-xing,CHEN Pei,ZHOU Zi-hong,YANG Chang-yuan,YANG Guang

DOI: https://doi.org/10.19554/j.cnki.1001-3563.2023.08.008

2023-01-01

Abstract:The work aims to sort out the research status of sensory feedback simulation methods in multi-modal human-computer interaction and summarize the principles and implementation methods of related techniques in existing research, so as to provide references for the subsequent research and practice. Firstly, the techniques used to simulate haptic,olfactory, and gustatory feedback were summarized respectively by reviewing literature and studying typical cases. Then,the cognitive theories about multi-modal sensory integration were outlined and the specific implementation methods were introduced by taking visual-aural-haptic and visual-aural-olfactory feedback integration as examples. Finally, the application of multi-modal sensory feedback in typical scenarios such as medical rehabilitation, virtual shopping, and intelligent education was expounded. The research of sensory feedback simulation methods in multi-modal human-computer interaction is expected to support a new type of human-computer interaction with low load and high efficiency. It can even deliver an interactive experience that goes beyond reality, representing a wide range of application potential.
Intelligence Methods of Multi-Modal Information Fusion in Human-Computer Interaction

Minghao YANG,Jianhua TAO

DOI: https://doi.org/10.1360/n112017-00211

2018-01-01

Scientia Sinica Informationis

Abstract:We first introduce the concepts of single-modal information processing and multi-modal information fusion in cognitive science. Some classical multi-modal information fusion models and their computer implementations in history are also explained. Under the conditions that each channels information can be obtained, and their features could be unified representation synchronously, the fusion of multi-modal information can be transformed into classification or regression problems. For practical human-computer interaction systems, the performance of multi-modal information fusion largely relies on the accuracy of the single-modal information identification and the design of the interactive system. We present a practical example of multi-modal information fusion system, and discuss its performances on human computer interaction. Finally, the possible and important development trends for multi-modal human-computer interaction techniques and systems are discussed.
Designing Transparent Interaction For Ubiquitous Computing: Theory And Application

Weining Yue,Heng Wang,Guoping Wang

DOI: https://doi.org/10.1007/978-3-540-73105-4_37

2007-01-01

Abstract:Designing transparent interaction is important for ubiquitous computing (ubicomp). A psychology framework that characterizes user's cognitive behavior in ubicomp environments would be invaluable for guiding the interaction design to be optimally compatible with human capabilities and limitations. By analyzing the cognitive skill and attention selectivity, such a framework is proposed in this paper. Correspondingly, a context-sensitive multimodal architecture is presented on the level of technology. A case study, where the theory was implemented in a handheld hypermedia guide and deployed into the context of authentic use, is then discussed.
Multimodal Information Processing For Affective Computing

Jianhua Tao

DOI: https://doi.org/10.1007/978-0-387-73819-2_9

2010-01-01

Abstract: Affective computing is interaction that relates to, arises from or deliberately influences emotions [1]; it tries to assign computers the human-like capabilities of observation, interpretation and generation of affect features. It is an important topic in human–computer interaction (HCI), because it helps increase the quality of human to computer communications.
Intuitive interaction flow: A Dual-Loop Human-Machine Collaboration Task Allocation Model and an experimental study

Jiang Xu,Qiyang Miao,Ziyuan Huang,Yilin Lu,Lingyun Sun,Tianyang Yu,Jingru Pei,Qichao Zhao

2024-10-17

Abstract:This study investigates the issue of task allocation in Human-Machine Collaboration (HMC) within the context of Industry 4.0. By integrating philosophical insights and cognitive science, it clearly defines two typical modes of human behavior in human-machine interaction(HMI): skill-based intuitive behavior and knowledge-based intellectual behavior. Building on this, the concept of 'intuitive interaction flow' is innovatively introduced by combining human intuition with machine humanoid intelligence, leading to the construction of a dual-loop HMC task allocation model. Through comparative experiments measuring electroencephalogram (EEG) and electromyogram (EMG) activities, distinct physiological patterns associated with these behavior modes are identified, providing a preliminary foundation for future adaptive HMC frameworks. This work offers a pathway for developing intelligent HMC systems that effectively integrate human intuition and machine intelligence in Industry 4.0.

Human-Computer Interaction
P‐13: Presentation of Visual and Audio Information for a Human‐Computer Interface

Yanfang Liu,Xiaolan Fu,Changxu Wu

DOI: https://doi.org/10.1889/1.1831930

2001-01-01

SID Symposium Digest of Technical Papers

Abstract:One experiment was conducted to compare 4 information displays on a multimodal interface. The results show that in term of memory accuracy, cognitive workload, the visual display of information was more predominant than the audio display of information. This research on multimodal information display has focused on how complementary information in different modalities can be combined to arrive at a more informative representation.
Noncontact Human-Machine Interface Using Complementary Information Fusion Based on MEMS and Triboelectric Sensors

Xianhao Le,Qiongfeng Shi,Zhongda Sun,Jin Xie,Chengkuo Lee

DOI: https://doi.org/10.1002/advs.202201056

Abstract:Current noncontact human-machine interfaces (HMIs) either suffer from high power consumption, complex signal processing circuits, and algorithms, or cannot support multidimensional interaction. Here, a minimalist, low-power, and multimodal noncontact interaction interface is realized by fusing the complementary information obtained from a microelectromechanical system (MEMS) humidity sensor and a triboelectric sensor. The humidity sensor composed of a two-port aluminum nitride (AlN) bulk wave resonator operating in its length extensional mode and a layer of graphene oxide (GO) film with uniform and controllable thickness, possesses an ultra-tiny form factor (200 × 400 µm2 ), high signal strength (Q = 1729.5), and low signal noise level (±0.31%RH), and is able to continuously and steadily interact with an approaching finger. Meanwhile, the facile triboelectric sensor made of two annular aluminum electrodes enables the interaction interface to rapidly recognize the multidirectional finger movements. By leveraging the resonant frequency changes of the humidity sensor and output voltage waveforms of the triboelectric sensor, the proposed interaction interface is successfully demonstrated as a game control interface to manipulate a car in virtual reality (VR) space and a password input interface to enter high-security 3D passwords, indicating its great potential in diversified applications in the future Metaverse.
Visual Target Interaction Modeling with Multichannel Data Fusion

Mingwei Zhao,Changlong Cai,Wenbo Huang,Longtao Mu,Hao Zhang,Lingzhi Yan,Yonghui Liu,Jiaming Wang

DOI: https://doi.org/10.1142/s0218001424550085

IF: 1.261

2024-09-19

International Journal of Pattern Recognition and Artificial Intelligence

Abstract:International Journal of Pattern Recognition and Artificial Intelligence, Ahead of Print. Facing the demand for accurate, fast, and natural human–computer interaction in the multi-screen control environment, the traditional exchange method of frequent switching control of multi-screen through mouse, keyboard, and other manual methods has problems of low efficiency, weak flexibility, and poor experience. In this paper, we research multi-screen precision control technology based on eye movement to break through the difficulties of precise recognition of human vision under complex environments and poor stability of vision estimation under frequent switching control of multi-screen. This study explores the technology for controlling multiple screens with high precision by tracking eye movements. It overcomes the challenges associated with reliably discerning human visual focus in intricate settings and the instability of visual assessments during rapid transitions between multiple screens. The experimental results of 1143 eye-movement capture tasks are studied, including the accurate recognition technology of human head posture and pupil gaze direction under complex backgrounds, illumination, and different visual field environments, and the theoretical modeling method of multi-screen precise target interaction based on eye-movement is explored. The expression data will be processed using convolutional neural networks, and the dataset will be trained through Python programming and tested on test samples collected by simulated flight crews in the laboratory with an accuracy rate of more than 85%.

computer science, artificial intelligence
Multimodal Human–Robot Interaction for Human‐Centric Smart Manufacturing: A Survey

Tian Wang,Pai Zheng,Shufei Li,Lihui Wang

DOI: https://doi.org/10.1002/aisy.202300359

IF: 7.298

2023-12-01

Advanced Intelligent Systems

Abstract:Human–robot interaction (HRI) has escalated in notability in recent years, and multimodal communication and control strategies are necessitated to guarantee a secure, efficient, and intelligent HRI experience. In spite of the considerable focus on multimodal HRI, comprehensive disquisitions delineating various modalities and intricately analyzing their combinations remain elusive, consequently limiting holistic understanding and future advancements. This article aspires to bridge this inadequacy by conducting a profound exploration of multimodal HRI, predominantly concentrating on four principal modalities: vision, auditory and language, haptics, and physiological sensing. An extensive review encapsulating algorithmic dissection, interface devices, and applicative dimensions forms part of this discourse. This manuscript distinctively combines multimodal HRI with cognitive science, deeply probing into the three dimensions, perception, cognition, and action, thereby demystifying algorithms intrinsic to multimodal HRI. Finally, it accentuates the empirical challenges and contours preemptive trajectories for multimodal HRI in human‐centric smart manufacturing.
Multimodal Interfaces for Effective Teleoperation

Eleftherios Triantafyllidis,Christopher McGreavy,Jiacheng Gu,Zhibin Li

DOI: https://doi.org/10.48550/arXiv.2003.14392

IF: 6.4588

2020-03-31

Human-Computer Interaction

Abstract:Research in multi-modal interfaces aims to provide solutions to immersion and increase overall human performance. A promising direction is combining auditory, visual and haptic interaction between the user and the simulated environment. However, no extensive comparisons exist to show how combining audiovisuohaptic interfaces affects human perception reflected on task performance. Our paper explores this idea. We present a thorough, full-factorial comparison of how all combinations of audio, visual and haptic interfaces affect performance during manipulation. We evaluate how each interface combination affects performance in a study (N=25) consisting of manipulating tasks of varying difficulty. Performance is assessed using both subjective, assessing cognitive workload and system usability, and objective measurements, incorporating time and spatial accuracy-based metrics. Results show that regardless of task complexity, using stereoscopic-vision with the VRHMD increased performance across all measurements by 40% compared to monocular-vision from the display monitor. Using haptic feedback improved outcomes by 10% and auditory feedback accounted for approximately 5% improvement.
A Multi-modal Virtual-Real Fusion System for Multi-task Human-Computer Interaction.

Xinlei Zhang,Jiahui Yu,Yuxiang Sun,Min Li,Yang Song,Xianzhong Zhou

DOI: https://doi.org/10.1109/icnsc55942.2022.10004097

2022-01-01

Abstract:Due to the complexity of the task and the diversification of the scene, the traditional interactive control method can no longer meet the requirements of users. To solve this problem, a multi-modal virtual-real fusion system for multi-task human-computer interaction is proposed in this paper, which integrates eye movement, gesture and voice. Frist, aiming at the phenomenon of multi-task and multi-modal, a task-modal matching model is established. Then, the task-modal matching model is abstracted into a multi-objective optimization problem, and a method for solving this problem is designed and a matching scheme is successfully obtained. Meanwhile, the construction of the system is completed for the virtual-real fusion environment, and the control of unmanned car and virtual car is realized. The system can carry out multi-modal interaction and complete multiple tasks in real scene, virtual scene, parallel system and virtual-real fusion scene. Finally, the experiment proves the stability and reliability of the system.
NeuroManagement and Intelligent Computing Method on Multimodal Interaction

Weihui Dai

DOI: https://doi.org/10.1145/3357160.3357664

2019-01-01

Abstract:In recent years, the research of multimodal interaction has made rapid progress owing to the development of artificial intelligence and big data technology, as well as the new findings of human psychology and behavior's study. Nevertheless, the human- computer's multimodal interaction is actually a dynamic and evolutionary process, so there are still some difficult issues that need to be further explored such as how to comprehensively deal with the multimodal fusion data, adaptively consider the user's personalized habits, and precisely understand the expresser's intentions. NeuroManagement, an emerging interdisciplinary subject to study human's management based on neural mechanism, provides new theory and methodology for probing into the above issues. From the perspective of NeuroManagement, this keynote paper analyzes the user's neural mechanism of multimodal interactions, discusses the characteristics of neural activities in that process, and proposes a framework of intelligent computation method to understand and measure the user's intentions as well as his/her psychological states. In order to promote the interdisciplinary development of artificial intelligence and neuroscience, Neuroartificial Intelligence, a new subject field is proposed by this paper.
Effects of Complexity of Human Machine Interaction on Human Error in Digital Control System

ZOU Ping-ping,ZHANG Li,JIANG Jian-jun

2013-01-01

Abstract:According to field investigation and video observation,the new characteristics of digital human-machine interaction in digital control system are identified by comparing with the traditional analog control systems,such as the transformation of operator's role,the increases of cognitive load,the transformation from hard control to soft control and the more ample human machine interface.Through the four key influencing factors of the complexity of human machine interaction,named operator,task,interface management and manipulation,the effects on human factor error are analyzed,expecting to provide theoretical support for reducing the complexity of human machine interaction.
User Interaction Interface Design and Innovation Based on Artificial Intelligence Technology

Xuanyi Li,Haotian Zheng,Jianlong Chen,Yanqi Zong,Liqiang Yu

DOI: https://doi.org/10.53469/jtpes.2024.04(03).01

2024-03-19

Abstract:At a time when artificial intelligence is widely used in all walks of life, the way users interact with the digital world also needs to incorporate intelligent elements to reduce the cost of connectivity. This cost can be quantified through "experience metrics", which reveal the problems users encounter when using the interface (UI), and make targeted optimization. With AI, deep learning and prediction of user behavior can be achieved to anticipate and address potential barriers to use in UI design. This will not only improve the user experience, but also promote the development of UI design in a more user-friendly and intelligent direction. Through accurate analysis of experience indicators and combined with AI technology to optimize design, the gap between users and the digital world can be greatly reduced, making digital products more suitable for user needs and achieving seamless interactive experience.
Affective multimodal human-computer interaction

Maja Pantic,Nicu Sebe,Jeffrey F. Cohn,Thomas Huang

DOI: https://doi.org/10.1145/1101149.1101299

2005-01-01

Abstract:Social and emotional intelligence are aspects of human intelligence that have been argued to be better predictors than IQ for measuring aspects of success in life, especially in social interactions, learning, and adapting to what is important. When it comes to machines, not all of them will need such skills. Yet to have machines like computers, broadcast systems, and cars, capable of adapting to their users and of anticipating their wishes, endowing them with the ability to recognize user's affective states is necessary. This article discusses the components of human affect, how they might be integrated into computers, and how far are we from realizing affective multimodal human-computer interaction.

Interference inhibition of multimodal information in digital interfaces and its rule of cognitive processing

The Attention Aware System: A Human-Computer Interaction Interface in the Multi-Task Context

Effect of human–machine interface infotainment systems and automated vehicles on driver distraction

Human-Machine Plan Conflict and Conflict Resolution in a Visual Search Task

Design of Human-Machine Interaction Interface for Autonomous Vehicles Based on Multidimensional Perceptual Context

Human-machine Interface for a Master-Slave Hydraulic Manipulator with Vision Enhancement and Auditory Feedback

Review of Sensory Feedback Simulation Methods in Multi-modal Human-computer Interaction

Intelligence Methods of Multi-Modal Information Fusion in Human-Computer Interaction

Designing Transparent Interaction For Ubiquitous Computing: Theory And Application

Multimodal Information Processing For Affective Computing

Intuitive interaction flow: A Dual-Loop Human-Machine Collaboration Task Allocation Model and an experimental study

P‐13: Presentation of Visual and Audio Information for a Human‐Computer Interface

Noncontact Human-Machine Interface Using Complementary Information Fusion Based on MEMS and Triboelectric Sensors

Visual Target Interaction Modeling with Multichannel Data Fusion

Multimodal Human–Robot Interaction for Human‐Centric Smart Manufacturing: A Survey

Multimodal Interfaces for Effective Teleoperation

A Multi-modal Virtual-Real Fusion System for Multi-task Human-Computer Interaction.

NeuroManagement and Intelligent Computing Method on Multimodal Interaction

Effects of Complexity of Human Machine Interaction on Human Error in Digital Control System

User Interaction Interface Design and Innovation Based on Artificial Intelligence Technology

Affective multimodal human-computer interaction