Abstract:A novel multimodal data processing and application framework to support the implementation of an aircraft human‐machine interaction assistant (AHMIA) is presented. The AHIMIA enables active‐command and passive‐response interactions with pilots and process three input modes, including voice inputs, situation inputs, and plan inputs. During aircraft operations, pilots rely on human‐machine interaction platforms to access essential information services. However, the development of a highly usable aerial assistant necessitates the incorporation of two interaction modes: active‐command and passive‐response modes, along with three input modes: voice inputs, situation inputs, and plan inputs. This research focuses on the design of an aircraft human‐machine interaction assistant (AHMIA), which serves as a multimodal data processing and application framework for human‐to‐machine interaction in a fully voice‐controlled manner. For the voice mode, a finetuned FunASR model is employed, leveraging private aeronautical datasets to enable specific aeronautical speech recognition. For the situation mode, a hierarchical situation events extraction model is proposed, facilitating the utilization of high‐level situational features. For the plan mode, a multi‐formations double‐code network plan diagram with a timeline is utilized to effectively represent plan information. Notably, to bridge the gap between human language and machine language, a hierarchical knowledge engine named process‐event‐condition‐order‐skill (PECOS) is introduced. PECOS provides three distinct products: the PECOS model, the PECOS state chart, and the PECOS knowledge description. Simulation results within the air confrontation scenario demonstrate that AHMIA enables active‐command and passive‐response interactions with pilots, thereby enhancing the overall interaction modality.

The Integration Method of Multimodal Human-Computer Interaction Framework

A Multimodal Interaction Framework for Command Post

A Parallel Multimodal Integration Framework and Application for Cake Shopping

A Multi-modal Virtual-Real Fusion System for Multi-task Human-Computer Interaction.

A Multimodal Interaction Model for Battlefield

Research on Multiple and Multimodal Interaction in C2

Multimodal Human-Computer Interaction Based on Bayesian Classification Algorithm

Multimodal Systems: Taxonomy, Methods, and Challenges

Intelligence Methods of Multi-Modal Information Fusion in Human-Computer Interaction

Human-Cyber-Physical Ubiquitous Intelligent Communication: System Architecture, Key Technologies, and Challenges.

Study on Human-Computer Interaction Framework of Pervasive Computing

An adaptive multidisciplinary integration framework for mechatronic systems collaborative design

Design and Implementation of Multi-modal Interaction System of Electronic Sand Table Application

An Architecture for Multimodal Multi-Agent Interactive System

Research on multimodal human-computer interaction technology based on audiovisual fusion

Application of the Multimodal Human-Computer Interaction Technology in Product Virtual Display

Enhancing Human–Robot Collaboration through a Multi-Module Interaction Framework with Sensor Fusion: Object Recognition, Verbal Communication, User of Interest Detection, Gesture and Gaze Recognition

Aircraft human‐machine interaction assistant design: A novel multimodal data processing and application framework

Research on Multimodal Human-Robot Interaction Based on Speech and Gesture.

Multimodal Human–Robot Interaction for Human‐Centric Smart Manufacturing: A Survey

A multisensory Interaction Framework for Human-Cyber–Physical System based on Graph Convolutional Networks