Abstract:A novel multimodal data processing and application framework to support the implementation of an aircraft human‐machine interaction assistant (AHMIA) is presented. The AHIMIA enables active‐command and passive‐response interactions with pilots and process three input modes, including voice inputs, situation inputs, and plan inputs. During aircraft operations, pilots rely on human‐machine interaction platforms to access essential information services. However, the development of a highly usable aerial assistant necessitates the incorporation of two interaction modes: active‐command and passive‐response modes, along with three input modes: voice inputs, situation inputs, and plan inputs. This research focuses on the design of an aircraft human‐machine interaction assistant (AHMIA), which serves as a multimodal data processing and application framework for human‐to‐machine interaction in a fully voice‐controlled manner. For the voice mode, a finetuned FunASR model is employed, leveraging private aeronautical datasets to enable specific aeronautical speech recognition. For the situation mode, a hierarchical situation events extraction model is proposed, facilitating the utilization of high‐level situational features. For the plan mode, a multi‐formations double‐code network plan diagram with a timeline is utilized to effectively represent plan information. Notably, to bridge the gap between human language and machine language, a hierarchical knowledge engine named process‐event‐condition‐order‐skill (PECOS) is introduced. PECOS provides three distinct products: the PECOS model, the PECOS state chart, and the PECOS knowledge description. Simulation results within the air confrontation scenario demonstrate that AHMIA enables active‐command and passive‐response interactions with pilots, thereby enhancing the overall interaction modality.

A Civil Aircraft Cockpit Control Device Design Using Mixed Reality Device

MRStudio: A Mixed Reality Display System for Aircraft Cockpit

Elicitation and Evaluation of Hand-based Interaction Language for 3D Conceptual Design in Mixed Reality

An Integrated Virtual Design And Virtual Assembly Environment

Aircraft Maintenance Training and Assessment: the Potential Applications of Mixed Reality

Smart Cockpit Layout Design Framework Based on Human-Machine Experience

Real-Time Semi-Physical Simulation of Industry Design System Based on Virtual Reality Technology

The Application Value of Human-Vehicle Interaction Theory in Intelligent Cockpit Design

Mixed Reality for Mechanical Design and Assembly Planning

Towards Workplace Metaverse: A Human-Centered Approach for Designing and Evaluating XR Virtual Displays

Application of Physical Interactive Mixed Reality System Based on MLAT in the Field of Stage Performance

User Centered Design (VII): From Automated Flight Deck to Intelligent Flight Deck

Human-computer interaction interface design of flight simulator based on situation awareness

Interactive control of manufacturing assemblies with Mixed Reality

Development and Application of Integrated Human Machine Interaction Simulation System

Design of Human-Machine Interaction Interface for Autonomous Vehicles Based on Multidimensional Perceptual Context

Aircraft human‐machine interaction assistant design: A novel multimodal data processing and application framework

CoAug-MR: An MR-based Interactive Office Workstation Design System via Augmented Multi-Person Collaboration

User-centered framework for assessing the performance of smart car cockpits

Cabin Operation and Management Model Based on VR Technology

Research on Holographic Visualization Verification Platform for Construction Machinery Based on Mixed Reality Technology