Abstract:In recent years, digital twin technology has become increasingly popular and been widely applied across different industries including air traffic management (ATM). While previous studies on digital twin in ATM focused on factors such as system architecture, synthetic data generation and transfer, and real-time simulation, there is currently a lack of research on visualization and interaction design of digital twin from human factors’ perspective. Moreover, recent advances in virtual and mixed reality (VR/MR) technology give potential for designing interactive digital twin in 3D space that can assist human operators in performing their duties. Therefore, this research aims to explore visualization and interaction design of an MR-based digital twin prototype that can assist air traffic controllers (ATCOs) in carrying out ground control tasks. Specifically, MR headsets (Microsoft HoloLens 2) are leveraged to project out-of-tower view of airport traffic onto a 3D printed airport model at a 1:1 scale. The spatially aligned tangible system enables ATCOs to perform typical ATM operations by directly touching the 3D printed airport model. In addition, collaborative features are implemented into the system, which enable multiple ATCOs to have a shared view of the airport traffic and collaboratively perform ATM operations when wearing the MR headsets. Finally, a user study was conducted with the recruitment of ten licensed ATCOs to investigate usability of the system and ATCOs’ workload, situational awareness and trust when using the system. The feedback from the ATCOs in the user study suggested that: (1) The visual and interactive design were acceptable; (2) The ATCOs indicated varied workload under different scenarios; (3) The ATCOs developed a high level of shared situational awareness with shared view of the system; and (4) The ATCOs were open and confident to use the system in its’ mature form. The findings show that the MR-based tangible airport digital tower system has high potential in collaborative ATM operations by improving user performance and user experience.

Mixed Reality Interface for Digital Twin of Plant Factory

Digital Twins Visualization of Large Electromechanical Equipment

A Mixed Reality Interface for a Digital Twin Based Crane

An Integrated Mobile Augmented Reality Digital Twin Monitoring System

Mixed reality-based digital twin implementation approach for flexible manufacturing system design

A 3D Mixed Reality Interface for Human-Robot Teaming

Toward Programming a Collaborative Robot by Interacting with Its Digital Twin in a Mixed Reality Environment

Mobile Mixed-Reality Interfaces That Enhance Human–Robot Interaction in Shared Spaces

Mixed Reality for Mechanical Design and Assembly Planning

Interactive control of manufacturing assemblies with Mixed Reality

Using Mixed Reality for Control and Monitoring of Robot Model Based on Robot Operating System 2

Tangible digital twin with shared visualization for collaborative air traffic management operations

iFANnpp: Nuclear Power Plant Digital Twin for Robots and Autonomous Intelligence

Digital twin framework for smart greenhouse management using next-gen mobile networks and machine learning

An Approach to Develop Digital Twins in Industry

Advancing Mixed Reality Digital Twins Through 3D Reconstruction of Fresh Produce

Immersive Assistance System for Intuitive Robot Programming using Mixed-Reality and Digital Twin

Visual Exploration and Planning of the Automated Material Handling System for Smart Factory in the Immersive Environment.

Towards a Cloud-based Architecture for Visualization and Augmented Reality to Support Collaboration in Manufacturing Automation

Plant Factory: A New Playground of Industrial Communication and Computing

Digital twin and virtual reality: a co-simulation environment for design and assessment of industrial workstations