Abstract:Navigation and autonomous operation in farmland or greenhouse environments present significant challenges for agricultural mobile robots on account of unstructured scene conditions and complex task requirements. The immersive interfaces based on human–robot interaction have been extensively developed, which bridge the high‐level decision of human beings and the precise sensing and high motion ability of robots. In this study, we propose a human–robot interaction framework based on three‐dimensional mapping and virtual reality (VR) visualization. On the construction client, dense three‐dimensional point‐cloud maps are created based on the simultaneous localization and mapping real‐time appearance‐based mapping algorithm with a Kinect‐style depth camera. Subsequently, point‐cloud filtering and mesh creation methods are introduced for postprocessing the map models. A data communication network is employed to transmit the optimized models to the VR‐based exploration client. Utilizing VR visualization, the operator can gain an intuitive and comprehensive understanding of the environments to be explored. The three‐dimensional agricultural scenes are mapped into VR models through the system, which can combine the physical worlds with the VR spaces in a better manner. Compared with the video streaming‐based method for three‐dimensional mapping, the map models with colors and textures are entirely displayed in the VR headset, thus providing robust and global human–robot interaction interfaces and enhancing the immersive exploration for human beings to a considerable extent. Experimental results indicate that the three‐dimensional map models exhibit relatively sufficient construction integrity and yield favorable optimization effects. Additionally, according to a user questionnaire survey, immersive VR‐based interfaces can be more effectively utilized for the three‐dimensional mapping of mobile robots. Research evidence and results demonstrate that our proposed system framework offers positive assistance and support for enhancing human–robot interaction and executing specific robotic tasks in unstructured field agricultural environments. Code implementation and related data sets are shared at the link https://github.com/WangDongBUAA/Mapping_Interfacing.

The Research on Mobile Robot Simulation and Visualization under Virtual Reality

HumRoboSim: an Autonomous Humanoid Robot Simulation System

Virtual Avatar Control Using Wireless Sensors.

Research on the Real-Time Mechatronic System Simulation Based on Virtual Reality

Research on Virtual Roaming Technology of Urban Scene Based on Multi-mode Human-Computer Interaction

Review of Three-Dimensional Environment Information Perception and Reconstruction Methods for Mobile Robot Based on Multi-Sensor Fusion

Digital twin for mobile robot modeling and simulation

Intuitive Robot Integration via Virtual Reality Workspaces

Research and Realization of Robot Experiment System Based on Virtuality and Actuality

Multi-View Reconstruction Fusing Ultrasonic Phased Array and Camera for Mobile Robots in Simulation Environment

Multi-sensor Fusion for Interactive Visual Computing in Mixed Environment.

A Simulation & Training System of Robot Assisted Surgery Based on Virtual Reality

Three‐dimensional mapping and immersive human–robot interfacing utilize Kinect‐style depth cameras and virtual reality for agricultural mobile robots

A VR System for Immersive Teleoperation and Live Exploration with a Mobile Robot

Human-Guided Robot 3d Mapping Using Virtual Reality Technology

A Simulation and Training Systems of Robot Assisted Surgery Based on Virtual Reality.

Virtual Reality Teleoperation System for Mobile Robot Manipulation

Seeing Thru Walls: Visualizing Mobile Robots in Augmented Reality

Seamless Virtual Reality with Integrated Synchronizer and Synthesizer for Autonomous Driving

Application of Virtual Reality in Developing the Digital Twin for an Integrated Robot Learning System

Synchronized Control of Robotic Arm based on Virtual Reality and IMU Sensing