Abstract:Navigation and autonomous operation in farmland or greenhouse environments present significant challenges for agricultural mobile robots on account of unstructured scene conditions and complex task requirements. The immersive interfaces based on human–robot interaction have been extensively developed, which bridge the high‐level decision of human beings and the precise sensing and high motion ability of robots. In this study, we propose a human–robot interaction framework based on three‐dimensional mapping and virtual reality (VR) visualization. On the construction client, dense three‐dimensional point‐cloud maps are created based on the simultaneous localization and mapping real‐time appearance‐based mapping algorithm with a Kinect‐style depth camera. Subsequently, point‐cloud filtering and mesh creation methods are introduced for postprocessing the map models. A data communication network is employed to transmit the optimized models to the VR‐based exploration client. Utilizing VR visualization, the operator can gain an intuitive and comprehensive understanding of the environments to be explored. The three‐dimensional agricultural scenes are mapped into VR models through the system, which can combine the physical worlds with the VR spaces in a better manner. Compared with the video streaming‐based method for three‐dimensional mapping, the map models with colors and textures are entirely displayed in the VR headset, thus providing robust and global human–robot interaction interfaces and enhancing the immersive exploration for human beings to a considerable extent. Experimental results indicate that the three‐dimensional map models exhibit relatively sufficient construction integrity and yield favorable optimization effects. Additionally, according to a user questionnaire survey, immersive VR‐based interfaces can be more effectively utilized for the three‐dimensional mapping of mobile robots. Research evidence and results demonstrate that our proposed system framework offers positive assistance and support for enhancing human–robot interaction and executing specific robotic tasks in unstructured field agricultural environments. Code implementation and related data sets are shared at the link https://github.com/WangDongBUAA/Mapping_Interfacing.

Model Construction and Cooperative Control in VR

Advancing Virtual Reality Interaction: A Ring-Shaped Controller and Pose Tracking

Development of a Synchronized Human-Robot-Virtuality Interaction System Using Cooperative Robot and Motion Capture Device

Constrained Behavior Manipulation for Interactive Assembly in a Virtual Environment

Constraint-based Virtual Solid Modeling

An Approach to Solid Modeling in a Semi-Immersive Virtual Environment.

3d Interaction Oriented Object Model

Research of Motion Simulation for Humanoid Robot Based on VRML

Interactive task planning in virtual assembly.

Virtual reality in human-robot interaction: Challenges and benefits

An Integrated Virtual Design And Virtual Assembly Environment

Data and model hybrid-driven virtual reality robot operating system

Multi-User VR Cooperative Puzzle Game

Integrating Large Language Models with Multimodal Virtual Reality Interfaces to Support Collaborative Human-Robot Construction Work

Virtual Experiment of Innovative Construction Operations

Virtual reality interaction based on visual attention and kinesthetic information

Modeling Of Buildings For Collaborative Design In A Virtual Environment

Development of a VR-based Manipulation System for Dual-arm Robots

Virtual Manipulation in an Immersive Virtual Environment: Simulation of Virtual Assembly

Three‐dimensional mapping and immersive human–robot interfacing utilize Kinect‐style depth cameras and virtual reality for agricultural mobile robots

Exploring Configurations for Multi-user Communication in Virtual Reality