Abstract:Navigation and autonomous operation in farmland or greenhouse environments present significant challenges for agricultural mobile robots on account of unstructured scene conditions and complex task requirements. The immersive interfaces based on human–robot interaction have been extensively developed, which bridge the high‐level decision of human beings and the precise sensing and high motion ability of robots. In this study, we propose a human–robot interaction framework based on three‐dimensional mapping and virtual reality (VR) visualization. On the construction client, dense three‐dimensional point‐cloud maps are created based on the simultaneous localization and mapping real‐time appearance‐based mapping algorithm with a Kinect‐style depth camera. Subsequently, point‐cloud filtering and mesh creation methods are introduced for postprocessing the map models. A data communication network is employed to transmit the optimized models to the VR‐based exploration client. Utilizing VR visualization, the operator can gain an intuitive and comprehensive understanding of the environments to be explored. The three‐dimensional agricultural scenes are mapped into VR models through the system, which can combine the physical worlds with the VR spaces in a better manner. Compared with the video streaming‐based method for three‐dimensional mapping, the map models with colors and textures are entirely displayed in the VR headset, thus providing robust and global human–robot interaction interfaces and enhancing the immersive exploration for human beings to a considerable extent. Experimental results indicate that the three‐dimensional map models exhibit relatively sufficient construction integrity and yield favorable optimization effects. Additionally, according to a user questionnaire survey, immersive VR‐based interfaces can be more effectively utilized for the three‐dimensional mapping of mobile robots. Research evidence and results demonstrate that our proposed system framework offers positive assistance and support for enhancing human–robot interaction and executing specific robotic tasks in unstructured field agricultural environments. Code implementation and related data sets are shared at the link https://github.com/WangDongBUAA/Mapping_Interfacing.

Immersive Human-Machine Teleoperation Framework for Precision Agriculture: Integrating UAV-based Digital Mapping and Virtual Reality Control

Three‐dimensional mapping and immersive human–robot interfacing utilize Kinect‐style depth cameras and virtual reality for agricultural mobile robots

Real-time 3D unstructured environment reconstruction utilizing VR and Kinect-based immersive teleoperation for agricultural field robots

Augmented-Reality Enabled Crop Monitoring with Robot Assistance

AeroVr: Virtual Reality-based Teleoperation with Tactile Feedback for Aerial Manipulation

On the Use of Immersive Digital Technologies for Designing and Operating UAVs

Aerial UAV-IoT Sensing for Ubiquitous Immersive Communication and Virtual Human Teleportation

Multi-Agent Collaborative Framework for Automated Agriculture

Building an Aerial-Ground Robotics System for Precision Farming: An Adaptable Solution

Amplifying robotics capacities with a human touch: An immersive low-latency panoramic remote system

PSO-based Integration Navigation and LiDAR-based Remote Sensing Algorithms for 3D Agriculture Digital Modeling with Light Multi-Rotor UAV

Improving Usability, Efficiency, and Safety of UAV Path Planning through a Virtual Reality Interface

Virtual Reality Aided High-Quality 3D Reconstruction by Remote Drones

Digital twins in smart farming: An autoware-based simulator for autonomous agricultural vehicles

A Collaborative Path Planning Method for Intelligent Agricultural Machinery Based on Unmanned Aerial Vehicles

Intuitive robot teleoperation for civil engineering operations with virtual reality and deep learning scene reconstruction

Design, Multiperspective Investigations, and Performance Analysis of Multirotor Unmanned Aerial Vehicle for Precision Farming

Placement and drone flight path mapping of agricultural soil sensors using machine learning

AI Meets UAVs: A Survey on AI Empowered UAV Perception Systems for Precision Agriculture

Real-time Control of 3D Virtual Human Motion Using a Depth-Sensing Camera for Agricultural Machinery Training

Terrain Analytics for Precision Agriculture with Automated Vehicle Sensors and Data Fusion