Abstract:Navigation and autonomous operation in farmland or greenhouse environments present significant challenges for agricultural mobile robots on account of unstructured scene conditions and complex task requirements. The immersive interfaces based on human–robot interaction have been extensively developed, which bridge the high‐level decision of human beings and the precise sensing and high motion ability of robots. In this study, we propose a human–robot interaction framework based on three‐dimensional mapping and virtual reality (VR) visualization. On the construction client, dense three‐dimensional point‐cloud maps are created based on the simultaneous localization and mapping real‐time appearance‐based mapping algorithm with a Kinect‐style depth camera. Subsequently, point‐cloud filtering and mesh creation methods are introduced for postprocessing the map models. A data communication network is employed to transmit the optimized models to the VR‐based exploration client. Utilizing VR visualization, the operator can gain an intuitive and comprehensive understanding of the environments to be explored. The three‐dimensional agricultural scenes are mapped into VR models through the system, which can combine the physical worlds with the VR spaces in a better manner. Compared with the video streaming‐based method for three‐dimensional mapping, the map models with colors and textures are entirely displayed in the VR headset, thus providing robust and global human–robot interaction interfaces and enhancing the immersive exploration for human beings to a considerable extent. Experimental results indicate that the three‐dimensional map models exhibit relatively sufficient construction integrity and yield favorable optimization effects. Additionally, according to a user questionnaire survey, immersive VR‐based interfaces can be more effectively utilized for the three‐dimensional mapping of mobile robots. Research evidence and results demonstrate that our proposed system framework offers positive assistance and support for enhancing human–robot interaction and executing specific robotic tasks in unstructured field agricultural environments. Code implementation and related data sets are shared at the link https://github.com/WangDongBUAA/Mapping_Interfacing.

VRKitchen: an Interactive 3D Environment for Learning Real Life Cooking Tasks

VRKitchen: an Interactive 3D Virtual Environment for Task-oriented Learning

Interactive Mixed Reality Cooking Assistant for Unskilled Operating Scenario

VRKitchen2.0-IndoorKit: A Tutorial for Augmented Indoor Scene Building in Omniverse

Infrastructure of Web-based VR-form virtual laboratory

Words in Kitchen: An Instance of Leveraging Virtual Reality Technology to Learn Vocabulary

An Intelligent Virtual Environment to Support Cognitive Rehabilitation: Relearning Meal Preparation Skills for Independent Living

Using Virtual Reality to Enhance Food Technology Education

The Chef's Hat Simulation Environment for Reinforcement-Learning-Based Agents

Augmented Reality Based Interactive Cooking Guide

VRGym: A Virtual Testbed for Physical and Interactive AI

Learning from demonstrations: An intuitive VR environment for imitation learning of construction robots

Reinforcement Learning for Sparse-Reward Object-Interaction Tasks in First-person Simulated 3D Environments

VRFromX: from Scanned Reality to Interactive Virtual Experience with Human-in-the-Loop.

The Design of a Virtual Prototyping System for Authoring Interactive Virtual Reality Environments from Real-World Scans.

Three‐dimensional mapping and immersive human–robot interfacing utilize Kinect‐style depth cameras and virtual reality for agricultural mobile robots

Interaction Challenges in AI Equipped Environments Built to Teach Foreign Languages Through Dialogue and Task-Completion

VirtualHome: Simulating Household Activities via Programs

The Ingredients of Real-World Robotic Reinforcement Learning

A virtual reality platform for dynamic human-scene interaction

Virtual Reality for Memory Rehabilitation.