Abstract:Grasping is critical for intelligent robots to accomplish sophisticated tasks. Even with multimodal sensor fusion, accurately and reliably estimating grasp poses for complex-shaped objects remains a challenge. In this paper, we design a vision-based grasping platform for a more general case, that is, grasping a variety of objects by a simple parallel gripper with the grasp detection model consuming RGB sensing or depth sensing. Focusing on the grasp pose estimation part, we propose a deep grasp detector that uses a densely connected Feature Pyramid Network (FPN) feature extractor and multiple two-stage detection units to achieve dense grasp pose predictions. Specifically, for the feature extractor, the fusion of different layer feature maps can increase both the model's capacity to detect the various size grasp areas and the accuracy of the regressed grasp positions. For each of the two-stage detection unit, the first stage generates horizontal candidate grasp areas, while the second stage refines them to predict the rotated grasp poses. We train and validate our grasp pose estimation algorithm on the Cornell Grasp Dataset and the Jacquard Dataset. The model achieves the detection accuracy of 93.3% and 89.6%, respectively. We further design real-world grasp experiments to verify the effectiveness of our vision-based robotic grasping system. The real scenario trials validate that the system is capable of grasping unseen objects, in particular, achieving robust and accurate grasp pose detection and gripper opening width measurement based on depth sensing only.

Grasping Recognition Based on Visual Awareness in an Augmented Reality System

A Bare Hand Controlled Ar Map Navigation System

Visualizing Robot Intent for Object Handovers with Augmented Reality

A touchable virtual screen interaction system with handheld Kinect camera

Handling Occlusions in Video-Based Augmented Reality Using Depth Information

An Optical See-Through Augmented Reality System with Gesture-Based Interaction

VirtualGrasp

A Realtime Virtual Grasping System for Manipulating Complex Objects.

Control with Vergence Eye Movement in Augmented Reality See-Through Vision

The Prosthetic Hand Can Grasp Occluded Objects with the Help of 3rd-Eye Based on Artificial Vision and AR Technology

Commonsense Knowledge-Driven Joint Reasoning Approach for Object Retrieval in Virtual Reality

Self-supervised 6-DoF Robot Grasping by Demonstration via Augmented Reality Teleoperation System

GrabAR: Occlusion-aware Grabbing Virtual Objects in AR

A Vision-based Robot Grasping System

Realtime Hand-Object Interaction Using Learned Grasp Space for Virtual Environments

What You See is What You Grasp: User-Friendly Grasping Guided by Near-eye-tracking

Grasping 3D Objects With Virtual Hand in VR Environment

Grasping Object: Challenges and Innovations in Robotics and Virtual Reality

A Wearable Visually Impaired Assistive System Based on Semantic Vision SLAM for Grasping Operation

Gesture-based target acquisition in virtual and augmented reality.

Gaze-Vergence-Controlled See-Through Vision in Augmented Reality