Abstract:In order to explore robotic grasping in unstructured and dynamic environments, this work addresses the visual perception phase involved in the task. This phase involves the processing of visual data to obtain the location of the object to be grasped, its pose and the points at which the robot`s grippers must make contact to ensure a stable grasp. For this, the Cornell Grasping dataset is used to train a convolutional neural network that, having an image of the robot`s workspace, with a certain object, is able to predict a grasp rectangle that symbolizes the position, orientation and opening of the robot`s grippers before its closing. In addition to this network, which runs in real-time, another one is designed to deal with situations in which the object moves in the environment. Therefore, the second network is trained to perform a visual servo control, ensuring that the object remains in the robot`s field of view. This network predicts the proportional values of the linear and angular velocities that the camera must have so that the object is always in the image processed by the grasp network. The dataset used for training was automatically generated by a Kinova Gen3 manipulator. The robot is also used to evaluate the applicability in real-time and obtain practical results from the designed algorithms. Moreover, the offline results obtained through validation sets are also analyzed and discussed regarding their efficiency and processing speed. The developed controller was able to achieve a millimeter accuracy in the final position considering a target object seen for the first time. To the best of our knowledge, we have not found in the literature other works that achieve such precision with a controller learned from scratch. Thus, this work presents a new system for autonomous robotic manipulation with high processing speed and the ability to generalize to several different objects.

Depth-Based Visual Servoing Using Low-Accurate Arm

Visual Servoing with an Uncalibrated Eye-in-hand Camera

Hybrid Multi-camera Visual Servoing to Moving Target

Depth-PC: A Visual Servo Framework Integrated with Cross-Modality Fusion for Sim2Real Transfer

Hand-eye Robotic Visual Servoing Around Moving Objects Using Active Deformable Models

A Motion Planning Method for Visual Servoing Using Deep Reinforcement Learning in Autonomous Robotic Assembly

Autonomous Robotic Manipulation: Real-Time, Deep-Learning Approach for Grasping of Unknown Objects

Neuromorphic Eye-in-Hand Visual Servoing

Vision-Based Robotic Object Grasping—A Deep Reinforcement Learning Approach

Adaptive Visual Servoing Using Point and Line Features With an Uncalibrated Eye-in-Hand Camera

Self-Supervised Learning of Visual Servoing for Low-Rigidity Robots Considering Temporal Body Changes

Visually Guided Object Grasping

Real-Time Deep Learning Approach to Visual Servo Control and Grasp Detection for Autonomous Robotic Manipulation

Adaptive visual servoing using angle and distance

Toward Scalable Visual Servoing Using Deep Reinforcement Learning and Optimal Control

Visual Servoing Based on Geometric Features by Surface Structured Light

Deep learning-based method for vision-guided robotic grasping of unknown objects

Object Detection and Pose Estimation from RGB and Depth Data for Real-time, Adaptive Robotic Grasping

Visual Servoing for Pose Control of Soft Continuum Arm in a Structured Environment

Robotic Visual Servoing Around a Static Target: An Example of Controlled Active Vision

Uncalibrated Visual Servoing Of Mobile Manipulators With An Eye-To-Hand Camera