Abstract:In order to explore robotic grasping in unstructured and dynamic environments, this work addresses the visual perception phase involved in the task. This phase involves the processing of visual data to obtain the location of the object to be grasped, its pose and the points at which the robot`s grippers must make contact to ensure a stable grasp. For this, the Cornell Grasping dataset is used to train a convolutional neural network that, having an image of the robot`s workspace, with a certain object, is able to predict a grasp rectangle that symbolizes the position, orientation and opening of the robot`s grippers before its closing. In addition to this network, which runs in real-time, another one is designed to deal with situations in which the object moves in the environment. Therefore, the second network is trained to perform a visual servo control, ensuring that the object remains in the robot`s field of view. This network predicts the proportional values of the linear and angular velocities that the camera must have so that the object is always in the image processed by the grasp network. The dataset used for training was automatically generated by a Kinova Gen3 manipulator. The robot is also used to evaluate the applicability in real-time and obtain practical results from the designed algorithms. Moreover, the offline results obtained through validation sets are also analyzed and discussed regarding their efficiency and processing speed. The developed controller was able to achieve a millimeter accuracy in the final position considering a target object seen for the first time. To the best of our knowledge, we have not found in the literature other works that achieve such precision with a controller learned from scratch. Thus, this work presents a new system for autonomous robotic manipulation with high processing speed and the ability to generalize to several different objects.

Real-time deep learning–based image processing for pose estimation and object localization in autonomous robot applications

Object Detection and Pose Estimation from RGB and Depth Data for Real-time, Adaptive Robotic Grasping

Autonomous Robotic Manipulation: Real-Time, Deep-Learning Approach for Grasping of Unknown Objects

Vision-Based Robotic Object Grasping—A Deep Reinforcement Learning Approach

6D Pose Estimation with Combined Deep Learning and 3D Vision Techniques for a Fast and Accurate Object Grasping

Robotic Grasping With Multi-View Image Acquisition and Model-Based Pose Estimation

Deep learning-based method for vision-guided robotic grasping of unknown objects

An Attention-Based Deep Learning Approach for Inertial Motion Recognition and Estimation in Human-Robot Collaboration

Real-Time Deep Learning Approach to Visual Servo Control and Grasp Detection for Autonomous Robotic Manipulation

LocaliseBot: Multi-view 3D object localisation with differentiable rendering for robot grasping

Pose Estimation of a Cobot Implemented on a Small AI-Powered Computing System and a Stereo Camera for Precision Evaluation

A Manufacturing-Oriented Intelligent Vision System Based on Deep Neural Network for Object Recognition and 6D Pose Estimation

A Novel Deep Learning-Based Pose Estimation Method for Robotic Grasping of Axisymmetric Bodies in Industrial Stacked Scenarios

Real Time Application for Automatic Object and 3D Position Detection and Sorting with Robotic Manipulator

Deep Learning-Based Object Classification and Position Estimation Pipeline for Potential Use in Robotized Pick-and-Place Operations

A Practical Robotic Grasping Method by Using 6-D Pose Estimation With Protective Correction

Effective Deployment of CNNs for 3DoF Pose Estimation and Grasping in Industrial Settings

AI-Based Multi-Object Relative State Estimation with Self-Calibration Capabilities

Grasping Pose Estimation for Robots Based on Convolutional Neural Networks

Learning Pose Estimation for High-Precision Robotic Assembly Using Simulated Depth Images

Deep Learning-Based Ensemble Approach for Autonomous Object Manipulation with an Anthropomorphic Soft Robot Hand