Abstract:In order to explore robotic grasping in unstructured and dynamic environments, this work addresses the visual perception phase involved in the task. This phase involves the processing of visual data to obtain the location of the object to be grasped, its pose and the points at which the robot`s grippers must make contact to ensure a stable grasp. For this, the Cornell Grasping dataset is used to train a convolutional neural network that, having an image of the robot`s workspace, with a certain object, is able to predict a grasp rectangle that symbolizes the position, orientation and opening of the robot`s grippers before its closing. In addition to this network, which runs in real-time, another one is designed to deal with situations in which the object moves in the environment. Therefore, the second network is trained to perform a visual servo control, ensuring that the object remains in the robot`s field of view. This network predicts the proportional values of the linear and angular velocities that the camera must have so that the object is always in the image processed by the grasp network. The dataset used for training was automatically generated by a Kinova Gen3 manipulator. The robot is also used to evaluate the applicability in real-time and obtain practical results from the designed algorithms. Moreover, the offline results obtained through validation sets are also analyzed and discussed regarding their efficiency and processing speed. The developed controller was able to achieve a millimeter accuracy in the final position considering a target object seen for the first time. To the best of our knowledge, we have not found in the literature other works that achieve such precision with a controller learned from scratch. Thus, this work presents a new system for autonomous robotic manipulation with high processing speed and the ability to generalize to several different objects.

Fast robust peg-in-hole insertion with continuous visual servoing

Hybrid Visual Servoing for Rivet-in-hole Insertion Based on Super-twisting Sliding Mode Control

Quickly Inserting Pegs into Uncertain Holes using Multi-view Images and Deep Network Trained on Synthetic Data

Learning A Simulation-based Visual Policy for Real-world Peg in Unseen Holes.

Learning A Simulation-based Visual Policy for Real-world Peg In Unseen Holes

Sensor-Based Control Using an Image Point and Distance Features for Rivet-in-Hole Insertion.

Multiple Peg-in-Hole Assembly of Tightly Coupled Multi-manipulator Using Learning-based Visual Servo

Learning-based Optimization Algorithms Combining Force Control Strategies for Peg-in-Hole Assembly.

An adaptive ball-head positioning visual servoing method for aircraft digital assembly

Vision-based robotic peg-in-hole research: integrating object recognition, positioning, and reinforcement learning

Image-based visual servoing using a set for multiple pin-in-hole assembly

Non-vector Space Visual Servoing for Multiple Pin-in-hole Assembly by Robot

Vision-Based Robotic Object Grasping—A Deep Reinforcement Learning Approach

A Noncontact Control Strategy for Circular Peg-in-Hole Assembly Guided by the 6-DOF Robot Based on Hybrid Vision

Neuromorphic Eye-in-Hand Visual Servoing

Visual Servoing Based on 3D Features: Design and Implementation for Robotic Insertion Tasks

Visual Servoing with an Uncalibrated Eye-in-hand Camera

Position/Force Visual-Sensing-Based Robotic Sheet-Like Peg-in-Hole Assembly

Peg-in-hole assembly in live-line maintenance based on generative mapping and searching network

Real-Time Deep Learning Approach to Visual Servo Control and Grasp Detection for Autonomous Robotic Manipulation

Deep Siamese Neural Network-Driven Model for Robotic Multiple Peg-in-Hole Assembly System