Abstract:Humans are remarkably proficient at controlling their limbs and tools from a wide range of viewpoints and angles, even in the presence of optical distortions. In robotics, this ability is referred to as visual servoing: moving a tool or end-point to a desired location using primarily visual feedback. In this paper, we study how viewpoint-invariant visual servoing skills can be learned automatically in a robotic manipulation scenario. To this end, we train a deep recurrent controller that can automatically determine which actions move the end-point of a robotic arm to a desired object. The problem that must be solved by this controller is fundamentally ambiguous: under severe variation in viewpoint, it may be impossible to determine the actions in a single feedforward operation. Instead, our visual servoing system must use its memory of past movements to understand how the actions affect the robot motion from the current viewpoint, correcting mistakes and gradually moving closer to the target. This ability is in stark contrast to most visual servoing methods, which either assume known dynamics or require a calibration phase. We show how we can learn this recurrent controller using simulated data and a reinforcement learning objective. We then describe how the resulting model can be transferred to a real-world robot by disentangling perception from control and only adapting the visual layers. The adapted model can servo to previously unseen objects from novel viewpoints on a real-world Kuka IIWA robotic arm. For supplementary videos, see: https://fsadeghi.github.io/Sim2RealViewInvariantServo

Asymptotic Stability of Uncalibrated Eye-in-Hand Visual Servoing: an Affine Invariance Perspective

Visual Servoing with an Uncalibrated Eye-in-hand Camera

Simultaneous Tracking and Regulation Visual Servoing of Wheeled Mobile Robots with Uncalibrated Extrinsic Parameters

Adapting for Calibration Disturbances: A Neural Uncalibrated Visual Servoing Policy

Unconstrained Self-Calibration of Stereo Camera on Visually Impaired Assistance Devices.

AN AUTONOMOUS HAND-EYE CALIBRATION METHOD

Visual servo of uncalibrated eye-in-hand system with time-delay compensation

An Exponential Class Of Visual Servoing Controllers In The Presence Of Uncertain Camera Calibration

A Exponential Class Of Model-Free Visual Servoing Controllers In The Presence Of Uncertain Camera Calibration

Adaptive Event-Triggered Control For Uncalibrated Visual Servoing of Robot Manipulators

Fully Uncalibrated Image-Based Visual Servoing of 2DOFs Planar Manipulators With a Fixed Camera

Analysis on an uncalibrated image-based visual servoing for 6 DOF industrial welding robots

Uncalibrated Eye-in-hand Visual Servoing Using Recursive Least Squares

Image-Based Composite Learning Robot Visual Servoing With an Uncalibrated Eye-to-Hand Camera

A Study on Robust Finite-Time Visual Servoing with a Gyro-Stabilized Surveillance System

Sim2Real View Invariant Visual Servoing by Recurrent Control

Visual Servoing of Flexible Manipulators With Unknown Camera Intrinsic Parameters and Vibration States

Robust Uncalibrated Visual Servoing Control Based on Disturbance Observer.

Uncalibrated Eye-in-hand Visual Servoing Using Adaptive Recursive Least Squares

Adaptive Robust Motion Control Of Uncertain Manipulators Through Immersion And Invariance Adaptive Visual Servoing

Unified Tracking and Regulation Visual Servoing of Wheeled Mobile Robots with Euclidean Reconstruction.