Abstract:Robotic vision plays a major role in factory automation to service robot applications. However, the traditional use of frame-based camera sets a limitation on continuous visual feedback due to their low sampling rate and redundant data in real-time image processing, especially in the case of high-speed tasks. Event cameras give human-like vision capabilities such as observing the dynamic changes asynchronously at a high temporal resolution ($1\mu s$) with low latency and wide dynamic range. In this paper, we present a visual servoing method using an event camera and a switching control strategy to explore, reach and grasp to achieve a manipulation task. We devise three surface layers of active events to directly process stream of events from relative motion. A purely event based approach is adopted to extract corner features, localize them robustly using heat maps and generate virtual features for tracking and alignment. Based on the visual feedback, the motion of the robot is controlled to make the temporal upcoming event features converge to the desired event in spatio-temporal space. The controller switches its strategy based on the sequence of operation to establish a stable grasp. The event based visual servoing (EVBS) method is validated experimentally using a commercial robot manipulator in an eye-in-hand configuration. Experiments prove the effectiveness of the EBVS method to track and grasp objects of different shapes without the need for re-tuning.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the real - time image processing delay problem in robot visual servo control caused by the low sampling rate and redundant data processing of traditional frame - based cameras, especially in high - speed tasks. To solve these problems, the paper proposes a visual servo method based on event cameras (event camera) to achieve more efficient and low - latency object grasping and manipulation. ### Specific problem description 1. **Limitations of traditional cameras**: - Traditional frame - based cameras have problems of low sampling rate and redundant data when processing continuous visual feedback, resulting in delays in real - time image processing. - These delays are particularly evident in high - dynamic scenes or high - speed tasks, affecting the real - time performance and accuracy of robots. 2. **Advantages of event cameras**: - Event cameras can asynchronously capture dynamic changes with high - time resolution (microsecond - level), having low latency and a wide dynamic range. - Event cameras reduce redundant data and improve processing efficiency by only recording the timestamps of illumination changes. 3. **Target problem**: - How to utilize the characteristics of event cameras to achieve efficient visual servo control, enabling robots to quickly and accurately track and grasp objects of different shapes without the need to readjust parameters. ### Solutions proposed in the paper The paper proposes a visual servo method based on event cameras (Event - Based Visual Servoing, EVBS), which specifically includes the following aspects: 1. **Three - layer active event surfaces**: - **Surface of Active Events (SAE)**: Records the latest event timestamp for each pixel. - **Surface of Active Corner Events (SACE)**: Extracts and locates corner events. - **Surface of Active Virtual Events (SAVE)**: Generates virtual features for tracking and alignment. 2. **Feature detection and tracking**: - Uses the Harris corner detection algorithm adapted to event cameras (e - Harris) to extract corner features. - Clusters corner events through heat - maps to calculate the object center as a high - level feature. - Utilizes the moving average method to quickly track corners, ensuring real - time performance in high - speed applications. 3. **Switching strategy**: - Dynamically adjusts the control strategy according to the operation stages (exploration, approach, alignment) to ensure that the robot can stably complete the grasping task. 4. **Experimental verification**: - Experiments were carried out on the UR10 robotic arm to verify the effectiveness and robustness of this method on objects of different shapes. Through these methods, the paper successfully solves the problems of delay and redundant data in traditional visual servo, achieving efficient and low - latency object grasping and manipulation.

Neuromorphic Eye-in-Hand Visual Servoing

Visual Servoing with an Uncalibrated Eye-in-hand Camera

Real-Time Grasping Strategies Using Event Camera

Visual Servo Kinematic Control For Robotic Manipulators

Hand-eye Robotic Visual Servoing Around Moving Objects Using Active Deformable Models

Hybrid Multi-camera Visual Servoing to Moving Target

Kinematics-based Incremental Visual Servo for Robotic Capture of Non-Cooperative Target.

Robotic Visual Servoing Around a Static Target: An Example of Controlled Active Vision

Predictive visual servo kinematic control for autonomous robotic capture of non-cooperative space target

Real-Time Deep Learning Approach to Visual Servo Control and Grasp Detection for Autonomous Robotic Manipulation

Adaptive Visual Servoing Using Point and Line Features With an Uncalibrated Eye-in-Hand Camera

A Motion Planning Method for Visual Servoing Using Deep Reinforcement Learning in Autonomous Robotic Assembly

Incremental Visual Servo Control of Robotic Manipulator for Autonomous Capture of Non-Cooperative Target

Hand-Eye Servo And Flexible Control Of An Anthropomorphic Arm

Adaptive Event-Triggered Control For Uncalibrated Visual Servoing of Robot Manipulators

A Recurrent Neural Network Approach for Visual Servoing of Manipulators

A Neural Controller for Image-Based Visual Servoing of Manipulators with Physical Constraints

Visual Servoing for Pose Control of Soft Continuum Arm in a Structured Environment

Visual Servoing with Deep Learning and Data Augmentation for Robotic Manipulation

A Nonparametric-Learning Visual Servoing Framework for Robot Manipulator in Unstructured Environments

Visual Servoing in Robotic Manufacturing Systems for Accurate Positioning