Abstract:Vision is the main component of current robotics systems that is used for manipulating objects. However, solely relying on vision for hand-object pose tracking faces challenges such as occlusions and objects moving out of view during robotic manipulation. In this work, we show that object kinematics can be inferred from local haptic feedback at the robot-object contact points, combined with robot kinematics information given an initial vision estimate of the object pose. A planar, dual-arm, teleoperated robotic setup was built to manipulate an object with hands shaped like circular discs. The robot hands were built with rubber cladding to allow for rolling contact without slipping. During stable grasping by the dual arm robot, under quasi-static conditions, the surface of the robot hand and object at the contact interface is defined by local geometric constraints. This allows one to define a relation between object orientation and robot hand orientation. With rolling contact, the displacement of the contact point on the object surface and the hand surface must be equal and opposite. This information, coupled with robot kinematics, allows one to compute the displacement of the object from its initial location. The mathematical formulation of the geometric constraints between robot hand and object is detailed. This is followed by the methodology in acquiring data from experiments to compute object kinematics. The sensors used in the experiments, along with calibration procedures, are presented before computing the object kinematics from recorded haptic feedback. Results comparing object kinematics obtained purely from vision and from haptics are presented to validate our method, along with the future ideas for perception via haptic manipulation.

Low-Cost Teleoperation with Haptic Feedback through Vision-based Tactile Sensors for Rigid and Soft Object Manipulation

Beyond End-Effector: Utilizing High-Resolution Tactile Signals for Physical Human–Robot Interaction

Enabling Robot Manipulation of Soft and Rigid Objects with Vision-based Tactile Sensors

Integrating and Evaluating Visuo-tactile Sensing with Haptic Feedback for Teleoperated Robot Manipulation

Haptic and Visual Feedback Assistance for Dual-Arm Robot Teleoperation in Surface Conditioning Tasks

Towards Haptic-Based Dual-Arm Manipulation

Haptic Stiffness Perception Using Hand Exoskeletons in Tactile Robotic Telemanipulation

Virtual Reality Based Tactile Sensing Enhancements for Bilateral Teleoperation System with In-Hand Manipulation

Teleoperation Control of an Underactuated Bionic Hand: Comparison between Wearable and Vision-Tracking-Based Methods

Haptic-ACT: Bridging Human Intuition with Compliant Robotic Manipulation via Immersive VR

Haptic Augmentation for Teleoperation through Virtual Grasping Points

An Agile Large-Workspace Teleoperation Interface Based on Human Arm Motion and Force Estimation

Robot teleoperation using a vision-based manipulation method

Safe haptic teleoperations of admittance controlled robots with virtualization of the force feedback

Tactile-Driven Gentle Grasping for Human-Robot Collaborative Tasks

Combining haptics and inertial motion capture to enhance remote control of a dual-arm robot

Space Robot Teleoperation Based on Active Vision

Fine Manipulation and Dynamic Interaction in Haptic Teleoperation

Bimanual Telemanipulation with Force and Haptic Feedback and Predictive Limit Avoidance

Haptic Teleoperation of Robotic Manipulator

Shared Telemanipulation with VR controllers in an anti slosh scenario