Abstract:Soft robotic grippers are increasingly desired in applications that involve grasping of complex and deformable objects. However, their flexible nature and non-linear dynamics makes the modelling and control difficult. Numerical techniques such as Finite Element Analysis (FEA) present an accurate way of modelling complex deformations. However, FEA approaches are computationally expensive and consequently challenging to employ for real-time control tasks. Existing analytical techniques simplify the modelling by approximating the deformed gripper geometry. Although this approach is less computationally demanding, it is limited in design scope and can lead to larger estimation errors. In this paper, we present a learning based framework that is able to predict contact forces as well as stress distribution from soft Fin Ray Effect (FRE) finger images in real-time. These images are used to learn internal representations for deformations using a deep neural encoder, which are further decoded to contact forces and stress maps using separate branches. The entire network is jointly learned in an end-to-end fashion. In order to address the challenge of having sufficient labelled data for training, we employ FEA to generate simulated images to supervise our framework. This leads to an accurate prediction, faster inference and availability of large and diverse data for better generalisability. Furthermore, our approach is able to predict a detailed stress distribution that can guide grasp planning, which would be particularly useful for delicate objects. Our proposed approach is validated by comparing the predicted contact forces to the computed ground-truth forces from FEA as well as real force sensor. We rigorously evaluate the performance of our approach under variations in contact point, object material, object shape, viewing angle, and level of occlusion.

ForceSight: Text-Guided Mobile Manipulation with Visual-Force Goals

MOMA-Force: Visual-Force Imitation for Real-World Mobile Manipulation

ImageManip: Image-based Robotic Manipulation with Affordance-guided Next View Selection

Force/Torque Sensing for Soft Grippers using an External Camera

A Deep Learning Method for Vision Based Force Prediction of a Soft Fin Ray Gripper Using Simulation Data

Toward Force Estimation in Robot-Assisted Surgery using Deep Learning with Vision and Robot State

ForceMimic: Force-Centric Imitation Learning with Force-Motion Capture System for Contact-Rich Manipulation

RGBManip: Monocular Image-based Robotic Manipulation through Active Object Pose Estimation

FeelAnyForce: Estimating Contact Force Feedback from Tactile Sensation for Vision-Based Tactile Sensors

Neural feels with neural fields: Visuo-tactile perception for in-hand manipulation

TransForce: Transferable Force Prediction for Vision-based Tactile Sensors with Sequential Image Translation

EyeSight Hand: Design of a Fully-Actuated Dexterous Robot Hand with Integrated Vision-Based Tactile Sensors and Compliant Actuation

Robust Learning of Tactile Force Estimation through Robot Interaction

Manipulate by Seeing: Creating Manipulation Controllers from Pre-Trained Representations

Multifingered Robot Hand Compliant Manipulation Based on Vision-Based Demonstration and Adaptive Force Control

Just Add Force for Contact-Rich Robot Policies

NeuralFeels with neural fields: Visuotactile perception for in-hand manipulation.

Visual Contact Pressure Estimation for Grippers in the Wild

AugInsert: Learning Robust Visual-Force Policies via Data Augmentation for Object Assembly Tasks

Q-Attention: Enabling Efficient Learning for Vision-Based Robotic Manipulation

Feeling the Force: Integrating Force and Pose for Fluent Discovery Through Imitation Learning to Open Medicine Bottles.