Abstract:Soft robots offer significant advantages in adaptability, safety, and dexterity compared to conventional rigid-body robots. However, it is challenging to equip soft robots with accurate proprioception and tactile sensing due to their high flexibility and elasticity. In this work, we describe the development of a vision-based proprioceptive and tactile sensor for soft robots called GelFlex, which is inspired by previous GelSight sensing techniques. More specifically, we develop a novel exoskeleton-covered soft finger with embedded cameras and deep learning methods that enable high-resolution proprioceptive sensing and rich tactile sensing. To do so, we design features along the axial direction of the finger, which enable high-resolution proprioceptive sensing, and incorporate a reflective ink coating on the surface of the finger to enable rich tactile sensing. We design a highly underactuated exoskeleton with a tendon-driven mechanism to actuate the finger. Finally, we assemble 2 of the fingers together to form a robotic gripper and successfully perform a bar stock classification task, which requires both shape and tactile information. We train neural networks for proprioception and shape (box versus cylinder) classification using data from the embedded sensors. The proprioception CNN had over 99% accuracy on our testing set (all six joint angles were within 1° of error) and had an average accumulative distance error of 0.77 mm during live testing, which is better than human finger proprioception. These proposed techniques offer soft robots the high-level ability to simultaneously perceive their proprioceptive state and peripheral environment, providing potential solutions for soft robots to solve everyday manipulation tasks. We believe the methods developed in this work can be widely applied to different designs and applications.

Vision-Based Tactile Perception for Soft Robotic Hand

Soft Robotic Finger Embedded with Visual Sensor for Bending Perception

A Dual-Modal Vision-Based Tactile Sensor for Robotic Hand Grasping.

Development of Flexible Tactile Sensor for the Envelop of Curved Robotic Hand Finger in Grasping Force Sensing

Soft Hand Perception Based on Embedded Optical Fiber Bending Sensor

Progress on Flexible Tactile Sensors in Robotic Applications on Objects Properties Recognition, Manipulation and Human-Machine Interactions

Exoskeleton-covered soft finger with vision-based proprioception and tactile sensing

A Novel Humanoid Soft Hand with Variable Stiffness and Multi-modal Perception

Proprioception and Exteroception of a Soft Robotic Finger Using Neuromorphic Vision-Based Sensing

Vision-based Tactile Sensor Using Depth from Defocus for Artificial Finger in Hand Prosthesis

Robot Tactile Sensing: Vision Based Tactile Sensor for Force Perception

Development of a Visual-Tactile Fingertip Sensor and an Object Manipulation Method using a Multi-Fingered Robotic Hand

Dexterous object manipulation by a multi-fingered robotic hand with visual-tactile fingertip sensors

Soft Contact Simulation and Manipulation Learning of Deformable Objects with Vision-based Tactile Sensor

A soft robot tactile fingertip for grasping surface posture detection

STEV: Stretchable Triboelectric E-skin Enabled Proprioceptive Vibration Sensing for Soft Robot

Sensor Embedded Soft Fingertip for Precise Manipulation and Softness Recognition

Soft Tactile Sensing for Object Classification and Fine Grasping Adjustment Using a Pneumatic Hand With an Inflatable Palm

Learning-Based Optoelectronically Innervated Tactile Finger for Rigid-Soft Interactive Grasping

Nondestructive identification of softness via bioinspired multisensory electronic skins integrated on a robotic hand

Design of a Tactile Sensing Robotic Gripper and Its Grasping Method