Abstract:The precise control of soft and continuum robots requires knowledge of their shape. The shape of these robots has, in contrast to classical rigid robots, infinite degrees of freedom. To partially reconstruct the shape, proprioceptive techniques use built-in sensors resulting in inaccurate results and increased fabrication complexity. Exteroceptive methods so far rely on placing reflective markers on all tracked components and triangulating their position using multiple motion-tracking cameras. Tracking systems are expensive and infeasible for deformable robots interacting with the environment due to marker occlusion and damage. Here, we present a regression approach for 3D shape estimation using a convolutional neural network. The proposed approach takes advantage of data-driven supervised learning and is capable of real-time marker-less shape estimation during inference. Two images of a robotic system are taken simultaneously at 25 Hz from two different perspectives, and are fed to the network, which returns for each pair the parameterized shape. The proposed approach outperforms marker-less state-of-the-art methods by a maximum of 4.4% in estimation accuracy while at the same time being more robust and requiring no prior knowledge of the shape. The approach can be easily implemented due to only requiring two color cameras without depth and not needing an explicit calibration of the extrinsic parameters. Evaluations on two types of soft robotic arms and a soft robotic fish demonstrate our method's accuracy and versatility on highly deformable systems in real-time. The robust performance of the approach against different scene modifications (camera alignment and brightness) suggests its generalizability to a wider range of experimental setups, which will benefit downstream tasks such as robotic grasping and manipulation.

Comparison of Graph Fitting and Sparse Deep Learning Model for Robot Pose Estimation

Object tracking with 3D LIDAR via multi-task sparse learning

Recurrent Volume-based 3D Feature Fusion for Real-time Multi-view Object Pose Estimation

Robust And Accurate Multiple-Camera Pose Estimation Toward Robotic Applications

Nothing But Geometric Constraints: A Model-Free Method for Articulated Object Pose Estimation

Sparse Convolution Based 6D Pose Estimation for Robotic Bin-Picking with Point Clouds

ViSE: Vision-Based 3D Online Shape Estimation of Continuously Deformable Robots

Image-similarity-based Convolutional Neural Network for Robot Visual Relocalization

3D Robot Pose Estimation from 2D Images

Efficient Fully Convolutional Network and Optimization Approach for Robotic Grasping Detection Based on RGB-D Images

Real-Time Accurate 3D Head Tracking and Pose Estimation with Consumer RGB-D Cameras

Vision-Based Intelligent Robot Grasping Using Sparse Neural Network

Pose Estimation of a Cobot Implemented on a Small AI-Powered Computing System and a Stereo Camera for Precision Evaluation

A Real-Time Visual Tracking Approach Using Sparse Autoencoder and Extreme Learning Machine

Learning Analysis-by-Synthesis for 6D Pose Estimation in RGB-D Images

Real-Time Robot End-Effector Pose Estimation with Deep Network

Vision-Based Real-Time Shape Estimation of Self-Occluding Soft Parallel Robots Using Neural Networks

SparseGrasp: Robotic Grasping via 3D Semantic Gaussian Splatting from Sparse Multi-View RGB Images

Robust Visual Tracking Using Sparse Discriminative Graph Embedding.

CtRNet-X: Camera-to-Robot Pose Estimation in Real-world Conditions Using a Single Camera

Accurate Robot Arm Attitude Estimation Based on Multi-View Images and Super-Resolution Keypoint Detection Networks