Abstract:The camera provides rich visual information regarding objects and becomes one of the most mainstream sensors in the automation community. However, it is often difficult to be applicable when the objects are not visually distinguished. On the other hand, tactile sensors can be used to capture multiple object properties, such as textures, roughness, spatial features, compliance, and friction, and therefore provide another important modality for the perception. Nevertheless, effective combination of the visual and tactile modalities is still a challenging problem. In this paper, we develop a visual–tactile fusion framework for object recognition tasks. This paper uses the multivariate-time-series model to represent the tactile sequence and the covariance descriptor to characterize the image. Further, we design a joint group kernel sparse coding (JGKSC) method to tackle the intrinsically weak pairing problem in visual–tactile data samples. Finally, we develop a visual–tactile data set, composed of 18 household objects for validation. The experimental results show that considering both visual and tactile inputs is beneficial and the proposed method indeed provides an effective strategy for fusion. Note to Practitioners—Visual and tactile measurements offer complementary properties that make them particularly suitable for fusion in order to address the robust and accurate recognition of objects, which is a necessity for many automation systems. In this paper, we investigate a widely applicable scenario in grasp manipulation. When identifying an object, the manipulator may see it using the camera and touch it using its hand. Thus, we obtain a pair of test samples, including one image sample and one tactile sample. The manipulator then utilizes this sample pair to identify this object with a classifier that is constructed using the previously collected training samples. However, when collecting training samples, we may collect the image samples and the tactile samples separately. In other words, the training samples may not be paired, while the test samples are paired. This paper addresses this practical problem by developing a JGKSC method, which encourages the effects of the same group, but different atoms. Although our focus is on combining visual and tactile information, the described problem framework is common in the automation community. The algorithm described in this paper can therefore work with weak pairings between a variety of sensors.

Object shape recognition approach for sparse point clouds from tactile exploration

A New Approach of Point Cloud Processing and Scene Segmentation for Guiding the Visually Impaired

Model Recovery Of Unknown Objects From Discrete Tactile Points

3D Surface Segmentation from Point Clouds Via Quadric Fits Based on DBSCAN Clustering

Surface Extraction and Boundary Detection Based on DBSCAN Clustering in 3D Point Clouds

Haptic Perception Of Unknown Object By Robot Hand: Exploration Strategy And Recognition Approach

3D Object Recognition and Pose Estimation From Point Cloud Using Stably Observed Point Pair Feature

Efficient Center Voting for Object Detection and 6D Pose Estimation in 3D Point Cloud

Adaptive Edge-to-Edge Interaction Learning for Point Cloud Analysis

Generic Primitive Detection in Point Clouds Using Novel Minimal Quadric Fits

A Coarse-to-Fine Algorithm for Matching and Registration in 3D Cross-Source Point Clouds

A Fast Point Cloud Recognition Algorithm Based on Keypoint Pair Feature

PointGT: A Method for Point-Cloud Classification and Segmentation Based on Local Geometric Transformation

Robust And Accurate Feature Detection on Point Clouds

HCCG: Efficient high compatibility correspondence grouping for 3D object recognition and 6D pose estimation in cluttered scenes

SSN: Shape Signature Networks for Multi-class Object Detection from Point Clouds

Visual–Tactile Fusion for Object Recognition

Class-aware tiny object recognition over large-scale 3D point clouds

Visual–Tactile Fusion Object Recognition Using Joint Sparse Coding

PCPNET: Learning Local Shape Properties from Raw Point Clouds

Shape Features for Projected 2D Images Based 3D Object Recognition