Abstract:Lifelong machine learning can learn a sequence of consecutive robotic perception tasks via transferring previous experiences. However, 1) most existing lifelong learning based perception methods only take advantage of visual information for robotic tasks, while neglecting another important tactile sensing modality to capture discriminative material properties; 2) Meanwhile, they cannot explore the intrinsic relationships across different modalities and the common characterization among different tasks of each modality, due to the distinct divergence between heterogeneous feature distributions. To address above challenges, we propose a new Lifelong Visual-Tactile Learning (LVTL) model for continuous robotic visual-tactile perception tasks, which fully explores the latent correlations in both intra-modality and cross-modality aspects. Specifically, a modality-specific knowledge library is developed for each modality to explore common intra-modality representations across different tasks, while narrowing intra-modality mapping divergence between semantic and feature spaces via an auto-encoder mechanism. Moreover, a sparse constraint based modality-invariant space is constructed to capture underlying cross-modality correlations and identify the contributions of each modality for new coming visual-tactile tasks. We further propose a modality consistence regularizer to efficiently align the heterogeneous visual and tactile samples, which ensures the semantic consistence between different modality-specific knowledge libraries. After deriving an efficient model optimization strategy, we conduct extensive experiments on several representative datasets to validate the superiority of our LVTL model. Comparison experiments show that our proposed model significantly outperforms existing state-of-the-art methods with about 1.29%<math>∼</math>15.36% improvement under different lifelong visual-tactile perception scenarios.

Lifelong Learning For Heterogeneous Multi-Modal Tasks

Multimodal Continual Learning Using Online Dictionary Updating.

Lifelong Metric Learning

Bi-level Dynamic Learning for Jointly Multi-modality Image Fusion and Beyond

3FM: Multi-modal Meta-learning for Federated Tasks

Learning transferable cross-modality representations for few-shot hyperspectral and LiDAR collaborative classification

Online Multi-modal Imitation Learning Via Lifelong Intention Encoding.

Lifelong robotic visual-tactile perception learning

Adaptive online continual multi-view learning

Learning Modality-Specific and -Agnostic Representations for Asynchronous Multimodal Language Sequences

Supervised Multi-Modal Fission Learning

A cross modal hierarchical fusion multimodal sentiment analysis method based on multi-task learning

A Simple Lifelong Learning Approach

Multi-modal broad learning for material recognition

A Conceptual Framework for Lifelong Learning

Cross-lingual Lifelong Learning

DMTMV: A Unified Learning Framework for Deep Multi-task Multi-view Learning.

Heterogeneous feature learning network for multimodal remote sensing image collaborative classification

Visual–tactile fusion material identification using dictionary learning

Learning and Fusing Multimodal Features from and for Multi-task Facial Computing

Comprehensive Semi-Supervised Multi-Modal Learning.