Bio‐Plausible Multimodal Learning with Emerging Neuromorphic Devices

Haonan Sun,Haoxiang Tian,Yihao Hu,Yi Cui,Xinrui Chen,Minyi Xu,Xianfu Wang,Tao Zhou

DOI: https://doi.org/10.1002/advs.202406242

2024-01-01

Abstract:AbstractMultimodal machine learning, as a prospective advancement in artificial intelligence, endeavors to emulate the brain's multimodal learning abilities with the objective to enhance interactions with humans. However, this approach requires simultaneous processing of diverse types of data, leading to increased model complexity, longer training times, and higher energy consumption. Multimodal neuromorphic devices have the capability to preprocess spatio‐temporal information from various physical signals into unified electrical signals with high information density, thereby enabling more biologically plausible multimodal learning with low complexity and high energy‐efficiency. Here, this work conducts a comparison between the expression of multimodal machine learning and multimodal neuromorphic computing, followed by an overview of the key characteristics associated with multimodal neuromorphic devices. The bio‐plausible operational principles and the multimodal learning abilities of emerging devices are examined, which are classified into heterogeneous and homogeneous multimodal neuromorphic devices. Subsequently, this work provides a detailed description of the multimodal learning capabilities demonstrated by neuromorphic circuits and their respective applications. Finally, this work highlights the limitations and challenges of multimodal neuromorphic computing in order to hopefully provide insight into potential future research directions.