Intelligent Multi-Modal Sensing-Communication Integration: Synesthesia of Machines

Xiang Cheng,Haotian Zhang,Jianan Zhang,Shijian Gao,Sijiang Li,Ziwei Huang,Lu Bai,Zonghui Yang,Xinhu Zheng,Liuqing Yang

2023-11-20

Abstract:In the era of sixth-generation (6G) wireless communications, integrated sensing and communications (ISAC) is recognized as a promising solution to upgrade the physical system by endowing wireless communications with sensing capability. Existing ISAC is mainly oriented to static scenarios with radio-frequency (RF) sensors being the primary participants, thus lacking a comprehensive environment feature characterization and facing a severe performance bottleneck in dynamic environments. To date, extensive surveys on ISAC have been conducted but are limited to summarizing RF-based radar sensing. Currently, some research efforts have been devoted to exploring multi-modal sensing-communication integration but still lack a comprehensive review. Therefore, we generalize the concept of ISAC inspired by human synesthesia to establish a unified framework of intelligent multi-modal sensing-communication integration and provide a comprehensive review under such a framework in this paper. The so-termed Synesthesia of Machines (SoM) gives the clearest cognition of such intelligent integration and details its paradigm for the first time. We commence by justifying the necessity of the new paradigm. Subsequently, we offer a definition of SoM and zoom into the detailed paradigm, which is summarized as three operation modes. To facilitate SoM research, we overview the prerequisite of SoM research, i.e., mixed multi-modal (MMM) datasets. Then, we introduce the mapping relationships between multi-modal sensing and communications. Afterward, we cover the technological review on SoM-enhance-based and SoM-concert-based applications. To corroborate the superiority of SoM, we also present simulation results related to dual-function waveform and predictive beamforming design. Finally, we propose some potential directions to inspire future research efforts.

Signal Processing

What problem does this paper attempt to address?

The problem that this paper attempts to solve is that, in the context of the development of the sixth - generation (6G) wireless communication network, the existing Integrated Sensing and Communication (ISAC) technology is mainly limited to static scenarios, and with radio - frequency (RF) sensors as the main participants, lacking a comprehensive characterization of environmental features and facing severe performance bottlenecks in dynamic environments. The paper proposes a new paradigm - Synesthesia of Machines (SoM), aiming to overcome these limitations through the intelligent integration of multi - modal sensing and communication. The concept of SoM is inspired by the human synesthesia phenomenon, emphasizing the role of multi - modal sensing, expanding to a wider range of targets, considering new design methods, and promoting more challenging application scenarios.

Specifically, the main contributions of the paper include:

1. **Proposing the SoM concept**: For the first time, an attempt is made to establish a system framework for the integration of intelligent multi - modal sensing and communication. SoM is developed from the widely - studied RF ISAC, emphasizing the importance of multi - modal sensing, expanding to more diverse targets, considering new design methods, and promoting more challenging scenarios.

2. **Summarizing the SoM paradigm**: For the first time, three operation modes of SoM are proposed: SoM - evoke, SoM - enhance, and SoM - concert, which show how multi - modal sensing and communication can enhance each other.

3. **Discussing SoM in detail**: Reviewed the prerequisites for SoM research, namely the hybrid multi - modal (MMM) dataset, which provides "dataset guidelines" for researchers. In addition, the current research status and challenges in communication and sensing systems for applications based on SoM - enhance and SoM - concert are also outlined.

4. **Showing simulation results**: Demonstrated the beneficial effects of SoM, including exploring the mapping relationship between multi - modal sensing and communication, dual - function waveform design, and simulation results of prediction beamforming design based on SoM - enhance.

5. **Proposing open questions and research directions**: Listed open questions and research directions from two perspectives, aiming to comprehensively stimulate SoM research. These directions include further consolidating its research foundation (such as constructing MMM datasets and exploring mapping relationships) and exploring its future applications in communication and sensing systems (i.e., the future applications of the three operation modes).

In summary, this paper aims to promote the research on the integration of multi - modal sensing and communication by proposing the SoM concept and its detailed implementation framework, in order to meet the challenges brought by dynamic application scenarios in future 6G networks.