Abstract:In the era of sixth-generation (6G) wireless communications, integrated sensing and communications (ISAC) is recognized as a promising solution to upgrading the physical system by endowing wireless communications with sensing capability. Existing ISAC is mainly oriented to static scenarios with radio-frequency sensors being the primary participants, thus lacking a comprehensive environment feature characterization and facing a severe performance bottleneck in dynamic environments. In light of this, we generalize the concept of ISAC by mimicking human synesthesia to support intelligent multi-modal sensing-communication integration. The so-termed Synesthesia of Machines (SoM) is not only oriented to generic scenarios, but also particularly suitable for solving challenges arising from dynamic scenarios. We commence by justifying the necessity and potentials of SoM. Subsequently, we offer the definition of SoM and zoom into the specific operating modes, followed by discussions of the state-of-the-art, corresponding objectives, and challenges. To facilitate SoM research, we overview the prerequisite of SoM research, that is, mixed multi-modal (MMM) datasets, and introduce our work. Built upon the MMM datasets, we introduce the mapping relationships between multi-modal sensing and communications, and discuss how channel modeling can be customized to support the exploration of such relationships. Afterwards, we delve into the current research state and implementing challenges of SoM-enhance-based and SoM-concert-based applications. We first overview the SoM-enhance-based communication system designs and present simulation results related to dual-function waveform and predictive beamforming design. Afterwards, we review the recent advances of SoM-concert for single-agent and multi-agent environment sensing. Finally, we propose some open issues and potential directions.

Intelligent Multi-Modal Sensing-Communication Integration: Synesthesia of Machines

Intelligent Multi-Modal Sensing-Communication Integration: Synesthesia of Machines

Multi-Modal Intelligent Channel Modeling: A New Modeling Paradigm via Synesthesia of Machines

Synesthesia of Machines (SoM)-Enhanced ISAC Precoding for Vehicular Networks with Double Dynamics

Multi-View Sensing for Wireless Communications: Architectures, Designs, and Opportunities

Integrated Sensing and Communications: Recent Advances and Ten Open Challenges

Assisting Living by Wireless Sensing: The Role of Integrated Sensing and Communications in 6G Era

Experimental Study on ISAC Performance with Different Sensing Sequences

Survey of Research on Multimodal Semantic Communication

Multimodal In-Sensor Computing Implemented by Easily-Fabricated Oxide-Heterojunction Optoelectronic Synapses

Mosic: Multimodal Semantic Integrated Communication for Health Monitoring in Iot Scenarios

Joint Multi-User Communication and Sensing Exploiting Both Signal and Environment Sparsity

Integrated Human Activity Sensing and Communications

Symbiotic Sensing and Communications Towards 6G: Vision, Applications, and Technology Trends

Cross-Modal Semantic Communications

Integrated Sensing and Communication with Multi-Domain Cooperation

6G Integrated Sensing and Communications Channel Modeling: Challenges and Opportunities

Integrated Sensing and Communication for Future Wireless Networks

Task-Oriented Mulsemedia Communication using Unified Perceiver and Conformal Prediction in 6G Metaverse Systems

Multi-Point Integrated Sensing and Communication: Fusion Model and Functionality Selection

Scatterer Recognition for Multi-Modal Intelligent Vehicular Channel Modeling via Synesthesia of Machines