Abstract:Purpose: Automated remote assessment and monitoring of patients' neurological and mental health is increasingly becoming an essential component of the digital clinic and telehealth ecosystem, especially after the COVID-19 pandemic. This review article reviews various modalities of health information that are useful for developing such remote clinical assessments in the real world at scale. Approach: We first present an overview of the various modalities of health information-speech acoustics, natural language, conversational dynamics, orofacial or full body movement, eye gaze, respiration, cardiopulmonary, and neural-which can each be extracted from various signal sources-audio, video, text, or sensors. We further motivate their clinical utility with examples of how information from each modality can help us characterize how different disorders affect different aspects of patients' spoken communication. We then elucidate the advantages of combining one or more of these modalities toward a more holistic, informative, and robust assessment. Findings: We find that combining multiple modalities of health information allows for improved scientific interpretability, improved performance on downstream health applications such as early detection and progress monitoring, improved technological robustness, and improved user experience. We illustrate how these principles can be leveraged for remote clinical assessment at scale using a real-world case study of the Modality assessment platform. Conclusion: This review article motivates the combination of human-centric information from multiple modalities to measure various aspects of patients' health, arguing that remote clinical assessment that integrates this complementary information can be more effective and lead to better clinical outcomes than using any one data stream in isolation.

The Potential of Multimodal Technology in Modern Medicine

Multimodal biomedical AI

Multimodal Machine Learning in Image-Based and Clinical Biomedicine: Survey and Prospects

As artificial intelligence goes multimodal, medical applications multiply.

The future of multimodal artificial intelligence models for integrating imaging and clinical metadata: a narrative review

The expanding horizon of neurotechnology: Is multimodal neuromodulation the future?

Multimodal Artificial Intelligence in Medicine

Multimodal Technologies for Remote Assessment of Neurological and Mental Health

Multimodal Machine Learning in Precision Health

Navigating the landscape of multimodal AI in medicine: a scoping review on technical challenges and clinical applications

Multimodal neuroimaging computing: a review of the applications in neuropsychiatric disorders

A Comprehensive Review on Synergy of Multi-Modal Data and AI Technologies in Medical Diagnosis

Use of Artificial Intelligence Including Multimodal Systems to Improve the Management of Cardiovascular Disease

Has Multimodal Learning Delivered Universal Intelligence in Healthcare? A Comprehensive Survey

Multimodal Machine Learning in Mental Health: A Survey of Data, Algorithms, and Challenges

Integrated multimodal artificial intelligence framework for healthcare applications

Multimodal Healthcare AI: Identifying and Designing Clinically Relevant Vision-Language Applications for Radiology

Artificial Intelligence in Medicine: Today and Tomorrow

Review of multimodal machine learning approaches in healthcare

Multimodal Cardiac Imaging Revisited by Artificial Intelligence: An Innovative Way of Assessment or Just an Aid?