Introduction to the Special Issue on Multimodal Machine Learning for Human Behavior Analysis

Shengping Zhang,Huiyu Zhou,Dong Xu,M. Emre Celebi,Thierry Bouwmans

DOI: https://doi.org/10.1145/3381917

2020-01-01

Abstract:No abstract available.

What problem does this paper attempt to address?

Editorial Note: Machine Learning for Multimodal Data Understanding

Weizhi Nie,Hanwang Zhang,Fei Wu,Zhao-Zheng Yin

DOI: https://doi.org/10.1007/s11042-018-6903-7

IF: 2.577

2019-01-01

Multimedia Tools and Applications

Abstract:Page 1. EDITORIAL NOTE Editorial Note: Machine Learning for Multimodal Data Understanding Published online: 23 November 2018 © Springer Science+Business Media, LLC, part of Springer Nature 2018 Multimedia Tools and Applications gratefully acknowledges the editorial work of the scholars listed below on the special issue entitled, BMachine Learning for Multimodal Data Understanding.^ Of 56 papers submitted to this issue, 24 were eventually accepted after a stringent peer-review process. Weizhi Nie (Corresponding Guest Editor) Tianjin University, China truman.nie@gmail.com Hanwang Zhang National University of Singapore, Singapore hanwangzhang@gmail.com Fei Wu Zhejiang University, China wufei@cs.zju.edu.cn Dr. Zhao-Zheng Yin Missouri University of Science and Technology, USA yinz@mst.edu Multimedia Tools and Applications (2019) 78:413 https://doi.org/10.1007/s11042-018-6903-7 …
A Survey of Multimodal Machine Learning

CHEN Peng,LI Qing,ZHANG De-zheng,YANG Yu-hang,CAI Zheng,LU Zi-yi

DOI: https://doi.org/10.13374/j.issn2095-9389.2019.03.21.003

2020-01-01

Chinese Journal of Engineering

Abstract:“Big data” is always collected from different resources that have different data structures. With the rapid development of information technologies, current precious data resources are characteristic of multimodes. As a result, based on classical machine learning strategies, multi-modal learning has become a valuable research topic, enabling computers to process and understand “big data”. The cognitive processes of humans involve perception through different sense organs. Signals from eyes, ears, the nose, and hands (tactile sense) constitute a person’s understanding of a special scene or the world as a whole. It reasonable to believe that multi-modal methods involving a higher ability to process complex heterogeneous data can further promote the progress of information technologies. The concepts of multimodality stemmed from psychology and pedagogy from hundreds of years ago and have been popular in computer science during the past decade. In contrast to the concept of “media”, a “mode” is a more fine-grained concept that is associated with a typical data source or data form. The effective utilization of multi-modal data can aid a computer understand a specific environment in a more holistic way. In this context, we first introduced the definition and main tasks of multi-modal learning. Based on this information, the mechanism and origin of multi-modal machine learning were then briefly introduced. Subsequently, statistical learning methods and deep learning methods for multi-modal tasks were comprehensively summarized. We also introduced the main styles of data fusion in multi-modal perception tasks, including feature representation, shared mapping, and co-training. Additionally, novel adversarial learning strategies for cross-modal matching or generation were reviewed. The main methods for multi-modal learning were outlined in this paper with a focus on future research issues in this field.
Guest Editorial: Special Issue on Interactive Visual Analysis of Human and Crowd Behaviors

Yu-Ru Lin,Nan Cao

DOI: https://doi.org/10.1145/3178569

IF: 3.4

2018-01-01

ACM Transactions on Interactive Intelligent Systems

Abstract:The analysis of human behaviors has impacted many social and commercial domains. How could interactive visual analytic systems be used to further provide behavioral insights? This editorial introduction features emerging research trend related to this question. The four articles accepted for this special issue represent recent progress: they identify research challenges arising from analysis of human and crowd behaviors, and present novel methods in visual analysis to address those challenges and help make behavioral data more useful.
Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions

Paul Pu Liang,Amir Zadeh,Louis-Philippe Morency

DOI: https://doi.org/10.48550/arXiv.2209.03430

2023-02-20

Abstract:Multimodal machine learning is a vibrant multi-disciplinary research field that aims to design computer agents with intelligent capabilities such as understanding, reasoning, and learning through integrating multiple communicative modalities, including linguistic, acoustic, visual, tactile, and physiological messages. With the recent interest in video understanding, embodied autonomous agents, text-to-image generation, and multisensor fusion in application domains such as healthcare and robotics, multimodal machine learning has brought unique computational and theoretical challenges to the machine learning community given the heterogeneity of data sources and the interconnections often found between modalities. However, the breadth of progress in multimodal research has made it difficult to identify the common themes and open questions in the field. By synthesizing a broad range of application domains and theoretical frameworks from both historical and recent perspectives, this paper is designed to provide an overview of the computational and theoretical foundations of multimodal machine learning. We start by defining three key principles of modality heterogeneity, connections, and interactions that have driven subsequent innovations, and propose a taxonomy of six core technical challenges: representation, alignment, reasoning, generation, transference, and quantification covering historical and recent trends. Recent technical achievements will be presented through the lens of this taxonomy, allowing researchers to understand the similarities and differences across new approaches. We end by motivating several open problems for future research as identified by our taxonomy.

Machine Learning,Artificial Intelligence,Computation and Language,Computer Vision and Pattern Recognition,Multimedia
Foundations & Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions

Paul Pu Liang,Amir Zadeh,Louis-Philippe Morency

DOI: https://doi.org/10.1145/3656580

IF: 16.6

2024-04-09

ACM Computing Surveys

Abstract:Multimodal machine learning is a vibrant multi-disciplinary research field that aims to design computer agents with intelligent capabilities such as understanding, reasoning, and learning through integrating multiple communicative modalities, including linguistic, acoustic, visual, tactile, and physiological messages. With the recent interest in video understanding, embodied autonomous agents, text-to-image generation, and multisensor fusion in application domains such as healthcare and robotics, multimodal machine learning has brought unique computational and theoretical challenges to the machine learning community given the heterogeneity of data sources and the interconnections often found between modalities. However, the breadth of progress in multimodal research has made it difficult to identify the common themes and open questions in the field. By synthesizing a broad range of application domains and theoretical frameworks from both historical and recent perspectives, this paper is designed to provide an overview of the computational and theoretical foundations of multimodal machine learning. We start by defining three key principles of modality heterogeneity , connections , and interactions that have driven subsequent innovations, and propose a taxonomy of six core technical challenges: representation , alignment , reasoning , generation , transference , and quantification covering historical and recent trends. Recent technical achievements will be presented through the lens of this taxonomy, allowing researchers to understand the similarities and differences across new approaches. We end by motivating several open problems for future research as identified by our taxonomy.

computer science, theory & methods
Multimodal Machine Learning in Mental Health: A Survey of Data, Algorithms, and Challenges

Zahraa Al Sahili,Ioannis Patras,Matthew Purver

2024-07-24

Abstract:The application of machine learning (ML) in detecting, diagnosing, and treating mental health disorders is garnering increasing attention. Traditionally, research has focused on single modalities, such as text from clinical notes, audio from speech samples, or video of interaction patterns. Recently, multimodal ML, which combines information from multiple modalities, has demonstrated significant promise in offering novel insights into human behavior patterns and recognizing mental health symptoms and risk factors. Despite its potential, multimodal ML in mental health remains an emerging field, facing several complex challenges before practical applications can be effectively developed. This survey provides a comprehensive overview of the data availability and current state-of-the-art multimodal ML applications for mental health. It discusses key challenges that must be addressed to advance the field. The insights from this survey aim to deepen the understanding of the potential and limitations of multimodal ML in mental health, guiding future research and development in this evolving domain.

Machine Learning,Artificial Intelligence,Computers and Society,Emerging Technologies
Modality Influence in Multimodal Machine Learning

Abdelhamid Haouhat,Slimane Bellaouar,Attia Nehar,Hadda Cherroun

DOI: https://doi.org/10.48550/arXiv.2306.06476

2023-06-10

Computation and Language

Abstract:Multimodal Machine Learning has emerged as a prominent research direction across various applications such as Sentiment Analysis, Emotion Recognition, Machine Translation, Hate Speech Recognition, and Movie Genre Classification. This approach has shown promising results by utilizing modern deep learning architectures. Despite the achievements made, challenges remain in data representation, alignment techniques, reasoning, generation, and quantification within multimodal learning. Additionally, assumptions about the dominant role of textual modality in decision-making have been made. However, limited investigations have been conducted on the influence of different modalities in Multimodal Machine Learning systems. This paper aims to address this gap by studying the impact of each modality on multimodal learning tasks. The research focuses on verifying presumptions and gaining insights into the usage of different modalities. The main contribution of this work is the proposal of a methodology to determine the effect of each modality on several Multimodal Machine Learning models and datasets from various tasks. Specifically, the study examines Multimodal Sentiment Analysis, Multimodal Emotion Recognition, Multimodal Hate Speech Recognition, and Multimodal Disease Detection. The study objectives include training SOTA MultiModal Machine Learning models with masked modalities to evaluate their impact on performance. Furthermore, the research aims to identify the most influential modality or set of modalities for each task and draw conclusions for diverse multimodal classification tasks. By undertaking these investigations, this research contributes to a better understanding of the role of individual modalities in multi-modal learning and provides valuable insights for future advancements in this field.
Human-Machine Learning for Data-Driven Scenarios

Fan Yang,Zhiwen Yu,Jiaqi Liu,Hui Wang,Liang Wang

DOI: https://doi.org/10.2139/ssrn.4591696

2023-01-01

Abstract:Deep learning has been broadly applied in many fields and greatly improved the efficiency and accuracy compared to traditional approaches. It cannot resolve issues well when lack of training samples, or in some varying cases it cannot give a clear output. Human beings and machines work in a collaborative and equal mode to address complicated data-driven tasks can achieve higher accuracy and clearer explanation. A novel human-machine learning for data-driven scenarios framework is proposed which integrates human intelligence and machine intelligence, taking advantage of both strengths to work out the complex task. Human beings are mainly responsible for highly decisive proportion of the task and provide empirical feedback to the model, whereas machines charge for repetitive computing part of the task. The model will be well trained in a shorter time and more robust for some hard samples recognition. Through video anomaly detection, person re-identification, and sound event detection application scenarios, human-machine learning mechanism is conducted and obtained much better accuracy and less time. Fusing human knowledge with deep learning processing, then confirm the final decision-making. In addition, we conducted abundant experiments to verify the effectiveness of the framework and obtained the competitive performance. Human-machine learning is also a new form of machine learning, especially reflects the advantage in dynamic and untrustworthy conditions.
Introduction to the Special Issue on Deep Learning for Multi-Modal Intelligence Across Speech, Language, Vision, and Heterogeneous Signals

Xiaodong He,Li Deng,Richard Rose,Minlie Huang,Isabel Trancoso,Chao Zhang

DOI: https://doi.org/10.1109/jstsp.2020.2989852

IF: 7.695

2020-01-01

IEEE Journal of Selected Topics in Signal Processing

Abstract:The ten papers included in this special section focus on deep learning for multi-modal intelligence across speech, language, vision, and heterogeneous signals. Thanks to the disruptive advances in deep learning, significant progress has been made in artificial intelligence (AI) applications with single modality, such as speech recognition, speech synthesis, image classification, object detection, as well as machine translation and reading comprehension, etc. However, many AI problems require more than one modality, and techniques developed for different modalities can often be successfully cross-fertilized. Therefore, the studies on the modeling and learning approaches across multiple modalities are of great interest. This special issue brings together a diverse but complementary set of contributions on emerging deep learning methods for problems based on multiple modalities including speech, text, image and video.
Vision+X: A Survey on Multimodal Learning in the Light of Data

Ye Zhu,Yu Wu,Nicu Sebe,Yan Yan

2024-06-08

Abstract:We are perceiving and communicating with the world in a multisensory manner, where different information sources are sophisticatedly processed and interpreted by separate parts of the human brain to constitute a complex, yet harmonious and unified sensing system. To endow the machines with true intelligence, multimodal machine learning that incorporates data from various sources has become an increasingly popular research area with emerging technical advances in recent years. In this paper, we present a survey on multimodal machine learning from a novel perspective considering not only the purely technical aspects but also the intrinsic nature of different data modalities. We analyze the commonness and uniqueness of each data format mainly ranging from vision, audio, text, and motions, and then present the methodological advancements categorized by the combination of data modalities, such as Vision+Text, with slightly inclined emphasis on the visual data. We investigate the existing literature on multimodal learning from both the representation learning and downstream application levels, and provide an additional comparison in the light of their technical connections with the data nature, e.g., the semantic consistency between image objects and textual descriptions, and the rhythm correspondence between video dance moves and musical beats. We hope that the exploitation of the alignment as well as the existing gap between the intrinsic nature of data modality and the technical designs, will benefit future research studies to better address a specific challenge related to the concrete multimodal task, prompting a unified multimodal machine learning framework closer to a real human intelligence system.

Computer Vision and Pattern Recognition
Multimodal Machine Learning: A Survey and Taxonomy

Tadas Baltrušaitis,Chaitanya Ahuja,Louis-Philippe Morency

2017-08-02

Abstract:Our experience of the world is multimodal - we see objects, hear sounds, feel texture, smell odors, and taste flavors. Modality refers to the way in which something happens or is experienced and a research problem is characterized as multimodal when it includes multiple such modalities. In order for Artificial Intelligence to make progress in understanding the world around us, it needs to be able to interpret such multimodal signals together. Multimodal machine learning aims to build models that can process and relate information from multiple modalities. It is a vibrant multi-disciplinary field of increasing importance and with extraordinary potential. Instead of focusing on specific multimodal applications, this paper surveys the recent advances in multimodal machine learning itself and presents them in a common taxonomy. We go beyond the typical early and late fusion categorization and identify broader challenges that are faced by multimodal machine learning, namely: representation, translation, alignment, fusion, and co-learning. This new taxonomy will enable researchers to better understand the state of the field and identify directions for future research.

Machine Learning
Vision-based Human Behavior Recognition and Behavior Understanding: A Survey

Yonggang Zhang,Yingdong Ma

DOI: https://doi.org/10.2991/icmmct-16.2016.378

2016-01-01

Abstract:Behavior recognition and behavior understanding has been a popular topic in computer science. It attracted increasing interest in computer vision research community. Vision-based behavior Analysis has been applied broadly in many fields such as intelligent surveillance, perceptual user interface, and virtual reality. This article analyses the up-to-date development of motion representation, the processing procedure of behavior analysis, and the commonly used methods of this field.
A Systematic Survey on Human Behavior Recognition Methods

Yuan Meixue,Deepsim Intelligence Technology Co.,Zhao Jindong,Sun Ming

DOI: https://doi.org/10.1007/s42979-021-00932-x

2021-01-01

SN Computer Science

Abstract:Human behavior is an essential component of social interaction and is of great significance to identify and analyze human behaviors in a variety of fields. Due to the rapid development of computer vision and machine learning technology, machine with intelligence has started replacing human beings to observe, perceive and analyze the explosive growth of image and video data. Computer vision and machine learning-based human behavior recognition is one of these tasks, which has become a particularly hot research topic in many different fields, such as intelligent monitoring, human–computer interaction, smart home, virtual reality, and medical diagnosis. In this study, we survey systematically the popular methods, algorithms, models and well-known action datasets in human behavior analysis in the past two decades. In addition, the advantages and disadvantages of the methods are discussed and propitious future research directions are also presented. The results of this survey reveal that paradigms of human behavior analysis is being shifted from traditional RGB to RGB-D, from deep learning to more intelligent and automated deep reinforcement learning, and from fixed camera devices to portable devices and channel state information (CSI), and paradigms based on automated deep reinforcement learning and portable devices and CSI would become some hot topics for future research on human behavior analysis.
Multimodal Methods for Analyzing Learning and Training Environments: A Systematic Literature Review

Clayton Cohn,Eduardo Davalos,Caleb Vatral,Joyce Horn Fonteles,Hanchen David Wang,Meiyi Ma,Gautam Biswas

2024-08-23

Abstract:Recent technological advancements have enhanced our ability to collect and analyze rich multimodal data (e.g., speech, video, and eye gaze) to better inform learning and training experiences. While previous reviews have focused on parts of the multimodal pipeline (e.g., conceptual models and data fusion), a comprehensive literature review on the methods informing multimodal learning and training environments has not been conducted. This literature review provides an in-depth analysis of research methods in these environments, proposing a taxonomy and framework that encapsulates recent methodological advances in this field and characterizes the multimodal domain in terms of five modality groups: Natural Language, Video, Sensors, Human-Centered, and Environment Logs. We introduce a novel data fusion category -- mid fusion -- and a graph-based technique for refining literature reviews, termed citation graph pruning. Our analysis reveals that leveraging multiple modalities offers a more holistic understanding of the behaviors and outcomes of learners and trainees. Even when multimodality does not enhance predictive accuracy, it often uncovers patterns that contextualize and elucidate unimodal data, revealing subtleties that a single modality may miss. However, there remains a need for further research to bridge the divide between multimodal learning and training studies and foundational AI research.

Machine Learning,Multimedia
Multimodal Characterization of Emotion within Multimedia Space

Dayo Samuel Banjo,Connice Trimmingham,Niloofar Yousefi,Nitin Agarwal

2023-11-21

Abstract:Technological advancement and its omnipresent connection have pushed humans past the boundaries and limitations of a computer screen, physical state, or geographical location. It has provided a depth of avenues that facilitate human-computer interaction that was once inconceivable such as audio and body language detection. Given the complex modularities of emotions, it becomes vital to study human-computer interaction, as it is the commencement of a thorough understanding of the emotional state of users and, in the context of social networks, the producers of multimodal information. This study first acknowledges the accuracy of classification found within multimodal emotion detection systems compared to unimodal solutions. Second, it explores the characterization of multimedia content produced based on their emotions and the coherence of emotion in different modalities by utilizing deep learning models to classify emotion across different modalities.

Multimedia
Detecting Human Behavior Models From Multimodal Observation in a Smart Home

Oliver Brdiczka,Matthieu Langet,Jérôme Maisonnasse,James L. Crowley,O. Brdiczka,M. Langet,J. Maisonnasse,J.L. Crowley

DOI: https://doi.org/10.1109/tase.2008.2004965

IF: 6.636

2009-10-01

IEEE Transactions on Automation Science and Engineering

Abstract:This paper addresses learning and recognition of human behavior models from multimodal observation in a smart home environment. The proposed approach is part of a framework for acquiring a high-level contextual model for human behavior in an augmented environment. A 3-D video tracking system creates and tracks entities (persons) in the scene. Further, a speech activity detector analyzes audio streams coming from head set microphones and determines for each entity, whether the entity speaks or not. An ambient sound detector detects noises in the environment. An individual role detector derives basic activity like “walking” or “interacting with table” from the extracted entity properties of the 3-D tracker. from the derived multimodal observations, different situations like “aperitif” or “presentation” are learned and detected using statistical models (HMMs). The objective of the proposed general framework is two-fold: the automatic offline analysis of human behavior recordings and the online detection of learned human behavior models. To evaluate the proposed approach, several multimodal recordings showing different situations have been conducted. The obtained results, in particular for offline analysis, are very good, showing that multimodality as well as multiperson observation generation are beneficial for situation recognition. Note to Practitioners—This paper was motivated by the problem of automatically recognizing human behavior and interactions in a smart home environment. The smart home environment is equipped with cameras and microphones that permit the observation of human activity in the scene. The objective is first to visualize the perceived human activities (e.g., for videoconferencing or surveillance of elderly people), and then to provide appropriate services based on these activities. We adopt a layered approach for human activity recognition in the environment. The layered framework is motivated by the human perception of human behavior in the scene (white box). The system first recognizes basic activities of individuals, called roles, like “interacting with table” or “walking.” Then, based on the recognized individual roles, group situations like “aperitif,” “presentation,” or “siesta” are recognized. in this paper, we describe an implementation that is based on a 3-D video tracking system, as well as speech activity detection using head set microphones. We evaluated the system for offline (a posteriori) situation classification and online (in scenario) situation recognition. A prototype system has been realized and installed at France Télécom R&D, visualizing current human behavior in the smart home to a distant user using a web interface. An open issue is still the detection of group dynamics and group formation, which is necessary for group situation recognition in (informal) real settings.

automation & control systems
Machine Learning for Multimodal Mental Health Detection: A Systematic Review of Passive Sensing Approaches

Lin Sze Khoo,Mei Kuan Lim,Chun Yong Chong,Roisin McNaney

DOI: https://doi.org/10.3390/s24020348

IF: 3.9

2024-01-07

Sensors

Abstract:As mental health (MH) disorders become increasingly prevalent, their multifaceted symptoms and comorbidities with other conditions introduce complexity to diagnosis, posing a risk of underdiagnosis. While machine learning (ML) has been explored to mitigate these challenges, we hypothesized that multiple data modalities support more comprehensive detection and that non-intrusive collection approaches better capture natural behaviors. To understand the current trends, we systematically reviewed 184 studies to assess feature extraction, feature fusion, and ML methodologies applied to detect MH disorders from passively sensed multimodal data, including audio and video recordings, social media, smartphones, and wearable devices. Our findings revealed varying correlations of modality-specific features in individualized contexts, potentially influenced by demographics and personalities. We also observed the growing adoption of neural network architectures for model-level fusion and as ML algorithms, which have demonstrated promising efficacy in handling high-dimensional features while modeling within and cross-modality relationships. This work provides future researchers with a clear taxonomy of methodological approaches to multimodal detection of MH disorders to inspire future methodological advancements. The comprehensive analysis also guides and supports future researchers in making informed decisions to select an optimal data source that aligns with specific use cases based on the MH disorder of interest.

engineering, electrical & electronic,chemistry, analytical,instruments & instrumentation
Multimodal Machine Learning in Image-Based and Clinical Biomedicine: Survey and Prospects

Elisa Warner,Joonsang Lee,William Hsu,Tanveer Syeda-Mahmood,Charles E. Kahn,Olivier Gevaert,Arvind Rao

DOI: https://doi.org/10.1007/s11263-024-02032-8

IF: 13.369

2024-04-24

International Journal of Computer Vision

Abstract:Machine learning (ML) applications in medical artificial intelligence (AI) systems have shifted from traditional and statistical methods to increasing application of deep learning models. This survey navigates the current landscape of multimodal ML, focusing on its profound impact on medical image analysis and clinical decision support systems. Emphasizing challenges and innovations in addressing multimodal representation, fusion, translation, alignment, and co-learning, the paper explores the transformative potential of multimodal models for clinical predictions. It also highlights the need for principled assessments and practical implementation of such models, bringing attention to the dynamics between decision support systems and healthcare providers and personnel. Despite advancements, challenges such as data biases and the scarcity of "big data" in many biomedical domains persist. We conclude with a discussion on principled innovation and collaborative efforts to further the mission of seamless integration of multimodal ML models into biomedical practice.

computer science, artificial intelligence
A Theory of Multimodal Learning

Zhou Lu

2023-12-16

Abstract:Human perception of the empirical world involves recognizing the diverse appearances, or 'modalities', of underlying objects. Despite the longstanding consideration of this perspective in philosophy and cognitive science, the study of multimodality remains relatively under-explored within the field of machine learning. Nevertheless, current studies of multimodal machine learning are limited to empirical practices, lacking theoretical foundations beyond heuristic arguments. An intriguing finding from the practice of multimodal learning is that a model trained on multiple modalities can outperform a finely-tuned unimodal model, even on unimodal tasks. This paper provides a theoretical framework that explains this phenomenon, by studying generalization properties of multimodal learning algorithms. We demonstrate that multimodal learning allows for a superior generalization bound compared to unimodal learning, up to a factor of $O(\sqrt{n})$, where $n$ represents the sample size. Such advantage occurs when both connection and heterogeneity exist between the modalities.

Machine Learning

Introduction to the Special Issue on Multimodal Machine Learning for Human Behavior Analysis

Editorial Note: Machine Learning for Multimodal Data Understanding

A Survey of Multimodal Machine Learning

Guest Editorial: Special Issue on Interactive Visual Analysis of Human and Crowd Behaviors

Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions

Foundations & Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions

Multimodal Machine Learning in Mental Health: A Survey of Data, Algorithms, and Challenges

Modality Influence in Multimodal Machine Learning

Human-Machine Learning for Data-Driven Scenarios

Introduction to the Special Issue on Deep Learning for Multi-Modal Intelligence Across Speech, Language, Vision, and Heterogeneous Signals

Vision+X: A Survey on Multimodal Learning in the Light of Data

Multimodal Machine Learning: A Survey and Taxonomy

Vision-based Human Behavior Recognition and Behavior Understanding: A Survey

A Systematic Survey on Human Behavior Recognition Methods

Multimodal Methods for Analyzing Learning and Training Environments: A Systematic Literature Review

Multimodal Characterization of Emotion within Multimedia Space

Detecting Human Behavior Models From Multimodal Observation in a Smart Home

Machine Learning for Multimodal Mental Health Detection: A Systematic Review of Passive Sensing Approaches

Multimodal Machine Learning in Image-Based and Clinical Biomedicine: Survey and Prospects

A Theory of Multimodal Learning