Abstract:Abstract This systematic review on data modalities synthesises the research findings in terms of how to optimally use and combine such modalities when investigating cognitive, motivational, and emotional learning processes. ERIC, WoS, and ScienceDirect databases were searched with specific keywords and inclusion criteria for research on data modalities, resulting in 207 relevant publications. We provide findings in terms of target journal, country, subject, participant characteristics, educational level, foci, type of data modality, research method, type of learning, learning setting, and modalities used to study the different foci. In total, 18 data modalities were classified. For the 207 multimodal publications, 721 occurrences of modalities were observed. The most popular modality was interview followed by survey and observation. The least common modalities were heart rate variability, facial expression recognition, and screen recording. From the 207 publications, 98 focused exclusively on the cognitive aspects of learning, followed by 27 publications that only focused on motivation, while only five publications exclusively focused on emotional aspects. Only 10 publications focused on a combination of cognitive, motivational, and emotional aspects of learning. Our results plea for the increased use of objective measures, highlight the need for triangulation of objective and subjective data, and demand for more research on combining various aspects of learning. Further, rather than researching cognitive, motivational, and emotional aspects of learning separately, we encourage scholars to tap into multiple learning processes with multimodal data to derive a more comprehensive view on the phenomenon of learning.

Multimodality in Group Communication Research

Introduction: Multimodal interaction

A Multimodal Social Signal Processing Approach to Team Interactions

An introduction to multimodal communication

Multimodal Systems: Taxonomy, Methods, and Challenges

Multimodal Methods for Analyzing Learning and Training Environments: A Systematic Literature Review

How does Modality Matter? Investigating the Synthesis and Effects of Multi-modal Robot Behavior on Social Intelligence

Survey of Research on Multimodal Semantic Communication

Multimodal Sensing, Recognizing and Browsing Group Social Dynamics

Multimodal English

Foundations & Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions

Multimodal Neuroimaging Computing: the Workflows, Methods, and Platforms.

Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions

Cross-modal Communication Technology: A Survey

Critical Multimodal Studies of Popular Discourse

Multimodal interaction: A review

Vision+X: A Survey on Multimodal Learning in the Light of Data

Perspectives on learning with multimodal technology

Multimodal data indicators for capturing cognitive, motivational, and emotional learning processes: A systematic literature review

A Systematic Literature Review on Multimodal Machine Learning: Applications, Challenges, Gaps and Future Directions

Beyond robotic speech: mutual benefits to cognitive psychology and artificial intelligence from the study of multimodal communication