Abstract:Natural human interactions for Mixed Reality Applications are overwhelmingly multimodal: humans communicate intent and instructions via a combination of visual, aural and gestural cues. However, supporting low-latency and accurate comprehension of such multimodal instructions (MMI), on resource-constrained wearable devices, remains an open challenge, especially as the state-of-the-art comprehension techniques for each individual modality increasingly utilize complex Deep Neural Network models. We demonstrate the possibility of overcoming the core limitation of latency–vs.–accuracy tradeoff by exploiting cross-modal dependencies–i.e., by compensating for the inferior performance of one model with an increased accuracy of more complex model of a different modality. We present a sensor fusion architecture that performs MMI comprehension in a quasi-synchronous fashion, by fusing visual, speech and gestural input. The architecture is reconfigurable and supports dynamic modification of the complexity of the data processing pipeline for each individual modality in response to contextual changes. Using a representative “classroom” context and a set of four common interaction primitives, we then demonstrate how the choices between low and high complexity models for each individual modality are coupled. In particular, we show that (a) a judicious combination of low and high complexity models across modalities can offer a dramatic 3-fold decrease in comprehension latency together with an increase $\sim$10-15% in accuracy, and (b) the right collective choice of models is context dependent, with the performance of some model combinations being significantly more sensitive to changes in scene context or choice of interaction.

Exploring Temporal Dependencies in Multimodal Referring Expressions with Mixed Reality

Exploring Multi-Modal Representations for Ambiguity Detection & Coreference Resolution in the SIMMC 2.0 Challenge

Self-paced Multi-grained Cross-modal Interaction Modeling for Referring Expression Comprehension

A Multi-Modal Context Reasoning Approach for Conditional Inference on Joint Textual and Visual Clues

Neuro-Symbolic Reasoning for Multimodal Referring Expression Comprehension in HMI Systems

A neuro-symbolic approach for multimodal reference expression comprehension

A Comparison of Visualisation Methods for Disambiguating Verbal Requests in Human-Robot Interaction

Jointly Optimizing Sensing Pipelines for Multimodal Mixed Reality Interaction

Tell and show: Combining multiple modalities to communicate manipulation tasks to a robot

Cognitive Principles in Robust Multimodal Interpretation

REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments.

Reference Resolution and Context Change in Multimodal Situated Dialogue for Exploring Data Visualizations

Probabilistic Multimodal Modeling for Human-Robot Interaction Tasks

MLLM as Retriever: Interactively Learning Multimodal Retrieval for Embodied Agents

Multimodal representation models for prediction and control from partial information

Real-Time Multimodal Turn-taking Prediction to Enhance Cooperative Dialogue during Human-Agent Interaction

Multimodal 3D Fusion and In-Situ Learning for Spatially Aware AI

MResT: Multi-Resolution Sensing for Real-Time Control with Vision-Language Models

Towards Mediating Shared Perceptual Basis in Situated Dialogue.

Modeling Multimodal Social Interactions: New Challenges and Baselines with Densely Aligned Representations