Multimodal 3D Fusion and In-Situ Learning for Spatially Aware AI

Chengyuan Xu,Radha Kumaran,Noah Stier,Kangyou Yu,Tobias Höllerer

2024-10-07

Abstract:Seamless integration of virtual and physical worlds in augmented reality benefits from the system semantically "understanding" the physical environment. AR research has long focused on the potential of context awareness, demonstrating novel capabilities that leverage the semantics in the 3D environment for various object-level interactions. Meanwhile, the computer vision community has made leaps in neural vision-language understanding to enhance environment perception for autonomous tasks. In this work, we introduce a multimodal 3D object representation that unifies both semantic and linguistic knowledge with the geometric representation, enabling user-guided machine learning involving physical objects. We first present a fast multimodal 3D reconstruction pipeline that brings linguistic understanding to AR by fusing CLIP vision-language features into the environment and object models. We then propose "in-situ" machine learning, which, in conjunction with the multimodal representation, enables new tools and interfaces for users to interact with physical spaces and objects in a spatially and linguistically meaningful manner. We demonstrate the usefulness of the proposed system through two real-world AR applications on Magic Leap 2: a) spatial search in physical environments with natural language and b) an intelligent inventory system that tracks object changes over time. We also make our full implementation and demo data available at (<a class="link-external link-https" href="https://github.com/cy-xu/spatially_aware_AI" rel="external noopener nofollow">this https URL</a>) to encourage further exploration and research in spatially aware AI.

Human-Computer Interaction,Artificial Intelligence,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem this paper attempts to address is: how to achieve semantic understanding and spatial perception of the physical environment in Augmented Reality (AR) so that users can perform various tasks through natural language queries or interactions with physical objects. Specifically, the authors propose a method of multimodal 3D fusion and in-situ learning, aiming to unify geometric, semantic, and language information into a single 3D representation to support more intelligent AR applications.

### Main Issues
1. **Semantic Understanding and Spatial Perception**:
- Existing AR systems have limitations in understanding the semantics of the physical environment, typically only recognizing predefined object categories.
- The authors aim to develop a system that can automatically recognize and understand arbitrary objects and support natural language queries and interactions.

2. **Multimodal 3D Fusion**:
- How to fuse visual-language features (such as those generated by the CLIP model) with 3D geometric models to improve the accuracy of environmental understanding.
- The authors propose a multi-channel voxel grid method that integrates geometric, semantic, and language information.

3. **In-situ Learning**:
- How to use user interaction data to train machine learning models so that they can remember and re-identify physical objects.
- The authors introduce the concept of "in-situ learning," which achieves tracking and recognition of physical objects through real-time data encoding and neural network updates.

### Solutions
1. **Multimodal 3D Reconstruction Process**:
- Use the TSDF (Truncated Signed Distance Function) fusion algorithm combined with visual-language features generated by the CLIP model to construct a multi-channel voxel grid.
- Improve the accuracy of feature and label estimation through multi-view fusion.

2. **Post-processing and Scene Management**:
- Extract triangular meshes from the TSDF volume to support rendering and integration with existing AR graphics pipelines.
- Perform 3D semantic segmentation to cluster voxels into individual objects, extracting complete object boundaries, shapes, and identities.
- Create an intelligent object inventory by associating each object's CLIP features to achieve automatic object recognition and tracking.

3. **In-situ Learning**:
- Use user interaction data with physical objects to train neural network models so that they can remember and re-identify objects.
- Support real-time data encoding and model updates to track changes in physical objects.

### Application Examples
1. **Natural Language Search in Physical Environments**:
- Users can perform searches in the physical environment through natural language queries (e.g., "items that might be dangerous for babies"), and the system will highlight relevant areas.

2. **Intelligent Inventory System**:
- Track changes in physical objects, such as moved or disappeared objects. With the time-tracing feature, users can view the historical locations of objects.

### Summary
This paper achieves semantic understanding and spatial perception of the physical environment through multimodal 3D fusion and in-situ learning, supporting more intelligent AR applications. These technologies not only enhance the interactivity and practicality of AR systems but also provide new directions for future research in spatial perception AI.