Abstract:We describe Platform for Situated Intelligence, an open-source framework for multimodal, integrative-AI systems. The framework provides infrastructure, tools, and components that enable and accelerate the development of applications that process multimodal streams of data and in which timing is critical. The framework is particularly well-suited for developing physically situated interactive systems that perceive and reason about their surroundings in order to better interact with people, such as social robots, virtual assistants, smart meeting rooms, etc. In this paper, we provide a brief, high-level overview of the framework and its main affordances, and discuss its implications for HRI.

What problem does this paper attempt to address?

This paper aims to solve the high - engineering - threshold problem faced when constructing multi - modal and integrated artificial intelligence systems (such as socially interactive autonomous robots). Despite the rapid progress and significant breakthroughs in individual AI technology fields such as natural language processing, computer vision, and robot control algorithms, integrating these technologies into end - to - end intelligent systems is still very challenging and time - consuming. Specifically, the paper points out the following main problems: 1. **Complex engineering tasks**: Before conducting this type of research, researchers often need to first overcome the arduous engineering tasks of constructing prototype systems, which include data - flow processing, real - time coordination of asynchronously executing components, etc. 2. **Lack of time - awareness support**: Existing programming languages and development tools lack important primitives and features for handling time and latency, resulting in a large amount of development time being spent on debugging low - level problems rather than solving high - level tasks. 3. **Insufficient debugging and visualization tools**: Multi - modal and integrated AI applications have specific debugging, visualization, and analysis requirements, and existing development tools do not fully meet these requirements. For example, standard breakpoint debugging or "print debugging" methods are not sufficient to meet the debugging requirements of these complex systems. 4. **Poor component reusability**: Researchers often need to "reinvent the wheel" and build their own infrastructures to represent and reason about important time constructs. These custom infrastructures are usually only applicable to specific applications and are difficult to generalize and reuse. To solve these problems, the paper introduces an open - source framework named "Platform for Situated Intelligence" (PSI for short). By providing runtime infrastructure, debugging and visualization tools, and an open component ecosystem, PSI aims to simplify the development, debugging, analysis, maintenance, and continuous evolution processes of multi - modal and integrated AI systems, thereby accelerating research and development in this field.

Accelerating the Development of Multimodal, Integrative-AI Systems with Platform for Situated Intelligence

Estuary: A Framework For Building Multimodal Low-Latency Real-Time Socially Interactive Agents

Multi-Platform Intelligent System for Multimodal Human-Computer Interaction

A Framework of Artificial Intelligence Augmented Design Support

SIGMA: An Open-Source Interactive System for Mixed-Reality Task Assistance Research

A middleware for implicit interaction

Enhancing Human–Robot Collaboration through a Multi-Module Interaction Framework with Sensor Fusion: Object Recognition, Verbal Communication, User of Interest Detection, Gesture and Gaze Recognition

Adaptive Multimodal and Multisensory Empathic Technologies for Enhanced Human Communication

Bidirectional Intent Communication: A Role for Large Foundation Models

SIMMC: Situated Interactive Multi-Modal Conversational Data Collection And Evaluation Platform

A Framework to Study Human-AI Collaborative Design Space Exploration

A Multilayer Architecture towards the Development and Distribution of Multimodal Interface Applications on the Edge

A Framework for Adapting Human-Robot Interaction to Diverse User Groups

Integrated multimodal artificial intelligence framework for healthcare applications

Enhancing Human Capabilities through Symbiotic Artificial Intelligence with Shared Sensory Experiences

Human Interaction Smart Subsystem—Extending Speech-Based Human-Robot Interaction Systems with an Implementation of External Smart Sensors

Intelligence Augmentation: Towards Building Human- machine Symbiotic Relationship

A multimodal interface to situated assembly robot systems

Development of a Human-Robot Interaction Platform for Dual-Arm Robots Based on ROS and Multimodal Artificial Intelligence

SIGVerse: A cloud-based VR platform for research on social and embodied human-robot interaction

Modular Customizable ROS-Based Framework for Rapid Development of Social Robots