On the Emergence of Symmetrical Reality

Zhenliang Zhang,Zeyu Zhang,Ziyuan Jiao,Yao Su,Hangxin Liu,Wei Wang,Song-Chun Zhu
2024-01-27
Abstract:Artificial intelligence (AI) has revolutionized human cognitive abilities and facilitated the development of new AI entities capable of interacting with humans in both physical and virtual environments. Despite the existence of virtual reality, mixed reality, and augmented reality for several years, integrating these technical fields remains a formidable challenge due to their disparate application directions. The advent of AI agents, capable of autonomous perception and action, further compounds this issue by exposing the limitations of traditional human-centered research approaches. It is imperative to establish a comprehensive framework that accommodates the dual perceptual centers of humans and AI agents in both physical and virtual worlds. In this paper, we introduce the symmetrical reality framework, which offers a unified representation encompassing various forms of physical-virtual amalgamations. This framework enables researchers to better comprehend how AI agents can collaborate with humans and how distinct technical pathways of physical-virtual integration can be consolidated from a broader perspective. We then delve into the coexistence of humans and AI, demonstrating a prototype system that exemplifies the operation of symmetrical reality systems for specific tasks, such as pouring water. Subsequently, we propose an instance of an AI-driven active assistance service that illustrates the potential applications of symmetrical reality. This paper aims to offer beneficial perspectives and guidance for researchers and practitioners in different fields, thus contributing to the ongoing research about human-AI coexistence in both physical and virtual environments.
Human-Computer Interaction,Artificial Intelligence
What problem does this paper attempt to address?
This paper discusses the integration of Artificial Intelligence (AI) with the physical and virtual world. Existing virtual reality, mixed reality, and augmented reality are technically independent, but with the development of AI, they require a unified framework to handle the dual-centered perception of humans and AI in physical and virtual environments. The paper proposes the concept of "Symmetrical Reality" (SR), which provides a unified representation that integrates various forms of fusion between the physical and virtual world. The SR framework allows researchers to better understand how AI collaborates with humans and integrates different technological paths of physical-virtual integration from a macro perspective. The importance of SR lies in the following aspects: 1. Currently, there is a lack of comprehensive physical-virtual world integration framework. 2. The emergence of AI agents increases system complexity, which cannot be fully described by existing VR and AR frameworks. 3. As a virtual entity, AI can observe and influence both the virtual and physical worlds simultaneously, requiring a framework like SR to accurately represent it. The paper also presents a prototype system of SR, used for specific tasks such as pouring water, as well as an AI-driven proactive assistance service to illustrate the potential applications of SR. The research objective is to provide useful perspectives and guidance for researchers in different fields, and promote the study of coexistence between humans and AI in physical and virtual environments. The structure of the paper includes a review of related work, the framework and technological challenges of SR, the impact of human-AI coexistence, and the design and application examples of SR system. Through the discussion of key concepts such as human-AI interaction and physical-virtual interaction, the paper provides a theoretical foundation for building human-compatible AI and enhancing human-machine interaction.