AR Cameras

Yuchen Ma,Yunhui Gao,Jiachen Wu,Liangcai Cao

DOI: https://doi.org/10.1364/fio.2023.fm5a.4

2023-01-01

Abstract:As the basis of virtual content creation, cameras are integral to AR applications. In this talk, we will introduce a flexible see-through camera for integration with AR devices and its applications.

What problem does this paper attempt to address?

See-through Camera for Integration with AR Devices

Yuchen Ma,Yunhui Gao,Jiachen Wu,Liangcai Cao,Zhengzhong Huang

DOI: https://doi.org/10.1117/12.3000603

2024-01-01

Abstract:As the basis of virtual content creation, cameras are integral to augmented reality (AR) applications. However, the opaque nature of the camera's appearance can prevent it from being integrated into a transparent AR display. Here we introduced an integrated, compact, and flexible see-through camera, which enables crucial functionalities like eye gaze tracking and eye-position perspective photography, enhancing the immersive experience and interaction possibilities.
Toward a See-Through Camera Via AR Lightguide.

Yuchen Ma,Yunhui Gao,Jiachen Wu,Liangcai Cao

DOI: https://doi.org/10.1364/ol.492370

IF: 3.6

2023-01-01

Optics Letters

Abstract:As the foundation of virtual content generation, cameras are crucial for augmented reality (AR) applications, yet their integration with transparent displays has remained a challenge. Prior efforts to develop see-through cameras have struggled to achieve high resolution and seamless integration with AR displays. In this work, we present LightguideCam, a compact and flexible see-through camera based on an AR lightguide. To address the overlapping artifacts in measurement, we present a compressive sensing algorithm based on an equivalent imaging model that minimizes computational consumption and calibration complexity. We validate our design using a commercial AR lightguide and demonstrate a field of view of 23.1° and an angular resolution of 0.1° in the prototype. Our LightguideCam has great potential as a plug-and-play extensional imaging component in AR head-mounted displays, with promising applications for eye-gaze tracking, eye-position perspective photography, and improved human-computer interaction devices, such as full-screen mobile phones.
Cloud-based Fusion Rendering for AR Applications

Jing Tang,Yuxi Zhou,Zhenyu Chen,Cheng Chen,Gang Xiong,Hongxin Zhang

DOI: https://doi.org/10.1109/dtpi59677.2023.10365411

2023-01-01

Abstract:To create digital twins, it is necessary to represent real objects or scenes in a virtual form. With the rapid development of Augmented Reality (AR) technology, its applications in various fields are becoming increasingly extensive. However, in traditional AR applications, the client has high performance requirements for device compute-intensive tasks, which cannot be met on low-performance devices. To solve this issue, in this paper, we propose and implement a novel digital twin method for AR technology, which is based on a cloud fusion platform. The approach includes deploying the Neural Radiance Field (NeRF) on a cloud server for high-quality model rendering and integrating simultaneous localization and mapping (SLAM) with AR visualization on the client side. This design effectively separates compute-intensive tasks from client-side rendering tasks, reducing the performance requirements of client devices and enhancing overall scalability and availability of the system. Practical application and tests show that the proposed AR system based on the cloud fusion platform performs well in terms of performance, stability, and scalability. This paper provides detailed discussions on the relevant technical methods and approaches, and showcases examples that serve as a foundation for the implementation of digital twins across diverse fields.
Stereo Camera System and Its Application in Networked Virtual Environment

L Chen,MH Tian,GC Chen,C Chen

DOI: https://doi.org/10.1109/icmlc.2002.1176744

2002-01-01

Abstract:With the appearance of different kinds of networked virtual environment systems, the technology of realistic video avatar has become a hotspot of research. How to create realistic video avatar with less hardware investment and limited computation is one of the purposes of these researches. We introduce a method that combines a single stereo camera system and a 3D head model to create a realistic video avatar. To create the video avatar's most important part-the face, we use a 2D image and a depth map, which is generated by the stereo camera system. The other parts of of the video avatar are created by mapping the head's texture to the 3D head model.
Multi-sensor Spatial Augmented Reality for Visualizing the Invisible Thermal Information of 3D Objects

Yanpeng Cao,Yafei Dong,Fan Wang,Jiangxin Yang,Yanlong Cao,Xin Li

DOI: https://doi.org/10.1016/j.optlaseng.2021.106634

IF: 5.666

2021-01-01

Optics and Lasers in Engineering

Abstract:In recent years, infrared imaging technology has been widely used to capture valuable thermal information of target objects for various applications such as construction engineering, advanced manufacturing, and medical diagnosis. However, it still remains a challenging task to visualize the invisible thermal information of three-dimensional objects for further evaluation and decision making. In this paper, we build a multi-sensor system consisting of a long-wave infrared camera, a RGBD camera, and a digital projector for multimodal data acquisition, real-time 3D thermographic reconstruction, and projector-based spatial augmented reality. We propose a hybrid method that combines the benefits of model-based ray casting and frame-based image warping to generate high-quality projected images with complete view coverage and abundant thermal details for visualizing the invisible thermal information on 3D objects. The effectiveness of the proposed method has been validated for two typical thermal imaging applications including human body temperature monitoring and non-destructive evaluation of composite materials.
A touchable virtual screen interaction system with handheld Kinect camera

Jianing Li,Yang Li,Junfei Zhang,Dongxiao Li,Lianghao Wang,Ming Zhang

DOI: https://doi.org/10.1109/WCSP.2013.6677202

2013-01-01

Abstract:In recent years, augmented reality (AR) has been a research hotpot. It emphasizes that virtual objects and real scene are merged with the correct perspective relationship. However, most of existing AR systems simply adds virtual objects on real scene image. It means that if a real hand is in front of the virtual object, the virtual object is still cover the hand with wrong occlusion relationship. On the other hand, most systems just present virtual information, preventing user from touching or interacting with the virtual object.. In this paper, we build an AR system achieving mutual occlusion and interaction. Instead of using structure from motion method, we make use of the depth image of Kinect camera to integrate a 3D space volume and track the camera pose as well. Then we propose a layer rendering method according to the depth relationship to implement mutual occlusion fusion. At last, we fulfill collision detection and interaction through a fast voxel detection method. We define the neighbor cube for every virtual surface point and implement the detection in parallel with GPU.
Multi-Camera Lighting Estimation for Photorealistic Front-Facing Mobile Augmented Reality

Yiqin Zhao,Sean Fanello,Tian Guo

DOI: https://doi.org/10.1145/3572864.3580337

2023-01-16

Abstract:Lighting understanding plays an important role in virtual object composition, including mobile augmented reality (AR) applications. Prior work often targets recovering lighting from the physical environment to support photorealistic AR rendering. Because the common workflow is to use a back-facing camera to capture the physical world for overlaying virtual objects, we refer to this usage pattern as back-facing AR. However, existing methods often fall short in supporting emerging front-facing mobile AR applications, e.g., virtual try-on where a user leverages a front-facing camera to explore the effect of various products (e.g., glasses or hats) of different styles. This lack of support can be attributed to the unique challenges of obtaining 360$^\circ$ HDR environment maps, an ideal format of lighting representation, from the front-facing camera and existing techniques. In this paper, we propose to leverage dual-camera streaming to generate a high-quality environment map by combining multi-view lighting reconstruction and parametric directional lighting estimation. Our preliminary results show improved rendering quality using a dual-camera setup for front-facing AR compared to a commercial solution.

Computer Vision and Pattern Recognition
Towards VR and AR

Chen Su,Yifan Peng,Qing Zhong,Haifeng Li,Rui Wang,Wolfgang Heidrich,Xü Liu

DOI: https://doi.org/10.1145/2988240.2988251

2016-01-01

Abstract:We present a 360° light field display to visualize 3D objects above the table-top screen, and provide a graspable and measurable virtual or augmented reality environment. The display is constructed by a spinning flat-plate deflected diffuser screen, a high-speed light field projector, and the user tracking and interaction components. A rear-projection light propagation is specially designed to reconstruct the light field. By calibrating this levitated light field to the physical dimension, and capturing user gestures, we enable multiple users mid-air interactions based on their virtual measurement perception.
Handling Occlusions in Video-Based Augmented Reality Using Depth Information

Jiejie Zhu,Zhigeng Pan,Chao Sun,Wenzhi Chen

DOI: https://doi.org/10.1002/cav.326

IF: 1.01

2010-01-01

Computer Animation and Virtual Worlds

Abstract:Augmented Reality (AR) composes virtual objects with real scenes in a mixed environment where human–computer interaction has more semantic meanings. To seamlessly merge virtual objects with real scenes, correct occlusion handling is a significant challenge. We present an approach to separate occluded objects in multiple layers by utilizing depth, color, and neighborhood information. Scene depth is obtained by stereo cameras and two Gaussian local kernels are used to represent color, spatial smoothness. These three cues are intelligently fused in a probability framework, where the occlusion information can be safely estimated. We apply our method to handle occlusions in video-based AR where virtual objects are simply overlapped on real scenes. Experiment results show the approach can correctly register virtual and real objects in different depth layers, and provide a spatial-awareness interaction environment. Copyright © 2009 John Wiley & Sons, Ltd. The paper presents an approach to do occlusion handling using depth information. (a), (b) show an example of the occlusions before and after our approach. (c) shows the depth relationship. Different color denotes for different depth layers.
Towards VR and AR enhancement: light field display with mid-air interaction.

Chen Su,Yifan Peng,Qing Zhong,Haifeng Li,Rui Wang,Wolfgang Heidrich,Xu Liu

DOI: https://doi.org/10.1145/2992138.2992151

2016-01-01

Abstract:We present a 360° light field display to visualize 3D objects above the table-top screen, and provide a graspable and measurable virtual or augmented reality environment. The display is constructed by a spinning flat-plate deflected diffuser screen, a high-speed light field projector, and the user tracking and interaction components. A rear projection light propagation is specially designed to reconstruct the light field. By calibrating this levitated light field to the physical dimension, and capturing user gestures, we enable multiple users mid-air interactions based on their virtual measurement perception.
Virtual reality realization technology and its application based on augmented reality.

Ning Li,Chenggang Yan,Feng Xu,Liang Li,Zunjie Zhu,Biyao Shao,Bingjian Gong

DOI: https://doi.org/10.1109/GlobalSIP.2017.8309188

2017-01-01

Abstract:We propose an Augmented Reality (AR) implementation technology based on Virtual Reality (VR), and build a system which contains a headset VR equipment, one depth camera, one color camera and a smart phone. We will collect the depth and color information through these two cameras. When the depth and color information are given, we can integrate them into four channel images fused by RGB color image and depth-image information (RGB-D),then use simultaneous localization and mapping (SLAM) to transform RGB-D image sequence into a real scene 3D model, and identify the layout of the scene on this basis. Furthermore, by the means of the Augmented Reality technology we can add auxiliary information about the objects in the scene and place the virtual objects arbitrarily into reality to achieve the fusion of virtual objects and the real scene. Finally we produce the 3D visual effect which is able to be observed by human eyes, and achieve virtual reality effect by the headset device.Through our method, virtual and real objects can be seen on the screen created by the depth and color camera to achieve the interaction of virtual and real scene.
Scene-Planning System Based on Augmented Reality Technique

王靖滨,李明,耿卫东,潘云鹤

DOI: https://doi.org/10.3969/j.issn.1006-8961.2002.04.018

2002-01-01

Abstract:Augmented Reality(AR) is a growing area in virtual reality research. It merges the virtual object or other additional information generated by the computer into the user's view of the real world. It is helpful to apply AR technique to the scene planning for architectural designers or beauty spot developers. In this paper, a typical AR application system--Scene Planning is introduced to demonstrate this idea.To combine the virtual buildings and real scene, the geometric coherence and illumination coherence problem must be sloved. The local 3D reconstruction and interactive camera calibration are used to resolve the geometric problem. For the illumination problem, the interactive and untomatic method is combined,and the shadow of the virtual object is also taken into account. The human's interaction is also involved during the above process. By evaluating the result image synthesized from various view angles, the designer can estimate and optimize the layout of virtual scene intuitively, and eliminate the inaccuracy during the planning process. Finally, some work under improvement and future research are also discussed.
A Survey of Monocular Simultaneous Localization and Mapping

Liu Haomin,Zhang Guofeng,Bao Hujun

DOI: https://doi.org/10.3969/j.issn.1003-9775.2016.06.001

2016-01-01

Abstract:Augmented reality (AR) is a technique that allows to seamlessly composite virtual objects or informa-tion into real scene. Compared to traditional text, images and videos, AR is a more effective and intuitive way for information presentation and has wide applications. Simultaneous localization and mapping (SLAM) is a key fundamental technique for augmented reality, which provides the ability of self-localization in an unknown envi-ronment and mapping the 3D environment simultaneously. The localization and mapping enables fusion of virtual objects and real scenes in a geometrically consistent way. In this paper, we describe the basic principles of Visual SLAM, and introduce some state-of-the-art monocular SLAM methods with deep analysis and comparison. Fi-nally, we discuss some research tendency in recent years and make conclusions.
Tele-AR System Based on Real-time Camera Tracking

Yanke Wang,Fan Zhong,Chao Li,Hui Xiang,Qunsheng Peng,Xueying Qin

DOI: https://doi.org/10.1109/icvrv.2012.13

2012-01-01

Abstract:We propose a visual tele-AR system, which is a combination of tele-existence and AR techniques. By wearing a video see-through HMD and moving his head, a user in this system is able to tele-operate a remote pan-tilt platform to observe the remote scene. Unlike traditional tele-existence systems which use mechanical or magnetic sensors for tele-operation, we propose a novel tele-operation method based on real-time tracking of master and slave cameras. The master camera is a component of the video see-through HMD mounted in front of user's eyes, while the slave camera is carried by a remote platform. We estimate use's head motion by tracking the master camera in real time, and translate head motions into motion commands to drive the remote pan-tilt platform. Ideally, motion of the remote platform will be exactly the same as user's head. Unfortunately, motion speed of the slave platform is usually limited by their mechanical capability, which may cause pose inconsistence between master and slave cameras, especially when user's head moves fast. We solve this problem by also tracking slave camera in real time, and warping video images according to difference between master and slave camera pose. Since slave camera is tracked, we can integrate virtual objects into remote scene, giving the user an augmented visual sense. To reduce system delay, our implementation is accelerated GPU parallel computation. Experimental results show the efficiency and robustness of our method.
Robust Keyframe-Based Monocular SLAM for Augmented Reality

Haomin Liu,Guofeng Zhang,Hujun Bao

DOI: https://doi.org/10.1109/ismar.2016.24

2016-01-01

Abstract:Keyframe-based SLAM has achieved great success in terms of accuracy, efficiency and scalability. However, due to parallax requirement and delay of map expansion, traditional keyframe-based methods easily encounter the robustness problem in the challenging cases especially for fast motion with strong rotation. For AR applications in practice, these challenging cases are easily encountered, since a home user may not carefully move the camera to avoid potential problems. With the above motivation, in this paper, we present RKSLAM, a robust keyframe-based monocular SLAM system that can reliably handle fast motion and strong rotation, ensuring good AR experiences. First, we propose a novel multihomography based feature tracking method which is robust and efficient for fast motion and strong rotation. Based on it, we propose a real-time local map expansion scheme to triangulate the observed 3D points immediately without delay. A sliding-window based camera pose optimization framework is proposed, which imposes the motion prior constraints between consecutive frames through simulated or real IMU data. Qualitative and quantitative comparisons with the state-of-the-art methods, and an AR application on mobile devices demonstrate the effectiveness of the proposed approach.
A Case for Camera-as-a-Service.

Mengwei Xu,Yunxin Liu,Xuanzhe Liu

DOI: https://doi.org/10.1109/mprv.2021.3052532

IF: 1.603

2021-01-01

IEEE Pervasive Computing

Abstract:A new vision of harnessing the ocean of video data captured by cameras in an efficient, secure, and manageable way. Following the hardware trends in reality, we advocate to treat cameras as the first-class computing and storage platform, and expose their services to third parties. To this end, the cameras should move from hard-coded to software-defined, from dumb data source to autonomous data processor, from isolated to orchestrated, and from one-shot to human-in-the-loop.
A Web-based Augmented Reality Plat-form Using Pictorial QR Code for Educational Purposes and Beyond.

Minh Nguyen,Huy Le,Phu Minh Lai,Wei Qi Yan

DOI: https://doi.org/10.1145/3359996.3364793

2019-01-01

Abstract:Augmented Reality (AR) provides the capability to overlay virtual 3D information onto a 2D printed flat surface; for example, displaying a 3D model on a single flat card that accompanies with the diagram shown in a learning text-book. The student can zoom in and out, rotate, and perceive the animation of the figure in real-time. This will make the educational theory more attractive; hence, motivates students to learn. AR is a great tool; however, the setup and display are not straight-forward (there are many different AR markers with different encryption, decryption methods, and displaying flat-forms). In this paper, we proposed a portable browser-based platform which uses the advantages of AR along with scan-able QR Code on mobile phones to enhance instant 3D visualisation. The user only needs a smart-phone (Apple iPhone or Android) with Internet-enabled; no specific Apps are needed to install. The user scans the QR Code embedded in a colour image, the code will link to a public website, and the website will produce AR Experience right on top of the browser. As a result, it provides a stress-free, low-cost, portable, and promising solution for not only educational purposes but also many other fields such as gaming, property selling, e-commerce, reporting. The set up is convenient: the user uploads a picture (e.g. a racing car), and what actions to be related to it (a 3D model to display, or a movie to play). The system will add on the picture one small colour QR code (to redirect to an online URL) and a thin black border. The user also uploads the 3D model (GLTF files) that he wants to display on top of the card to finish the set-up. At the display, the user can print the AR card, point their smart-phone towards the card, and pre-setup AR models or actions will appear on it. To students, these 3D graphics or animations will allow them to learn and understand the lessons in a much more intuitive way.
A Visual-Inertial Fusion Based Tracking System For Mobile Augmented Reality

cheng lin,lianghao wang,dongxiao li,ming zhang

DOI: https://doi.org/10.1109/icspcc.2015.7338954

2015-01-01

Abstract:Recently, smart phones are usually equipped with cameras and motion sensors. This means that mobile phones have become good platforms to deploy augmented reality (AR) applications. We propose a new tracking system for AR which uses inertial measurement units (gyroscope, accelerometer and magnetic field sensor) combined with a camera to automatically detect and track a dominant plane in the scene without prior knowledge about the environment. Meanwhile, it also estimates the pose of the device, so the rendered virtual objects can be correctly overplayed on the real view. While tracking the plane and computing the camera pose with respect to the scene, the solution also needs to construct and update the map of the scene, which consists of keyframes and keypoints in our scenario. Finally, we implement our system and test its performance on Android smart phones. Our experimental results show that the system can achieve real-time and drift-free performance and the tracking is quite accurate and robust.
An Indoor Scene Recognition-Based 3D Registration Mechanism for Real-Time AR-GIS Visualization in Mobile Applications

Wei Ma,Hanjiang Xiong,Xuefeng Dai,Xianwei Zheng,Yan Zhou

DOI: https://doi.org/10.3390/ijgi7030112

2018-03-15

Abstract:Mobile Augmented Reality (MAR) systems are becoming ideal platforms for visualization, permitting users to better comprehend and interact with spatial information. Subsequently, this technological development, in turn, has prompted efforts to enhance mechanisms for registering virtual objects in real world contexts. Most existing AR 3D Registration techniques lack the scene recognition capabilities needed to describe accurately the positioning of virtual objects in scenes representing reality. Moreover, the application of such registration methods in indoor AR-GIS systems is further impeded by the limited capacity of these systems to detect the geometry and semantic information in indoor environments. In this paper, we propose a novel method for fusing virtual objects and indoor scenes, based on indoor scene recognition technology. To accomplish scene fusion in AR-GIS, we first detect key points in reference images. Then, we perform interior layout extraction using a Fully Connected Networks (FCN) algorithm to acquire layout coordinate points for the tracking targets. We detect and recognize the target scene in a video frame image to track targets and estimate the camera pose. In this method, virtual 3D objects are fused precisely to a real scene, according to the camera pose and the previously extracted layout coordinate points. Our results demonstrate that this approach enables accurate fusion of virtual objects with representations of real world indoor environments. Based on this fusion technique, users can better grasp virtual three-dimensional representations on an AR-GIS platform.

geography, physical,remote sensing,computer science, information systems
HoloCamera: Advanced Volumetric Capture for Cinematic-Quality VR Applications

Jonathan Heagerty,Sida Li,Eric Lee,Shuvra Bhattacharyya,Sujal Bista,Barbara Brawn,Brandon Y. Feng,Susmija Jabbireddy,Joseph JaJa,Hernisa Kacorri,David Li,Derek Yarnell,Matthias Zwicker,Amitabh Varshney

DOI: https://doi.org/10.1109/tvcg.2024.3372123

IF: 5.2

2024-04-24

IEEE Transactions on Visualization and Computer Graphics

Abstract:High-precision virtual environments are increasingly important for various education, simulation, training, performance, and entertainment applications. We present HoloCamera, an innovative volumetric capture instrument to rapidly acquire, process, and create cinematic-quality virtual avatars and scenarios. The HoloCamera consists of a custom-designed free-standing structure with 300 high-resolution RGB cameras mounted with uniform spacing spanning the four sides and the ceiling of a room-sized studio. The light field acquired from these cameras is streamed through a distributed array of GPUs that interleave the processing and transmission of 4K resolution images. The distributed compute infrastructure that powers these RGB cameras consists of 50 Jetson AGX Xavier boards, with each processing unit dedicated to driving and processing imagery from six cameras. A high-speed Gigabit Ethernet network fabric seamlessly interconnects all computing boards. In this systems paper, we provide an in-depth description of the steps involved and lessons learned in constructing such a cutting-edge volumetric capture facility that can be generalized to other such facilities. We delve into the techniques employed to achieve precise frame synchronization and spatial calibration of cameras, careful determination of angled camera mounts, image processing from the camera sensors, and the need for a resilient and robust network infrastructure. To advance the field of volumetric capture, we are releasing a high-fidelity static light-field dataset, which will serve as a benchmark for further research and applications of cinematic-quality volumetric light fields.

computer science, software engineering

AR Cameras

See-through Camera for Integration with AR Devices

Toward a See-Through Camera Via AR Lightguide.

Cloud-based Fusion Rendering for AR Applications

Stereo Camera System and Its Application in Networked Virtual Environment

Multi-sensor Spatial Augmented Reality for Visualizing the Invisible Thermal Information of 3D Objects

A touchable virtual screen interaction system with handheld Kinect camera

Multi-Camera Lighting Estimation for Photorealistic Front-Facing Mobile Augmented Reality

Towards VR and AR

Handling Occlusions in Video-Based Augmented Reality Using Depth Information

Towards VR and AR enhancement: light field display with mid-air interaction.

Virtual reality realization technology and its application based on augmented reality.

Scene-Planning System Based on Augmented Reality Technique

A Survey of Monocular Simultaneous Localization and Mapping

Tele-AR System Based on Real-time Camera Tracking

Robust Keyframe-Based Monocular SLAM for Augmented Reality

A Case for Camera-as-a-Service.

A Web-based Augmented Reality Plat-form Using Pictorial QR Code for Educational Purposes and Beyond.

A Visual-Inertial Fusion Based Tracking System For Mobile Augmented Reality

An Indoor Scene Recognition-Based 3D Registration Mechanism for Real-Time AR-GIS Visualization in Mobile Applications

HoloCamera: Advanced Volumetric Capture for Cinematic-Quality VR Applications