Abstract:Establishing the spatial relationship between 2D images captured by real cameras and 3D models of the environment (2D and 3D space) is one way to achieve the virtual-real registration for Augmented Reality (AR) in outdoor environments. In this paper, we propose to match the 2D images captured by real cameras and the rendered images from the 3D image-based point cloud to indirectly establish the spatial relationship between 2D and 3D space. We call these two kinds of images as cross-domain images, because their imaging mechanisms and nature are quite different. However, unlike real camera images, the rendered images from the 3D image-based point cloud are inevitably contaminated with image distortion, blurred resolution, and obstructions, which makes image matching with the handcrafted descriptors or existing feature learning neural networks very challenging. Thus, we first propose a novel end-to-end network, AE-GAN-Net, consisting of two AutoEncoders (AEs) with Generative Adversarial Network (GAN) embedding, to learn invariant feature descriptors for cross-domain image matching. Second, a domain-consistent loss function, which balances image content and consistency of feature descriptors for cross-domain image pairs, is introduced to optimize AE-GAN-Net. AE-GAN-Net effectively captures domain-specific information, which is embedded into the learned feature descriptors, thus making the learned feature descriptors robust against image distortion, variations in viewpoints, spatial resolutions, rotation, and scaling. Experimental results show that AE-GAN-Net achieves state-of-the-art performance for image patch retrieval with the cross-domain image patch dataset, which is built from real camera images and the rendered images from 3D image-based point cloud. Finally, by evaluating virtual-real registration for AR on a campus by using the cross-domain image matching results, we demonstrate the feasibility of applying the proposed virtual-real registration to AR in outdoor environments.

Point Cloud Generation Using Deep Adversarial Local Features for Augmented and Mixed Reality Contents

Latent-Space Laplacian Pyramids for Adversarial Representation Learning with 3D Point Clouds

Generating 3D Adversarial Point Clouds under the Principle of LiDARs

DPR-GAN: Dual-Stream Progressive Refinement for Adversarial 3D Point Cloud Generation

Learning Progressive Point Embeddings for 3D Point Cloud Generation

Point AE-DCGAN: A Deep Learning Model for 3D Point Cloud Lossy Geometry Compression.

Learning Representations and Generative Models for 3D Point Clouds

3D Point Cloud Shape Generation with Collaborative Learning of Generative Adversarial Network and Auto-Encoder

Constructing an Efficient and Adaptive Learning Model for 3D Object Generation

Points2Pix: 3D Point-Cloud to Image Translation using conditional Generative Adversarial Networks

MPR-GAN: A Novel Neural Rendering Framework for MLS Point Cloud With Deep Generative Learning

Flow-based GAN for 3D Point Cloud Generation from a Single Image

MSG-Point-GAN: Multi-Scale Gradient Point GAN for Point Cloud Generation

PCAC-GAN: A Sparse-Tensor-Based Generative Adversarial Network for 3D Point Cloud Attribute Compression

Adaptive Local Adversarial Attacks on 3D Point Clouds for Augmented Reality

Generative AI for 3-D Point Clouds

Deep Auxiliary Learning for Point Cloud Generation

LP-GAN: Learning Perturbations Based on Generative Adversarial Networks for Point Cloud Adversarial Attacks

Ae-Gan-Net: Learning Invariant Feature Descriptor To Match Ground Camera Images And A Large-Scale 3d Image-Based Point Cloud For Outdoor Augmented Reality

Generating 3D Adversarial Point Clouds

Learning Efficient Point Cloud Generation for Dense 3D Object Reconstruction