MiraGe: Editable 2D Images using Gaussian Splatting

Joanna Waczyńska,Tomasz Szczepanik,Piotr Borycki,Sławomir Tadeja,Thomas Bohné,Przemysław Spurek

2024-10-02

Abstract:Implicit Neural Representations (INRs) approximate discrete data through continuous functions and are commonly used for encoding 2D images. Traditional image-based INRs employ neural networks to map pixel coordinates to RGB values, capturing shapes, colors, and textures within the network's weights. Recently, GaussianImage has been proposed as an alternative, using Gaussian functions instead of neural networks to achieve comparable quality and compression. Such a solution obtains a quality and compression ratio similar to classical INR models but does not allow image modification. In contrast, our work introduces a novel method, MiraGe, which uses mirror reflections to perceive 2D images in 3D space and employs flat-controlled Gaussians for precise 2D image editing. Our approach improves the rendering quality and allows realistic image modifications, including human-inspired perception of photos in the 3D world. Thanks to modeling images in 3D space, we obtain the illusion of 3D-based modification in 2D images. We also show that our Gaussian representation can be easily combined with a physics engine to produce physics-based modification of 2D images. Consequently, MiraGe allows for better quality than the standard approach and natural modification of 2D images.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is that the existing 2D image representation methods have limitations in editing and modification. Specifically:

1. **Limitations of Traditional Implicit Neural Representations (INRs)**:
- Although traditional neural - network - based INRs can well capture the shape, color, and texture of an image, they do not support intuitive editing and modification of the image.

2. **Limitations of GaussianImage**:
- GaussianImage represents 2D images by using the Gaussian function. Although it performs well in compression rate and reconstruction quality, it lacks the ability to perform human - driven adjustments on the image and cannot achieve natural image modification.

To overcome these limitations, the author proposes a new model - MiraGe (Mirrored Gaussian), whose main goals are:

- **Achieve High - Quality 2D Image Reconstruction**: MiraGe uses parameterized flat Gaussian components to represent 2D images, thereby achieving reconstruction quality comparable to or even better than existing methods.

- **Support 3D - Space Editing of 2D Images**: MiraGe allows 2D images to be embedded in 3D space and uses mirror reflection and a physics engine to achieve natural image modification, including perspective transformation and other complex operations.

- **Combine with a Physics Engine for Realistic Modification**: By integrating a physics engine, MiraGe can simulate physical interactions in the real world, making the modification of 2D images more realistic and natural.

### Specific Problems and Solutions

1. **High - Quality Image Reconstruction**:
- MiraGe represents 2D images by using flat Gaussians and achieves higher reconstruction quality than existing methods. Experimental results show that MiraGe is significantly superior to other methods in metrics such as PSNR and MS - SSIM.

2. **2D Image Editing in 3D Space**:
- MiraGe allows users to edit 2D images in 3D space, for example, adjusting the angle and position of an object, thereby creating images with 3D effects. This editing method is more intuitive and flexible.

3. **Physics Engine Integration**:
- MiraGe can be combined with a physics engine (such as Taichi_elements or Blender) to achieve physics - based image modification. This makes image editing not limited to static changes but also able to simulate real physical behaviors, such as elastic deformation and collision.

### Summary

The main contribution of MiraGe lies in the fact that it not only improves the reconstruction quality of 2D images but also provides powerful editing functions, especially in terms of natural modification in 3D space and physical simulation. This brings new possibilities to fields such as computer graphics and virtual reality.