Plenoptic PNG: Real-Time Neural Radiance Fields in 150 KB

Jae Yong Lee,Yuqun Wu,Chuhang Zou,Derek Hoiem,Shenlong Wang

2024-09-24

Abstract:The goal of this paper is to encode a 3D scene into an extremely compact representation from 2D images and to enable its transmittance, decoding and rendering in real-time across various platforms. Despite the progress in NeRFs and Gaussian Splats, their large model size and specialized renderers make it challenging to distribute free-viewpoint 3D content as easily as images. To address this, we have designed a novel 3D representation that encodes the plenoptic function into sinusoidal function indexed dense volumes. This approach facilitates feature sharing across different locations, improving compactness over traditional spatial voxels. The memory footprint of the dense 3D feature grid can be further reduced using spatial decomposition techniques. This design combines the strengths of spatial hashing functions and voxel decomposition, resulting in a model size as small as 150 KB for each 3D scene. Moreover, PPNG features a lightweight rendering pipeline with only 300 lines of code that decodes its representation into standard GL textures and fragment shaders. This enables real-time rendering using the traditional GL pipeline, ensuring universal compatibility and efficiency across various platforms without additional dependencies.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### What problems does this paper attempt to solve?

This paper aims to solve the following three key problems in order to achieve efficient storage, transmission, and cross - platform browsing of 3D immersive content:

1. **The model size must be small enough**: To ensure that the user experience is not degraded in instant messaging and web browsing, the representation method of 3D scenes needs to be very compact.
2. **Universality of viewing and interaction**: The viewing and interaction of 3D content should not depend on specific dependencies or hardware, so that it can be easily accessed on various devices.
3. **Real - time rendering and interaction**: The rendering and interaction of 3D content must be carried out smoothly and in real - time.

Although existing methods such as NeRF (Neural Radiance Field) and Gaussian point drawing have made progress in some aspects, they still face challenges. For example:
- Although NeRF and its variants are compact, many of them cannot achieve real - time rendering and rely on dedicated renderers such as CUDA.
- Explicit geometric methods such as Gaussian point drawing and NeRF baking are fast and general - purpose, but explicit geometry requires a large amount of storage space.

To solve these problems, this paper proposes a new framework named **Plenoptic Portable Neural Graphics (PPNG)**. This framework has the following characteristics:
- **Extremely small model size**: By encoding multi - view images into an extremely compact tensor representation, the file size of each 3D scene can be compressed to about 150 KB.
- **Efficient real - time rendering**: A lightweight rendering pipeline has been developed, which can decode and render the PPNG representation in real - time in WebGL, making it possible to view and interact without additional dependencies on any platform.
- **Cross - platform compatibility**: By using standard GL textures and fragment shaders, compatibility and efficiency on a wide range of platforms are ensured.

### Specific objectives

The specific objectives of the paper are:
- **Encode 3D scenes**: Encode 3D scenes from 2D images to generate an extremely compact representation.
- **Real - time decoding and rendering**: Ensure that this representation can be decoded and rendered in real - time on various platforms.
- **Cross - platform sharing**: Make this compact 3D content as easy to transfer and view between different devices as ordinary pictures.

Through these improvements, PPNG achieves a smaller model size, faster training speed, and better rendering quality than existing methods, while maintaining wide - ranging platform compatibility.