Abstract:Point-Based Neural Rendering (PBNR) is emerging as a promising class of rendering techniques, which are permeating all aspects of society, driven by a growing demand for real-time, photorealistic rendering in AR/VR and digital twins. Achieving real-time PBNR on mobile devices is challenging. This paper proposes MetaSapiens, a PBNR system that for the first time delivers real-time neural rendering on mobile devices while maintaining human visual quality. MetaSapiens combines three techniques. First, we present an efficiency-aware pruning technique to optimize rendering speed. Second, we introduce a Foveated Rendering (FR) method for PBNR, leveraging humans' low visual acuity in peripheral regions to relax rendering quality and improve rendering speed. Finally, we propose an accelerator design for FR, addressing the load imbalance issue in (FR-based) PBNR. Our evaluation shows that our system achieves an order of magnitude speedup over existing PBNR models without sacrificing subjective visual quality, as confirmed by a user study. The code and demo are available at: <a class="link-external link-https" href="https://horizon-lab.org/metasapiens/" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is: **How to achieve real - time point - based neural rendering (PBNR) on mobile devices while maintaining human visual quality**. Specifically, the authors proposed the MetaSapiens system, aiming to overcome the problem that existing PBNR models cannot achieve real - time rendering on mobile devices, especially in the context of the increasing demand for real - time and realistic rendering in application scenarios such as AR/VR and digital twins. ### Background of the Main Problem 1. **Demands and Challenges of Point - Based Neural Rendering (PBNR)**: - PBNR is an emerging rendering technique that provides realistic rendering effects by learning scene radiance. - Although PBNR is more efficient than traditional neural rendering methods (such as NeRF), it is still difficult to achieve real - time rendering on mobile devices, usually with a frame rate lower than 10 FPS. 2. **Insufficiencies of Existing Optimization Methods**: - Existing optimization methods mainly focus on reducing the number of points in the model, but this method cannot significantly improve the rendering speed because different points contribute differently to the computational cost. ### Solutions To solve the above problems, the paper proposed the MetaSapiens system, which combines the following three key techniques: 1. **Efficiency - Aware Pruning**: - By introducing the computational efficiency (CE) metric, it gives priority to pruning points that have a high computational cost but contribute little to the image quality. - The formula is defined as follows: \[ \text{CE}_i=\frac{\text{Val}_i}{\text{Comp}_i} \] where \(\text{Val}_i\) represents the contribution of point \(i\) to the pixel value, and \(\text{Comp}_i\) represents the computational cost of point \(i\). 2. **Foveated Rendering (FR)**: - Utilizing the low - resolution characteristics of the human visual system in the peripheral area, it gradually reduces the rendering quality in the edge area, thereby accelerating the rendering. - A new data representation method is introduced, which enables the sharing of parameters and calculations in different quality areas, reducing the storage requirements. 3. **Hardware Accelerator Design**: - Aiming at the load imbalance problem brought by FR, a scheme for dynamically merging tiles is designed, and the hardware utilization rate is improved by pipelining adjacent stages. ### Experimental Results The experimental results show that the MetaSapiens system achieves real - time rendering on mobile devices, and the subjective visual quality is comparable to that of the dense model. Compared with other state - of - the - art PBNR methods, MetaSapiens has significant improvements in both rendering quality and speed. ### Summary The main contribution of this paper is to propose a new method that can achieve real - time point - based neural rendering on mobile devices, solve the problem of insufficient performance of existing methods on mobile devices, and ensure high - quality rendering effects through multiple technological innovations.

MetaSapiens: Real-Time Neural Rendering with Efficiency-Aware Pruning and Accelerated Foveated Rendering