Abstract:Visual error metrics play a fundamental role in the quantification of perceived image similarity. Most recently, use cases for them in real-time applications have emerged, such as content-adaptive shading and shading reuse to increase performance and improve efficiency. A wide range of different metrics has been established, with the most sophisticated being capable of capturing the perceptual characteristics of the human visual system. However, their complexity, computational expense, and reliance on reference images to compare against prevent their generalized use in real-time, restricting such applications to using only the simplest available metrics. In this work, we explore the abilities of convolutional neural networks to predict a variety of visual metrics without requiring either reference or rendered images. Specifically, we train and deploy a neural network to estimate the visual error resulting from reusing shading or using reduced shading rates. The resulting models account for 70%-90% of the variance while achieving up to an order of magnitude faster computation times. Our solution combines image-space information that is readily available in most state-of-the-art deferred shading pipelines with reprojection from previous frames to enable an adequate estimate of visual errors, even in previously unseen regions. We describe a suitable convolutional network architecture and considerations for data preparation for training. We demonstrate the capability of our network to predict complex error metrics at interactive rates in a real-time application that implements content-adaptive shading in a deferred pipeline. Depending on the portion of unseen image regions, our approach can achieve up to $2\times$ performance compared to state-of-the-art methods.

What problem does this paper attempt to address?

This paper aims to solve the problem of visual error metric prediction in real - time applications. Specifically, it attempts to solve the following problems: 1. **Complex Metric Calculation in Real - Time Applications**: Most existing visual error metric calculations are complex and computationally expensive, relying on reference images for comparison, which limits their wide use in real - time applications. For example, in real - time rendering, these metrics can usually only be used in the simplest scenes and cannot handle complex, dynamically changing scenes. 2. **Error Prediction in Unseen Areas**: Existing methods mainly rely on reprojection data from previous frames to estimate perceptual errors, and this method is not effective when dealing with fast - moving cameras or newly emerging scene areas. For these unseen areas, traditional error metric methods cannot provide accurate predictions. To solve these problems, the paper proposes a method based on convolutional neural networks (CNN) that can predict various visual error metrics without the need for reference images and can provide accurate predictions even in unseen areas. Specific contributions include: 1. **Compact CNN Architecture**: A compact CNN is designed that can efficiently predict visual error metrics in real - time applications, applicable to both seen and unseen areas. 2. **Balanced Training Loss Function**: Two metric transformation methods (Clamped Transform and Logistic Transform) are proposed to balance the distribution of training data and improve the generalization ability of the model. 3. **Embedded Weber Correction**: By embedding Weber correction, the model can directly predict the corrected error metric, thereby reducing computational costs in real - time applications. 4. **Input Data Selection**: Analyze which screen - space data of the current frame are most valuable for predicting error metrics, and propose an effective 4 - channel texture input scheme, including a reprojection mask, one RGB channel of the reprojection color, one RGB channel of the diffuse material color, and the Z - axis of the view - space normal. 5. **Performance Evaluation**: Evaluate the quality, performance, and generalization ability of the proposed model, especially its performance in variable - rate shading (VRS) applications. Through these contributions, the paper provides a method for efficiently predicting complex visual error metrics in real - time applications, especially suitable for dynamically changing scenes and unseen areas.

Training and Predicting Visual Error for Real-Time Applications

Human Visual Perception Based Image Quality Assessment for Video Prediction

Learning to predict perceptual visibility of rendering deterioration in computer games

Adaptive Recurrent Frame Prediction with Learnable Motion Vectors.

Perceptual error optimization for Monte Carlo animation rendering

Image Visual Realism: from Human Perception to Machine Computation.

ErrorAug: Making Errors to Find Errors in Semantic Segmentation

RetiNerveNet: Using Recursive Deep Learning to Estimate Pointwise 24-2 Visual Field Data based on Retinal Structure

Error Model and Concise Temporal Network for Indirect Illumination in 3D Reconstruction

Learning Local Distortion Visibility From Image Quality Data-sets

Visual Analysis of Prediction Uncertainty in Neural Networks for Deep Image Synthesis

Perceptual Error Optimization for Monte Carlo Rendering

A Case Study of Image Enhancement Algorithms' Effectiveness of Improving Neural Networks' Performance on Adverse Images

Real-time Prediction of Segmentation Quality

Neural Two-Level Monte Carlo Real-Time Rendering

Real-Time Neural Appearance Models

A Multiresolution Network Architecture for Deferred Neural Lighting

VRS-NeRF: Visual Relocalization with Sparse Neural Radiance Field

Dataset And Metrics For Predicting Local Visible Differences

VisualNet: an End-to-End Human Visual System Inspired Framework to Reduce Inference Latency of Deep Neural Networks