Abstract:Full DNN-based image signal processors (ISPs) have been actively studied and have achieved superior image quality compared to conventional ISPs. In contrast to this trend, we propose a lightweight ISP that consists of simple conventional ISP functions but achieves high image quality by increasing expressiveness. Specifically, instead of tuning the parameters of the ISP, we propose to control them dynamically for each environment and even locally. As a result, state-of-the-art accuracy is achieved on various datasets, including other tasks like tone mapping and image enhancement, even though ours is lighter than DNN-based ISPs. Additionally, our method can process different image sensors with a single ISP through dynamic control, whereas conventional methods require training for each sensor.

What problem does this paper attempt to address?

### Problems the paper attempts to solve This paper aims to solve the problem of how to achieve perceptual quality when the image signal processor (ISP) processes RAW images from different image sensors. Specifically, the authors propose a lightweight and efficient ISP - PQDynamicISP, which achieves high - quality image processing by dynamically controlling the parameters of traditional ISP functions. #### Main problems and challenges 1. **Limitations of traditional ISPs**: - Although traditional ISPs are lightweight, they lack expressiveness and require individual parameter adjustment for each image sensor, which is time - consuming and has limited effectiveness. 2. **Limitations of fully DNN - based ISPs**: - Although DNN - based ISPs perform well in image quality, due to their high computational cost, especially when processing high - resolution images, they are difficult to be applied to practical devices. - Existing DNN - based ISPs are usually trained for specific sensors and cannot be used to process multiple sensors in general. 3. **Requirements for perceptual quality**: - For perceptual quality, it is necessary not only to eliminate the influence of sensor characteristics, but also to imitate the adaptation characteristics of the human eye to light and color, and carry out appropriate "seasoning" (for example, making the sky bluer) to ensure that the image looks natural and pleasant. 4. **Necessity of local and global control**: - In order to further improve image quality, it is necessary to perform inter - frame and local dynamic control of ISP parameters to better adapt to different environments and scenes. #### Proposed solutions - **Dynamically control traditional ISP functions**: By dynamically adjusting the parameters of traditional ISP functions, instead of relying entirely on deep - learning models like DNN - based ISPs, high - quality image processing can be achieved while maintaining lightweight. - **Local and global control**: Not only the inter - frame parameter control is achieved, but also a local control mechanism is introduced, enabling the ISP to dynamically adjust parameters according to different regions of the image. - **Automatic parameter adjustment method**: A method for automatically adjusting the parameter search space is proposed to avoid falling into local minima and ensure the stability and efficiency of the training process. - **Multi - task applicability**: This method is not only applicable to traditional ISP tasks, but can also be extended to other tasks such as tone mapping and image enhancement. Through these improvements, PQDynamicISP can achieve excellent perceptual quality while maintaining lightweight, and can process RAW images from different sensors without the need for individual training for each sensor.

PQDynamicISP: Dynamically Controlled Image Signal Processor for Any Image Sensors Pursuing Perceptual Quality

DynamicISP: Dynamically Controlled Image Signal Processor for Image Recognition

A Reconfigurable Convolution-in-Pixel CMOS Image Sensor Architecture

ReconfigISP: Reconfigurable Camera Image Processing Pipeline

AdaptiveISP: Learning an Adaptive Image Signal Processor for Object Detection

VisionISP: Repurposing the Image Signal Processor for Computer Vision Applications

A Tightly Coupled AI-ISP Vision Processor

LW-ISP: A Lightweight Model with ISP and Deep Learning

AI-assisted ISP hyperparameter auto tuning

ISP meets Deep Learning: A Survey on Deep Learning Methods for Image Signal Processing

Automatic ISP image quality tuning using non-linear optimization

DeepISP: Toward Learning an End-to-End Image Processing Pipeline

HISP: Heterogeneous Image Signal Processor Pipeline Combining Traditional and Deep Learning Algorithms Implemented on FPGA

Refactoring ISP for High-Level Vision Tasks

GenISP: Neural ISP for Low-Light Machine Cognition

Simple Image Signal Processing using Global Context Guidance

Towards Low-Cost Learning-based Camera ISP via Unrolled Optimization

ISP Parameter Optimization and FPGA Implementation for Object Detection in Low-Light Conditions

ParamISP: Learned Forward and Inverse ISPs using Camera Parameters

An Energy-Efficient Dynamic Feedback Image Signal Processor for Three-Dimensional Time-of-Flight Sensors