Abstract:This paper describes a physics-based end-to-end software simulation for image systems. We use the software to explore sensors designed to enhance performance in high dynamic range (HDR) environments, such as driving through daytime tunnels and under nighttime conditions. We synthesize physically realistic HDR spectral radiance images and use them as the input to digital twins that model the optics and sensors of different systems. This paper makes three main contributions: (a) We create a labeled (instance segmentation and depth), synthetic radiance dataset of HDR driving scenes. (b) We describe the development and validation of the end-to-end simulation framework. (c) We present a comparative analysis of two single-shot sensors designed for HDR. We open-source both the dataset and the software.

What problem does this paper attempt to address?

This paper attempts to address the challenge of obtaining high - quality images in high - dynamic - range (HDR) driving scenarios, especially when driving through tunnels during the day and at night. Specifically, the paper focuses on the following three main issues: 1. **Extremely High Luminance Difference**: - The light sources in driving scenes (such as headlights, street lights, traffic lights, etc.) are usually very bright and directional, and their radiation intensity may be five orders of magnitude higher than that of the nearby dark areas. This extreme dynamic range exceeds the capture capabilities of most single - exposure image sensors. - Mathematical expression: Let the radiance at a point \((x, y)\) be \(L(x, y)\), then in a high - dynamic - range scene, the difference between \(L_{bright}\) and \(L_{dark}\) can be expressed as: \[ L_{bright}=10^{5}\times L_{dark} \] 2. **Noise Problem in Low - light Conditions**: - In large - area regions of night - driving scenes, the absolute light intensity is low (for example, 1 - 10 cd/m²). For a sensor with a pixel size of 1 - 3 micrometers, an aperture number of about 4, and an exposure time of 60 Hz (16 ms), each pixel can only receive dozens of photons. - According to the Poisson distribution, even in the case of an ideal sensor, this photon level will severely limit the signal - to - noise ratio (Contrast - to - Noise Ratio, CNR) of the image. The Poisson noise formula is: \[ \sigma_{Poisson}=\sqrt{\lambda} \] where \(\lambda\) is the average number of photons. 3. **Effect of Optical Glare**: - The optical glare (flare) near strong light sources will significantly affect the image quality. Glare is caused by the aperture shape, surface scratches and dust, and reflections between optical elements, which leads to light diffusion and creates contrast that does not exist in the original scene. - Glare not only reduces the image contrast but may also introduce artifacts, masking important details (such as pedestrians or cyclists), thus posing a threat to driving safety. To solve these problems, the paper proposes a physics - based end - to - end software simulation framework named ISETHDR for generating synthetic HDR spectral radiance images and evaluating the design performance of different imaging systems. The main contributions of this framework include: - Creating a spectral scene dataset named ISETHDR with 2,000 semantically - annotated spectra, covering driving scenes under various lighting conditions. - Developing and validating an end - to - end simulation tool from scene spectral radiance to processed image - sensor data. - Comparing the design performance of two single - exposure HDR image sensors, aiming to improve the image quality and safety in night - driving scenes. These works are helpful for improving the perception ability of autonomous vehicles and other advanced driver - assistance systems (ADAS) in complex lighting conditions.

ISETHDR: A Physics-based Synthetic Radiance Dataset for High Dynamic Range Driving Scenes

A system for generating complex physically accurate sensor images for automotive applications

Car-Studio: Learning Car Radiance Fields from Single-View and Endless In-the-wild Images

Car-Studio: Learning Car Radiance Fields From Single-View and Unlimited In-the-Wild Images

I2-SDF: Intrinsic Indoor Scene Reconstruction and Editing Via Raytracing in Neural SDFs

NTIRE 2021 Challenge on High Dynamic Range Imaging: Dataset, Methods and Results

Using simulation to quantify the performance of automotive perception systems

HDRT: Infrared Capture for HDR Imaging

PLT-D3: A High-fidelity Dynamic Driving Simulation Dataset for Stereo Depth and Scene Flow

SynFog: A Photo-realistic Synthetic Fog Dataset based on End-to-end Imaging Simulation for Advancing Real-World Defogging in Autonomous Driving

PhyIR: Physics-based Inverse Rendering for Panoramic Indoor Images

DSEC: A Stereo Event Camera Dataset for Driving Scenarios

LidaRF: Delving into Lidar for Neural Radiance Field on Street Scenes

LightSim: Neural Lighting Simulation for Urban Scenes

3DRealCar: An In-the-wild RGB-D Car Dataset with 360-degree Views

A synthetic digital city dataset for robustness and generalisation of depth estimation models

HDR Light Field Imaging of Dynamic Scenes: A Learning-based Method and A Benchmark Dataset

Physically-inspired Deep Light Estimation from a Homogeneous-Material Object for Mixed Reality Lighting

HDR-NeRF: High Dynamic Range Neural Radiance Fields

SynthOutdoor: A synthetic dataset for 3D outdoor light estimation

Ithaca365: Dataset and Driving Perception under Repeated and Challenging Weather Conditions