Abstract:Real-world imaging systems acquire measurements that are degraded by noise, optical aberrations, and other imperfections that make image processing for human viewing and higher-level perception tasks challenging. Conventional cameras address this problem by compartmentalizing imaging from high-level task processing. As such, conventional imaging involves processing the RAW sensor measurements in a sequential pipeline of steps, such as demosaicking, denoising, deblurring, tone-mapping and compression. This pipeline is optimized to obtain a visually pleasing image. High-level processing, on the other hand, involves steps such as feature extraction, classification, tracking, and fusion. While this siloed design approach allows for efficient development, it also dictates compartmentalized performance metrics, without knowledge of the higher-level task of the camera system. For example, today's demosaicking and denoising algorithms are designed using perceptual image quality metrics but not with domain-specific tasks such as object detection in mind. We propose an end-to-end differentiable architecture that jointly performs demosaicking, denoising, deblurring, tone-mapping, and classification. The architecture learns processing pipelines whose outputs differ from those of existing ISPs optimized for perceptual quality, preserving fine detail at the cost of increased noise and artifacts. We demonstrate on captured and simulated data that our model substantially improves perception in low light and other challenging conditions, which is imperative for real-world applications. Finally, we found that the proposed model also achieves state-of-the-art accuracy when optimized for image reconstruction in low-light conditions, validating the architecture itself as a potentially useful drop-in network for reconstruction and analysis tasks beyond the applications demonstrated in this work.

Cubixel: a novel paradigm in image processing using three-dimensional pixel representation

XCube: Large-Scale 3D Generative Modeling using Sparse Voxel Hierarchies

HyperCube: Implicit Field Representations of Voxelized 3D Models

SCube: Instant Large-Scale Scene Reconstruction using VoxSplats

CUBE360: Learning Cubic Field Representation for Monocular 360 Depth Estimation for Virtual Reality

A novel 2D-to-3D conversion method based on blocks world

Vista3D: Unravel the 3D Darkside of a Single Image

Pix2Vox: Context-aware 3D Reconstruction from Single and Multi-view Images

Diving Segmentation Model into Pixels

Pixels and Panoramas: An Enhanced Cubic Mapping Scheme for Video\/Image-Based Virtual-Reality Scenes.

PixelSynth: Generating a 3D-Consistent Experience from a Single Image

Augmented Efficiency: Reducing Memory Footprint and Accelerating Inference for 3D Semantic Segmentation through Hybrid Vision

UniVision: A Unified Framework for Vision-Centric 3D Perception

A voxelized point clouds representation for object classification and segmentation on 3D data

Empir3D : A Framework for Multi-Dimensional Point Cloud Assessment

VoxelEmbed: 3D Instance Segmentation and Tracking with Voxel Embedding based Deep Learning

Dirty Pixels: Towards End-to-End Image Processing and Perception

Three-dimensional reconstruction of an object from a single image using deep convolutional neural networks

MetaPix: A Data-Centric AI Development Platform for Efficient Management and Utilization of Unstructured Computer Vision Data

PixelGen: Rethinking Embedded Camera Systems

Super Images -- A New 2D Perspective on 3D Medical Imaging Analysis