Abstract:Sensing and understanding large-scale dynamic scenes require a high-performance imaging system. Conventional imaging systems pursue higher capability by simply increasing the pixel resolution via stitching cameras at the expense of a bulky system. Moreover, they strictly follow the feedforward pathway: That is, their pixel-level sensing is independent of semantic understanding. Differently, a human visual system owns superiority with both feedforward and feedback pathways: The feedforward pathway extracts object representation (referred to as memory engram) from visual inputs, while, in the feedback pathway, the associated engram is reactivated to generate hypotheses about an object. Inspired by this, we propose a dual-pathway imaging mechanism, called engram-driven videography. We start by abstracting the holistic representation of the scene, which is associated bidirectionally with local details, driven by an instance-level engram. Technically, the entire system works by alternating between the excitation–inhibition and association states. In the former state, pixel-level details become dynamically consolidated or inhibited to strengthen the instance-level engram. In the association state, the spatially and temporally consistent content becomes synthesized driven by its engram for outstanding videography quality of future scenes. The association state serves as the imaging of future scenes by synthesizing spatially and temporally consistent content driven by its engram. Results of extensive simulations and experiments demonstrate that the proposed system revolutionizes the conventional videography paradigm and shows great potential for videography of large-scale scenes with multi-objects.

Pixel-wise Optical Encoder Driven by Video Prediction: Towards Single-Shot High Dynamic Range Moving Object Recognition

A novel high dynamic range microscopic video system based on REC.2020

Pixel-wise Exposure Control for Single-Shot HDR Imaging: A Joint Optimization Approach

Pix2HDR -- A pixel-wise acquisition and deep learning-based synthesis approach for high-speed HDR videos

InfPose: Real-Time Infrared Multi-Human Pose Estimation for Edge Devices Based on Encoder-Decoder CNN Architecture

A closed-loop all-electronic pixel-wise adaptive imaging system for high dynamic range video

Automatic exposure strategy network for robust visual odometry in environments with high dynamic range

Learning Rank-1 Diffractive Optics for Single-Shot High Dynamic Range Imaging

Visual Odometry in HDR Environments by Using Spatially Varying Exposure Camera

EventHDR: from Event to High-Speed HDR Videos and Beyond

Robust High Dynamic Range (HDR) Imaging with Complex Motion and Parallax.

Simultaneous encoder for high-dynamic-range and low-dynamic-range video

Deep Joint Demosaicing and High Dynamic Range Imaging Within a Single Shot

A closed-loop, all-electronic pixel-wise adaptive imaging system for high dynamic range videography

HDR Video Reconstruction with Tri-Exposure Quad-Bayer Sensors

Camera Attributes Control for Visual Odometry With Motion Blur Awareness

Engram-Driven Videography

Lightweight High-Speed Photography Built on Coded Exposure and Implicit Neural Representation of Videos

Event-based Asynchronous HDR Imaging by Temporal Incident Light Modulation

Deep Optics for Video Snapshot Compressive Imaging

Plug-and-Play Algorithms for Dynamic Non-line-of-sight Imaging