Abstract:Traditional computer-vision technology that relies on image sensors coupled with cloud processing or on-chip Artificial Intelligence (AI) processors have encountered significant challenges in terms of power consumption, delays arising from data transmission, and/or memory access. In-sensor and near-sensor computing have been reported to solve this issue by applying pixel or array level feature extraction [2–6]. In [2, 4], capacitors are utilized for analog-domain Haar filtering to reduce the processing power consumption, but sacrificing the Fill Factor (FF) due to the use of the capacitor array [2], or complicated pixel-level logic [4]. An image sensor in introduced in [3] with Hog feature-based object detection, achieving both low power and high accuracy for the detection of up to three object classes. However, it does not work on more complex tasks. Convolutional Neural Networks (CNNs) [5, 6] offer enhanced accuracy in practical applications, but increase the power consumption and circuit complexity. Optical-domain processing [7] is capable of parallel information processing through an optical architecture based on free space. However, in [7] this is based on diffraction and requires coherent light as input, which constrains the practicality for edge computing. This paper presents a Pulse Width Modulation (PWM) pixel-based image sensor array that integrates an optical-electronic hybrid 3-layer convolutional processing unit. The first layer of the convolution is performed in the optical domain without introducing extra power consumption. The integration of optical convolution with electrical convolution offers the potential to leverage the energy-efficient advantages of optical convolution, while simultaneously providing the essential programmability needed to address multiple use-cases through reprogramming within a single device. A dual-mode Processing Element (PE) is proposed enables both acquisition mode and calculation mode. The integration of a sum/differential dual-counter design and the incorporation of two different pixel reset methods are employed in the co-design of PWM pixels and PEs to effectively mitigate computational power consumption.

Non-imaging single-pixel sensing with optimized binary modulation

A Reconfigurable Convolution-in-Pixel CMOS Image Sensor Architecture

Single-pixel Multispectral Imaging Based on Macropixel Segmentation Method

Weighted sampling-adaptive single-pixel sensing

Spatial-spectral Encoding and Dictionary Optimization in Compressive Single-Pixel Hyperspectral Imaging Based on Mutual Coherence Minimization

Single-pixel Imaging Based on Compressive Sensing with Spectral-Domain Optical Mixing

Compressive Single-Pixel Spectral Imaging with Spatial-Spectral Modulation Optimization Via Coherence Minimization

Efficient single-pixel multispectral imaging via non-mechanical spatio-spectral modulation

Multispectral single pixel imaging based on spatial and spectral dimension decoupling

Feature Single-Pixel Imaging: What You See is What You Want

Noise-resilient single-pixel compressive sensing with single photon counting

Photon-limited non-imaging object detection and classification based on single-pixel imaging system

Single-Pixel Imaging Based on Deep Learning Enhanced Singular Value Decomposition

Resource-Saving and High-Robustness Image Sensing Based on Binary Optical Computing

One-dimensional modulation single-pixel imaging: Exceeding spatial light modulator resolution

6.9 A 0.35V 0.367TOPS/W Image Sensor with 3-Layer Optical-Electronic Hybrid Convolutional Neural Network

Single-pixel imaging based on deep learning

Super Sub-Nyquist Single-Pixel Imaging by Total Variation Ascending Ordering of the Hadamard Basis

High-speed Real-Time Single-Pixel Microscopy Based on Fourier Sampling

Single-pixel Non-Imaging Object Recognition by Means of Fourier Spectrum Acquisition

Single pixel imaging via unsupervised deep compressive sensing with collaborative sparsity in discretized feature space