Abstract:Traditional computer-vision technology that relies on image sensors coupled with cloud processing or on-chip Artificial Intelligence (AI) processors have encountered significant challenges in terms of power consumption, delays arising from data transmission, and/or memory access. In-sensor and near-sensor computing have been reported to solve this issue by applying pixel or array level feature extraction [2–6]. In [2, 4], capacitors are utilized for analog-domain Haar filtering to reduce the processing power consumption, but sacrificing the Fill Factor (FF) due to the use of the capacitor array [2], or complicated pixel-level logic [4]. An image sensor in introduced in [3] with Hog feature-based object detection, achieving both low power and high accuracy for the detection of up to three object classes. However, it does not work on more complex tasks. Convolutional Neural Networks (CNNs) [5, 6] offer enhanced accuracy in practical applications, but increase the power consumption and circuit complexity. Optical-domain processing [7] is capable of parallel information processing through an optical architecture based on free space. However, in [7] this is based on diffraction and requires coherent light as input, which constrains the practicality for edge computing. This paper presents a Pulse Width Modulation (PWM) pixel-based image sensor array that integrates an optical-electronic hybrid 3-layer convolutional processing unit. The first layer of the convolution is performed in the optical domain without introducing extra power consumption. The integration of optical convolution with electrical convolution offers the potential to leverage the energy-efficient advantages of optical convolution, while simultaneously providing the essential programmability needed to address multiple use-cases through reprogramming within a single device. A dual-mode Processing Element (PE) is proposed enables both acquisition mode and calculation mode. The integration of a sum/differential dual-counter design and the incorporation of two different pixel reset methods are employed in the co-design of PWM pixels and PEs to effectively mitigate computational power consumption.

Optical efficiency of image sensor pixels

A Reconfigurable Convolution-in-Pixel CMOS Image Sensor Architecture

A Cmos Image Sensor For Monochromatic Spectrum Imaging

Accurate Optical Optimization of Light-Emitting Diodes with Consideration of Coupling Between Purcell Factor and Transmittance Coefficient

Ensquared Energy and Optical Centroid Efficiency in Optical Sensors: Part 1, Theory

Ensquared Energy and Optical Centroid Efficiency in Optical Sensors: Part 2, Primary Aberrations

High Performance CMOS Image Sensor for Low Light Imaging

Characterization Study of an Intensified Complementary Metal-Oxide-semiconductor Active Pixel Sensor.

CMOS image sensor with micro-nano holes to improve NIR optical efficiency: micro-holes on top surface vs on bottom

A Review of Optical Sensors in CMOS

Performance analysis of a color CMOS photogate image sensor

Pixel design optimization of CMOS image sensor with large dynamic range and high charge transfer efficiency

FDTD-based optical simulations methodology for CMOS image sensors pixels architecture and process optimization

A Monolithically Integrated 640 × 512 CMOS-Perovskite Image Sensor

CMOS-Compatible Optoelectronic Imagers

PI-12 : Comparison of Modern CCD and CMOS Image Sensor Technologies and Systems for Low Resolution Imaging 1

A 51-Pj/pixel 33.7-Db PSNR 4× Compressive CMOS Image Sensor with Column-Parallel Single-Shot Compressive Sensing

The Effect of Pixel Design and Operation Conditions on Linear Output Range of 4T CMOS Image Sensors

Single Micro-hole per Pixel for Thin Ge-on-Si Image Sensor with Enhanced Sensitivity upto 1700 nm

6.9 A 0.35V 0.367TOPS/W Image Sensor with 3-Layer Optical-Electronic Hybrid Convolutional Neural Network

A Dual-Mode CMOS Image Sensor Based on In-Pixel Frame Differencing