Abstract:Traditional computer-vision technology that relies on image sensors coupled with cloud processing or on-chip Artificial Intelligence (AI) processors have encountered significant challenges in terms of power consumption, delays arising from data transmission, and/or memory access. In-sensor and near-sensor computing have been reported to solve this issue by applying pixel or array level feature extraction [2–6]. In [2, 4], capacitors are utilized for analog-domain Haar filtering to reduce the processing power consumption, but sacrificing the Fill Factor (FF) due to the use of the capacitor array [2], or complicated pixel-level logic [4]. An image sensor in introduced in [3] with Hog feature-based object detection, achieving both low power and high accuracy for the detection of up to three object classes. However, it does not work on more complex tasks. Convolutional Neural Networks (CNNs) [5, 6] offer enhanced accuracy in practical applications, but increase the power consumption and circuit complexity. Optical-domain processing [7] is capable of parallel information processing through an optical architecture based on free space. However, in [7] this is based on diffraction and requires coherent light as input, which constrains the practicality for edge computing. This paper presents a Pulse Width Modulation (PWM) pixel-based image sensor array that integrates an optical-electronic hybrid 3-layer convolutional processing unit. The first layer of the convolution is performed in the optical domain without introducing extra power consumption. The integration of optical convolution with electrical convolution offers the potential to leverage the energy-efficient advantages of optical convolution, while simultaneously providing the essential programmability needed to address multiple use-cases through reprogramming within a single device. A dual-mode Processing Element (PE) is proposed enables both acquisition mode and calculation mode. The integration of a sum/differential dual-counter design and the incorporation of two different pixel reset methods are employed in the co-design of PWM pixels and PEs to effectively mitigate computational power consumption.

A 1000fps@360,000pixels Mixed-Signal Sensing with Computing Macro Featuring Analog Compression and Maximum Parallelism for Objective Detection Tasks

A Reconfigurable Convolution-in-Pixel CMOS Image Sensor Architecture

Multiframe-integrated, In-Sensor Computing Using Persistent Photoconductivity

A 1000 Fps Vision Chip Based on a Dynamically Reconfigurable Hybrid Architecture Comprising a PE Array Processor and Self-Organizing Map Neural Network.

A 4.57 Μw@120fps Vision System of Sensing with Computing for BNN-Based Perception Applications

A Real-Time 17-Scale Object Detection Accelerator with Adaptive 2000-Stage Classification in 65nm CMOS.

An Encoded Photodetector Array for High-frame-rate Compressed Sensing and Computational Imaging

MACSen: A Processing-In-Sensor Architecture Integrating MAC Operations Into Image Sensor for Ultra-Low-Power BNN-Based Intelligent Visual Perception

Ultra-low Power In-Sensor Neuronal Computing with Oscillatory Retinal Neurons for Frequency-Multiplexed, Parallel Machine Vision

7.3 A 1000fps Vision Chip Based on a Dynamically Reconfigurable Hybrid Architecture Comprising a PE Array and Self-Organizing Map Neural Network

An Analog-Memoryless Near Sensor Computing Architecture for Always-On Intelligent Perception Applications

6.9 A 0.35V 0.367TOPS/W Image Sensor with 3-Layer Optical-Electronic Hybrid Convolutional Neural Network

Ultralow Power In-Sensor Neuronal Computing with Oscillatory Retinal Neurons for Frequency-Multiplexed, Parallel Machine Vision

Real-time Motion Detection with High-speed Vision and Local Parallel Processing

A 68 mw 2.2 Tops/w low bit-width and multiplierless DCNN object detection processor for visually impaired people

P2M-DeTrack: Processing-in-Pixel-in-Memory for Energy-efficient and Real-Time Multi-Object Detection and Tracking

A 58.6mW Real-Time Programmable Object Detector with Multi-Scale Multi-Object Support Using Deformable Parts Model on 1920x1080 Video at 30fps

AnalogNet: Convolutional Neural Network Inference on Analog Focal Plane Sensor Processors

High-speed Real-Time Single-Pixel Microscopy Based on Fourier Sampling

A Highly Parallelized Signal and Image Recognition Method Based on Spatial Optics

NS-Engine: Near-Sensor Neural Network Engine with SRAM-Based Compute-in-Memory Macro