Abstract:The astounding development of optical sensing imaging technology, coupled with the impressive improvements in machine learning algorithms, has increased our ability to understand and extract information from scenic events. In most cases, Convolution neural networks (CNNs) are largely adopted to infer knowledge due to their surprising success in automation, surveillance, and many other application domains. However, the convolution operations' overwhelming computation demand has somewhat limited their use in remote sensing edge devices. In these platforms, real-time processing remains a challenging task due to the tight constraints on resources and power. Here, the transfer and processing of non-relevant image pixels act as a bottleneck on the entire system. It is possible to overcome this bottleneck by exploiting the high bandwidth available at the sensor interface by designing a CNN inference architecture near the sensor. This paper presents an attention-based pixel processing architecture to facilitate the CNN inference near the image sensor. We propose an efficient computation method to reduce the dynamic power by decreasing the overall computation of the convolution operations. The proposed method reduces redundancies by using a hierarchical optimization approach. The approach minimizes power consumption for convolution operations by exploiting the Spatio-temporal redundancies found in the incoming feature maps and performs computations only on selected regions based on their relevance score. The proposed design addresses problems related to the mapping of computations onto an array of processing elements (PEs) and introduces a suitable network structure for communication. The PEs are highly optimized to provide low latency and power for CNN applications. While designing the model, we exploit the concepts of biological vision systems to reduce computation and energy. We prototype the model in a Virtex UltraScale+ FPGA and implement it in Application Specific Integrated Circuit (ASIC) using the TSMC 90nm technology library. The results suggest that the proposed architecture significantly reduces dynamic power consumption and achieves high-speed up surpassing existing embedded processors' computational capabilities.

Low-power In-pixel Computing with Current-modulated Switched Capacitors

A Reconfigurable Convolution-in-Pixel CMOS Image Sensor Architecture

6.9 A 0.35V 0.367TOPS/W Image Sensor with 3-Layer Optical-Electronic Hybrid Convolutional Neural Network

Data-Driven Pixel Control: Challenges and Prospects

A 5.9μw Ultra-Low-Power Dual-Resolution CIS Chip of Sensing-with-Computing for Always-on Intelligent Visual Devices

Utilizing Direct Photocurrent Computation and 2D Kernel Scheduling to Improve In-Sensor-Processing Efficiency

Selfputing: A 0.57 Μw @ 15 Fps Vision Chip with Self-powered In-Pixel Computing and In-Memory Computing for Visual Perception on the Edge

Voltage-Controlled Magnetic Tunnel Junction based ADC-less Global Shutter Processing-in-Pixel for Extreme-Edge Intelligence

An Always-on 4× Compressive VGA CMOS Imager with 51pj/pixel and >32db PSNR

Towards an Efficient CNN Inference Architecture Enabling In-Sensor Processing

A 51-Pj/pixel 33.7-Db PSNR 4× Compressive CMOS Image Sensor with Column-Parallel Single-Shot Compressive Sensing

A 2.17μw@120fps Ultra-Low-Power Dual-Mode CMOS Image Sensor with Senputing Architecture

A low-noise and low-area-consumption power management dedicated to CMOS pixel sensors for high energy physics experiments

Low-power In-Pixel Buffer Circuit for Smart Image Sensor

Processing Near Sensor Architecture in Mixed-Signal Domain with CMOS Image Sensor of Convolutional-Kernel-Readout Method

A closed-loop, all-electronic pixel-wise adaptive imaging system for high dynamic range videography

CMOS Image Sensor Data-Readout Method for Convolutional Operations with Processing Near Sensor Architecture.

Smart Pixels: In-pixel AI for on-sensor data filtering

A Low Power Dissipation High-Speed CMOS Image Sensor with Column-Parallel Sigma-Delta ADCs

Hardware-Algorithm Co-design Enabling Processing-in-Pixel-in-Memory (P2M) for Neuromorphic Vision Sensors

Image Acquisition System Using On Sensor Compressed Sampling Technique