Abstract:The astounding development of optical sensing imaging technology, coupled with the impressive improvements in machine learning algorithms, has increased our ability to understand and extract information from scenic events. In most cases, Convolution neural networks (CNNs) are largely adopted to infer knowledge due to their surprising success in automation, surveillance, and many other application domains. However, the convolution operations' overwhelming computation demand has somewhat limited their use in remote sensing edge devices. In these platforms, real-time processing remains a challenging task due to the tight constraints on resources and power. Here, the transfer and processing of non-relevant image pixels act as a bottleneck on the entire system. It is possible to overcome this bottleneck by exploiting the high bandwidth available at the sensor interface by designing a CNN inference architecture near the sensor. This paper presents an attention-based pixel processing architecture to facilitate the CNN inference near the image sensor. We propose an efficient computation method to reduce the dynamic power by decreasing the overall computation of the convolution operations. The proposed method reduces redundancies by using a hierarchical optimization approach. The approach minimizes power consumption for convolution operations by exploiting the Spatio-temporal redundancies found in the incoming feature maps and performs computations only on selected regions based on their relevance score. The proposed design addresses problems related to the mapping of computations onto an array of processing elements (PEs) and introduces a suitable network structure for communication. The PEs are highly optimized to provide low latency and power for CNN applications. While designing the model, we exploit the concepts of biological vision systems to reduce computation and energy. We prototype the model in a Virtex UltraScale+ FPGA and implement it in Application Specific Integrated Circuit (ASIC) using the TSMC 90nm technology library. The results suggest that the proposed architecture significantly reduces dynamic power consumption and achieves high-speed up surpassing existing embedded processors' computational capabilities.

Optical Convolution Based Computational Method for Low-Power Image Processing

A Reconfigurable Convolution-in-Pixel CMOS Image Sensor Architecture

Survey on Energy-Efficient Deep Neural Networks for Computer Vision

Direct Optical Convolution Computing Based on Arrayed Waveguide Grating Router

Spectral Convolutional Neural Network Chip for In-sensor Edge Computing of Incoherent Natural Light

Optical Convolutional Neural Networks -- Combining Silicon Photonics and Fourier Optics for Computer Vision

Optical-electronic Hybrid Fourier Convolutional Neural Network Based on Super-Pixel Complex-Valued Modulation

All-Optical Computing Based on Convolutional Neural Networks

Sophisticated Deep Learning with On-Chip Optical Diffractive Tensor Processing

Towards an Efficient CNN Inference Architecture Enabling In-Sensor Processing

6.9 A 0.35V 0.367TOPS/W Image Sensor with 3-Layer Optical-Electronic Hybrid Convolutional Neural Network

Compressed Meta-Optical Encoder for Image Classification

11 TeraFLOPs per second photonic convolutional accelerator for deep learning optical neural networks

Accelerating Convolutional Processing by Harnessing Channel Shifts in Arrayed Waveguide Gratings

Compact optical convolution processing unit based on multimode interference

High-accuracy optical convolution unit architecture for convolutional neural networks by cascaded acousto-optical modulator arrays

Diffraction‐Driven Parallel Convolution Processing with Integrated Photonics

Hybrid optical convolutional neural network with convolution kernels trained in the spatial domain

Low power Convolutional Neural Networks on a chip

An Optical Frontend for a Convolutional Neural Network

11 Tera-OPs/s Photonic Convolutional Accelerator and Deep Optical Neural Network Based on an Integrated Kerr Soliton Crystal Microcomb