Data-Driven Pixel Control: Challenges and Prospects

Saurabh Farkya,Zachary Alan Daniels,Aswin Raghavan,Gooitzen van der Wal,Michael Isnardi,Michael Piacentino,David Zhang

2024-08-09

Abstract:Recent advancements in sensors have led to high resolution and high data throughput at the pixel level. Simultaneously, the adoption of increasingly large (deep) neural networks (NNs) has lead to significant progress in computer vision. Currently, visual intelligence comes at increasingly high computational complexity, energy, and latency. We study a data-driven system that combines dynamic sensing at the pixel level with computer vision analytics at the video level and propose a feedback control loop to minimize data movement between the sensor front-end and computational back-end without compromising detection and tracking precision. Our contributions are threefold: (1) We introduce anticipatory attention and show that it leads to high precision prediction with sparse activation of pixels; (2) Leveraging the feedback control, we show that the dimensionality of learned feature vectors can be significantly reduced with increased sparsity; and (3) We emulate analog design choices (such as varying RGB or Bayer pixel format and analog noise) and study their impact on the key metrics of the data-driven system. Comparative analysis with traditional pixel and deep learning models shows significant performance enhancements. Our system achieves a 10X reduction in bandwidth and a 15-30X improvement in Energy-Delay Product (EDP) when activating only 30% of pixels, with a minor reduction in object detection and tracking precision. Based on analog emulation, our system can achieve a throughput of 205 megapixels/sec (MP/s) with a power consumption of only 110 mW per MP, i.e., a theoretical improvement of ~30X in EDP.

Computer Vision and Pattern Recognition,Artificial Intelligence,Systems and Control

What problem does this paper attempt to address?

The paper aims to address the issues of high bandwidth, high energy consumption, and high latency faced by high-resolution sensors when performing real-time complex scene analysis on edge devices. The authors propose a data-driven approach that combines dynamic pixel-level perception with computer vision analysis and introduces a feedback control loop to minimize data transmission between the sensor frontend and the computing backend without sacrificing detection and tracking accuracy. Specifically, the contributions of the paper include:

1. **Introduction of a Predictive Attention Mechanism**: This mechanism enables high-precision prediction by sparsely activating pixels.
2. **Utilization of Feedback Control to Reduce Feature Vector Dimensions**: By increasing sparsity, the dimensionality of the feature vectors is significantly reduced.
3. **Simulation of the Impact of Analog Design Choices**: The study investigates the impact of different pixel formats (such as RGB or Bayer) and analog noise on key system metrics.

Experimental results show that the system can reduce bandwidth by 10 times while activating only 30% of the pixels, and improve the energy-delay product (EDP) by 15 to 30 times. Additionally, the paper explores the effects of different input types (RGB or Bayer), pixel activation ratios, feature compression, and noise types on detection and multi-object tracking performance, and analyzes the impact of each component on the predictive attention mechanism. Overall, this work demonstrates how intelligent dynamic predictive perception combined with novel energy-efficient hardware can effectively reduce data transmission and improve processing efficiency.