Abstract:CMOS Image Sensors (CIS) are fundamental to emerging visual computing applications. While conventional CIS are purely imaging devices for capturing images, increasingly CIS integrate processing capabilities such as Deep Neural Network (DNN). Computational CIS expand the architecture design space, but to date no comprehensive energy model exists. This paper proposes CamJ, a detailed energy modeling framework that provides a component-level energy breakdown for computational CIS and is validated against nine recent CIS chips. We use CamJ to demonstrate three use-cases that explore architectural trade-offs including computing in vs. off CIS, 2D vs. 3D-stacked CIS design, and analog vs. digital processing inside CIS. The code of CamJ is available at: <a class="link-external link-https" href="https://github.com/horizon-research/CamJ" rel="external noopener nofollow">this https URL</a>

What problem does this paper attempt to address?

### Problems Addressed by the Paper The paper aims to address the lack of comprehensive tools for system-level energy consumption modeling and architectural exploration of computational CMOS image sensors (CIS). Specifically: 1. **Difference between Traditional CIS and Computational CIS**: - Traditional CIS is only responsible for image capture, while computational CIS integrates processing capabilities, such as signal preprocessing and deep neural networks (DNN). - Integrating computational capabilities expands the architectural design space, but there is currently a lack of detailed energy consumption models. 2. **Need for Energy Consumption Modeling**: - The design of computational CIS needs to consider various factors, including whether computation is performed inside or outside the sensor, 2D vs. 3D stacking designs, analog vs. digital processing, etc. - Designers face many choices and need a tool to evaluate the energy consumption and performance trade-offs of different designs. 3. **Limitations of Existing Tools**: - There is currently no comprehensive tool that can provide component-level energy consumption estimates for computational CIS. - Existing tools mainly focus on the digital domain and neglect energy consumption modeling in the analog domain. ### Solution To address this, the paper proposes CamJ, a detailed energy consumption modeling framework that provides component-level energy breakdown for computational CIS and is validated through nine recent CIS chips. The main contributions of CamJ include: 1. **Comprehensive Energy Consumption Modeling Tool**: - CamJ provides end-to-end optimization from photon intake to semantic results, covering major structures such as pixel sensing, analog processing, and digital processing. - It simplifies the user's workload by describing algorithms and hardware configurations through a declarative interface. 2. **Modeling Methods for Both Analog and Digital Domains**: - In the digital domain, CamJ directly requires users to provide the energy consumption of each processing element (PE) and each memory access. - In the analog domain, CamJ provides default energy consumption models based on classical CIS designs and allows expert users to input custom designs. 3. **Extensive Validation**: - CamJ demonstrates high accuracy and reliability through validation on nine CIS chips with different process nodes, pixel types, memory sizes, and processing forms. 4. **Practical Application Cases**: - The paper showcases the application of CamJ in architectural exploration through three use cases, including the trade-offs of computation inside vs. outside the sensor, comparison of 2D vs. 3D designs, and contrast between analog and digital processing. ### Summary In summary, by proposing CamJ, the paper fills the gap in system-level energy consumption modeling and architectural exploration for computational CIS, providing designers with a powerful tool to make optimal decisions in a complex design space.

CamJ: Enabling System-Level Energy Modeling and Architectural Exploration for In-Sensor Visual Computing

A Reconfigurable Convolution-in-Pixel CMOS Image Sensor Architecture

A Scalable 2T-1Fefet Based Content Addressable Memory Design for Energy Efficient Data Search

A Multimodal Sensing CMOS Imager Based on Dual‐Focus Imaging(Adv. Sci. 14/2023)

Energy-Efficient & Real-Time Computer Vision with Intelligent Skipping via Reconfigurable CMOS Image Sensors

Optical Convolution Based Computational Method for Low-Power Image Processing

Processing Near Sensor Architecture in Mixed-Signal Domain with CMOS Image Sensor of Convolutional-Kernel-Readout Method

A 5.9μw Ultra-Low-Power Dual-Resolution CIS Chip of Sensing-with-Computing for Always-on Intelligent Visual Devices

CMOS Image Sensor Data-Readout Method for Convolutional Operations with Processing Near Sensor Architecture.

An Analog-Memoryless Near Sensor Computing Architecture for Always-On Intelligent Perception Applications

SEE-MCAM: Scalable Multi-bit FeFET Content Addressable Memories for Energy Efficient Associative Search

Reducing the Energy Cost of Inference via In-sensor Information Processing

A 4.57 Μw@120fps Vision System of Sensing with Computing for BNN-Based Perception Applications

Multiframe-integrated, In-Sensor Computing Using Persistent Photoconductivity

Estimating Power, Performance, and Area for On-Sensor Deployment of AR/VR Workloads Using an Analytical Framework

NS-CIM: A Current-Mode Computation-in-Memory Architecture Enabling Near-Sensor Processing for Intelligent IoT Vision Nodes.

Utilizing Direct Photocurrent Computation and 2D Kernel Scheduling to Improve In-Sensor-Processing Efficiency

A 2.17μw@120fps Ultra-Low-Power Dual-Mode CMOS Image Sensor with Senputing Architecture

Design and Simulation of an Application Specific CMOS Image Sensor

Towards an Efficient CNN Inference Architecture Enabling In-Sensor Processing

A 51-Pj/pixel 33.7-Db PSNR 4× Compressive CMOS Image Sensor with Column-Parallel Single-Shot Compressive Sensing