Computational event-driven vision sensors for in-sensor spiking neural networks

Yue Zhou,Jiawei Fu,Zirui Chen,Fuwei Zhuge,Yasai Wang,Jianmin Yan,Sijie Ma,Lin Xu,Huanmei Yuan,Mansun Chan,Xiangshui Miao,Yuhui He,Yang Chai

DOI: https://doi.org/10.1038/s41928-023-01055-2

IF: 33.255

2023-11-13

Nature Electronics

Abstract:Neuromorphic event-based image sensors capture only the dynamic motion in a scene, which is then transferred to computation units for motion recognition. This approach, however, leads to time latency and can be power consuming. Here we report computational event-driven vision sensors that capture and directly convert dynamic motion into programmable, sparse and informative spiking signals. The sensors can be used to form a spiking neural network for motion recognition. Each individual vision sensor consists of two parallel photodiodes with opposite polarities and has a temporal resolution of 5 μs. In response to changes in light intensity, the sensors generate spiking signals with different amplitudes and polarities by electrically programming their individual photoresponsivity. The non-volatile and multilevel photoresponsivity of the vision sensors can emulate synaptic weights and can be used to create an in-sensor spiking neural network. Our computational event-driven vision sensor approach eliminates redundant data during the sensing process, as well as the need for data transfer between sensors and computation units.

engineering, electrical & electronic

What problem does this paper attempt to address?

This paper focuses on solving the redundant data and time delay issues in traditional frame-based image sensors when processing dynamic visual information. Although existing event-driven visual sensors can reduce redundant data, they still need to transmit the captured information to the processing unit, which consumes energy and introduces delay. The paper presents a computationally event-driven visual sensor that can directly convert dynamic motion into programmable, sparse, and information-rich spike signals, forming a spike neural network for motion recognition within the sensor. Each sensor consists of two photodiodes with opposite polarities, with a high time resolution of 5 microseconds. When there is a change in light intensity, the sensor generates spike signals with different amplitudes and polarities, and its non-volatility and multi-level light response can simulate synaptic weights to implement a spike neural network within the sensor. This approach eliminates redundant data during the sensing process, reduces the need for data transmission between the sensor and the computing unit, and improves time and energy efficiency.