Abstract:The advent of big data and the Internet of Things has created urgent demands for in-sensor computing hardware with multimodal perception that can effectively resolve the inefficiency, high latency, and excessive energy consumption challenges faced by conventional sensory systems. Here, a simple-structured optoelectronic synaptic device of In2O3SnO2/Nb:SrTiO3 (ITO/NSTO) heterostructure is proposed, which vividly demonstrates in-sensor computing and multimodal perception capabilities. First, with the ingenious synaptic responses of the device under both optical and electrical stimuli, a multimodal in-sensor neuromorphic computing system capable of concurrently perceiving and processing visual and auditory information is constructed. Using this multimodal system to perform a human emotion recognition task, the misjudgment arising from single-modal cognition can thereby be effectively avoided. Second, utilizing the integrated sensing and processing functions, along with the dynamic memories of the device array, a neuromorphic vision system is implemented for real-time monitoring of moving vehicles, which displays high recognition efficiency and accuracy. This research not only provides an optoelectronic synaptic device that is low-cost and easy to mass-produce, but also paves the way for next-generation in-sensor computing that can efficiently process multimodal signals. The separated sensing, storing, and computing modules along with single-mode perception in the conventional sensory system cause time-delay, massive energy consumption, as well as incomprehensive cognition issues. Here a neuromorphic device capable of performing multimodal perception and in-sensor computation within a single device is proposed, enabling a novel multimodal sensory system that is efficient, low-energy, and has minimal time latency. image

Senputing: An Ultra-Low-Power Always-On Vision Perception Chip Featuring the Deep Fusion of Sensing and Computing

A Reconfigurable Convolution-in-Pixel CMOS Image Sensor Architecture

A 5.9μw Ultra-Low-Power Dual-Resolution CIS Chip of Sensing-with-Computing for Always-on Intelligent Visual Devices

Breaking the Energy-Efficiency Barriers for Smart Sensing Applications with “sensing with Computing” Architectures

A 4.57 Μw@120fps Vision System of Sensing with Computing for BNN-Based Perception Applications

A 1.8mW Perception Chip with Near-Sensor Processing Scheme for Low-Power AIoT Applications

A 2.17μw@120fps Ultra-Low-Power Dual-Mode CMOS Image Sensor with Senputing Architecture

Selfputing: A 0.57 Μw @ 15 Fps Vision Chip with Self-powered In-Pixel Computing and In-Memory Computing for Visual Perception on the Edge

An Analog-Memoryless Near Sensor Computing Architecture for Always-On Intelligent Perception Applications

MACSen: A Processing-In-Sensor Architecture Integrating MAC Operations Into Image Sensor for Ultra-Low-Power BNN-Based Intelligent Visual Perception

Speck: A Smart event-based Vision Sensor with a low latency 327K Neuron Convolutional Neuronal Network Processing Pipeline

Processing Near Sensor Architecture in Mixed-Signal Domain with CMOS Image Sensor of Convolutional-Kernel-Readout Method

Ultra-low Power In-Sensor Neuronal Computing with Oscillatory Retinal Neurons for Frequency-Multiplexed, Parallel Machine Vision

Ultra-low-power-consuming Liquid-Water-based Optoelectronic Computing Chip

Vision Perception Unit: Next-Generation Smart CMOS Image Sensor

An Ultra-low Power TinyML System for Real-time Visual Processing at Edge

Multimodal In-Sensor Computing Implemented by Easily-Fabricated Oxide-Heterojunction Optoelectronic Synapses

Ultralow Power In-Sensor Neuronal Computing with Oscillatory Retinal Neurons for Frequency-Multiplexed, Parallel Machine Vision

A 1000 Fps Vision Chip Based on a Dynamically Reconfigurable Hybrid Architecture Comprising a PE Array Processor and Self-Organizing Map Neural Network.

Millimeter-Scale Ultra-Low-Power Imaging System for Intelligent Edge Monitoring

A vision chip with complementary pathways for open-world sensing