Ultralow‐Power Machine Vision with Self‐Powered Sensor Reservoir

Jie Lao,Mengge Yan,Bobo Tian,Chunli Jiang,Chunhua Luo,Zhuozhuang Xie,Qiuxiang Zhu,Zhiqiang Bao,Ni Zhong,Xiaodong Tang,Linfeng Sun,Guangjian Wu,Jianlu Wang,Hui Peng,Junhao Chu,Chungang Duan

DOI: https://doi.org/10.1002/advs.202106092

IF: 15.1

2022-03-13

Advanced Science

Abstract:Advanced Science, EarlyView.

materials science, multidisciplinary,nanoscience & nanotechnology,chemistry

What problem does this paper attempt to address?

Senputing: An Ultra-Low-Power Always-On Vision Perception Chip Featuring the Deep Fusion of Sensing and Computing

Han Xu,Ningchao Lin,Li Luo,Qi Wei,Runsheng Wang,Cheng Zhuo,Xunzhao Yin,Fei Qiao,Huazhong Yang

DOI: https://doi.org/10.1109/TCSI.2021.3090668

2022-01-01

Abstract:Always-on intelligent visual perception applications are widely deployed in edges in the AIoT era. In order to eliminate power costs of data conversion and transmission, this paper proposes Senputing, an ultra-low-power processing-in-sensor chip that completely fuses sensing and computing together for a BNN-based hierarchical processing system. This chip could operate in two modes. In computation ...
A Multimodal Sensing CMOS Imager Based on Dual‐Focus Imaging(Adv. Sci. 14/2023)

Hao Dong,Xubin Zheng,Chen Cheng,Libin Qian,Yaoxuan Cui,Weiwei Wu,Qingjun Liu,Xing Chen,Yanli Lu,Qing Yang,Fenni Zhang,Di Wang

DOI: https://doi.org/10.1002/advs.202206699

IF: 15.1

2023-01-01

Advanced Science

Abstract:Advanced machine intelligence is empowered not only by the ever-increasing computational capability for information processing but also by sensors for collecting multimodal information from complex environments. However, simply assembling different sensors can result in bulky systems and complex data processing. Herein, it is shown that a complementary metal-oxide-semiconductor (CMOS) imager can be transformed into a compact multimodal sensing platform through dual-focus imaging. By combining lens-based and lensless imaging, visual information, chemicals, temperature, and humidity can be detected with the same chip and output as a single image. As a proof of concept, the sensor is equipped on a micro-vehicle, and multimodal environmental sensing and mapping is demonstrated. A multimodal endoscope is also developed, and simultaneous imaging and chemical profiling along a porcine digestive tract is achieved. The multimodal CMOS imager is compact, versatile, and extensible and can be widely applied in microrobots, in vivo medical apparatuses, and other microdevices.
An Adaptive Power Management Unit for Self-Sustainable Smart Vision Sensor

Haijin Su,Maimaiti Nazhamaitit,Fanghua Zhangt,Xin Hone,Li Luo,Jing Zhang,Qi Wei,Xinghua Yang,Fei Qiao

DOI: https://doi.org/10.1109/newcas58973.2024.10666296

2024-01-01

Abstract:In this paper, a design of a fully integrated adaptive Power Management Unit (PMU) for self-sustainable sensor is presented. A compact image sensor is implemented to harvest the micro ambient solar energy and extract the relevant information as well. By combining PMU and communication block on the sensor, the PMU could efficiently manage harvested energy from focal plane, then power up the communication block for transmitting the processed image results out of image sensor. The system presents a complete solution for self-powered sensing and communication micro system, which is of great potential for the deployment of self-sustainable smart IoT nodes on a large scale. The proposed PMU is designed with TSMC 0.18 mu m CMOS technology. Through the incorporation of a reconfigurable main charge pump and adaptive control strategy, the PMU takes the capability to adjust its topology in response to the variations of light intensity dynamically, which greatly enlarges the input voltage range. By utilizing a cold-start circuit, the PMU is capable of starting up from a 0.08V harvested input voltage without any external kick-off signal. At a light intensity of 8.54kLux, the PMU achieves peak power conversion efficiency of 93.38%. The presented simultaneous energy harvesting and power management approach ensures self-sustainable operation of the entire system.
Low-Power Computer Vision: Status, Challenges, and Opportunities.

Sergei Alyamkin,Matthew Ardi,Alexander C. Berg,Achille Brighton,Bo Chen,Yiran Chen,Hsin-Pai Cheng,Zichen Fan,Chen Feng,Bo Fu,Kent Gauen,Abhinav Goel,Alexander Goncharenko,Xuyang Guo,Soonhoi Ha,Andrew Howard,Xiao Hu,Yuanjun Huang,Jaeyoun Kim,Jong Gook Ko,Alexander Kondratyev,Junhyeok Lee,Seungjae Lee,Suwoong Lee,Zichao Li,Zhiyu Liang,Juzheng Liu,Xin Liu,Yang Lu,Yung-Hsiang Lu,Deeptanshu Malik,Hong Hanh Nguyen,Eunbyung Park,Denis Repin,Liang Shen,Tao Sheng,Fei Sun,David Svitov,George K. Thiruvathukal,Baiwu Zhang,Jingchi Zhang,Xiaopeng Zhang,Shaojie Zhuo,D. Kang

DOI: https://doi.org/10.1109/jetcas.2019.2911899

IF: 5.877

2019-01-01

IEEE Journal on Emerging and Selected Topics in Circuits and Systems

Abstract:Computer vision has achieved impressive progress in recent years. Meanwhile, mobile phones have become the primary computing platforms for millions of people. In addition to mobile phones, many autonomous systems rely on visual data for making decisions, and some of these systems have limited energy (such as unmanned aerial vehicles also called drones and mobile robots). These systems rely on batteries, and energy efficiency is critical. This paper serves the following two main purposes. First, examine the state of the art for low-power solutions to detect objects in images. Since 2015, the IEEE Annual International Low-Power Image Recognition Challenge (LPIRC) has been held to identify the most energy-efficient computer vision solutions. This paper summarizes the 2018 winners’ solutions. Second, suggest directions for research as well as opportunities for low-power computer vision.
Low-Power Computer Vision: Status, Challenges, Opportunities

Sergei Alyamkin,Matthew Ardi,Alexander C. Berg,Achille Brighton,Bo Chen,Yiran Chen,Hsin-Pai Cheng,Zichen Fan,Chen Feng,Bo Fu,Kent Gauen,Abhinav Goel,Alexander Goncharenko,Xuyang Guo,Soonhoi Ha,Andrew Howard,Xiao Hu,Yuanjun Huang,Donghyun Kang,Jaeyoun Kim,Jong Gook Ko,Alexander Kondratyev,Junhyeok Lee,Seungjae Lee,Suwoong Lee,Zichao Li,Zhiyu Liang,Juzheng Liu,Xin Liu,Yang Lu,Yung-Hsiang Lu,Deeptanshu Malik,Hong Hanh Nguyen,Eunbyung Park,Denis Repin,Liang Shen,Tao Sheng,Fei Sun,David Svitov,George K. Thiruvathukal,Baiwu Zhang,Jingchi Zhang,Xiaopeng Zhang,Shaojie Zhuo

DOI: https://doi.org/10.48550/arXiv.1904.07714

2019-04-16

Abstract:Computer vision has achieved impressive progress in recent years. Meanwhile, mobile phones have become the primary computing platforms for millions of people. In addition to mobile phones, many autonomous systems rely on visual data for making decisions and some of these systems have limited energy (such as unmanned aerial vehicles also called drones and mobile robots). These systems rely on batteries and energy efficiency is critical. This article serves two main purposes: (1) Examine the state-of-the-art for low-power solutions to detect objects in images. Since 2015, the IEEE Annual International Low-Power Image Recognition Challenge (LPIRC) has been held to identify the most energy-efficient computer vision solutions. This article summarizes 2018 winners' solutions. (2) Suggest directions for research as well as opportunities for low-power computer vision.

Computer Vision and Pattern Recognition,Artificial Intelligence,Performance
A 5.9μw Ultra-Low-Power Dual-Resolution CIS Chip of Sensing-with-Computing for Always-on Intelligent Visual Devices

Ziwei Li,Han Xu,Li Luo,Qi Wei,Fei Qiao

DOI: https://doi.org/10.1109/iscas51556.2021.9401338

2021-01-01

Abstract:In the intelligent IoT edge devices, power consumption is increasing due to the deployment of high-precision algorithms, which greatly limits the working time of the devices. The power of A/D conversion and data transmission has become the bottleneck of traditional visual system. In this paper, a new sensing-with-computing (Senputing) architecture is proposed to reduce this power bottleneck by combining imaging and BNN 1st-layer feature map computation. This Senputing architecture has two working modes, Normal-Sensor mode and Direct-Photocurrent-Computation mode with different resolutions (128×128 and 32×32). An ultra-low-power CMOS image sensor (CIS) chip with Senputing architecture is proposed to verify the feasibility. Our CIS chip is simulated with 180nm CMOS technology, the power of feature map computation is 5.9μW, and the frame rate is 208fps. The computation efficiency reaches to 8.23TOPs/W, which is 10.1 x higher than previous works.
An Ultra-low Power TinyML System for Real-time Visual Processing at Edge

Kunran Xu,Huawei Zhang,Yishi Li,Yuhao Zhang,Rui Lai,Yi Liu

DOI: https://doi.org/10.1109/tcsii.2023.3239044

2023-01-01

Abstract:Tiny machine learning (TinyML), executing AI workloads on resource and power strictly restricted systems, is an important and challenging topic. This brief firstly presents an extremely tiny backbone to construct high efficiency CNN models for various visual tasks. Then, a specially designed neural co-processor (NCP) is interconnected with MCU to build an ultra-low power TinyML system, which stores all features and weights on chip and completely removes both of latency and power consumption in off-chip memory access. Moreover, an application specific instruction-set is further presented for realizing agile development and rapid deployment. Extensive experiments demonstrate that the proposed TinyML system based on our tiny model, NCP and instruction set yields considerable accuracy and achieves a record ultra-low power of 160mW while implementing object detection and recognition at 30FPS. The demo video is available on https://www.youtube.com/watch?v=mIZPxtJ-9EY.

engineering, electrical & electronic
Bioinspired and Low-Power 2D Machine Vision with Adaptive Machine Learning and Forgetting

Akhil Dodda,Darsith Jayachandran,Shiva Subbulakshmi Radhakrishnan,Andrew Pannone,Yikai Zhang,Nicholas Trainor,Joan M Redwing,Saptarshi Das

DOI: https://doi.org/10.1021/acsnano.2c02906

IF: 17.1

2022-12-27

ACS Nano

Abstract:Natural intelligence has many dimensions, with some of its most important manifestations being tied to learning about the environment and making behavioral changes. In primates, vision plays a critical role in learning. The underlying biological neural networks contain specialized neurons and synapses which not only sense and process visual stimuli but also learn and adapt with remarkable energy efficiency. Forgetting also plays an active role in learning. Mimicking the adaptive neurobiological mechanisms for seeing, learning, and forgetting can, therefore, accelerate the development of artificial intelligence (AI) and bridge the massive energy gap that exists between AI and biological intelligence. Here, we demonstrate a bioinspired machine vision system based on a 2D phototransistor array fabricated from large-area monolayer molybdenum disulfide (MoS2) and integrated with an analog, nonvolatile, and programmable memory gate-stack; this architecture not only enables dynamic learning and relearning from visual stimuli but also offers learning adaptability under noisy illumination conditions at miniscule energy expenditure. In short, our demonstrated "all-in-one" hardware vision platform combines "sensing", "computing", and "storage" to not only overcome the von Neumann bottleneck of conventional complementary metal-oxide-semiconductor (CMOS) technology but also to eliminate the need for peripheral circuits and sensors.
Millimeter-Scale Ultra-Low-Power Imaging System for Intelligent Edge Monitoring

Andrea Bejarano-Carbo,Hyochan An,Kyojin Choo,Shiyu Liu,Qirui Zhang,Dennis Sylvester,David Blaauw,Hun-Seok Kim

DOI: https://doi.org/10.48550/arXiv.2203.04496

2022-06-25

Abstract:Millimeter-scale embedded sensing systems have unique advantages over larger devices as they are able to capture, analyze, store, and transmit data at the source while being unobtrusive and covert. However, area-constrained systems pose several challenges, including a tight energy budget and peak power, limited data storage, costly wireless communication, and physical integration at a miniature scale. This paper proposes a novel 6.7$\times$7$\times$5mm imaging system with deep-learning and image processing capabilities for intelligent edge applications, and is demonstrated in a home-surveillance scenario. The system is implemented by vertically stacking custom ultra-low-power (ULP) ICs and uses techniques such as dynamic behavior-specific power management, hierarchical event detection, and a combination of data compression methods. It demonstrates a new image-correcting neural network that compensates for non-idealities caused by a mm-scale lens and ULP front-end. The system can store 74 frames or offload data wirelessly, consuming 49.6$\mu$W on average for an expected battery lifetime of 7 days.

Signal Processing
Application of Machine Vision Techniques in Low-Cost Devices to Improve Efficiency in Precision Farming

Juan Felipe Jaramillo-Hernández,Vicente Julian,Cedric Marco-Detchart,Jaime Andrés Rincón

DOI: https://doi.org/10.3390/s24030937

IF: 3.9

2024-02-01

Sensors

Abstract:In the context of recent technological advancements driven by distributed work and open-source resources, computer vision stands out as an innovative force, transforming how machines interact with and comprehend the visual world around us. This work conceives, designs, implements, and operates a computer vision and artificial intelligence method for object detection with integrated depth estimation. With applications ranging from autonomous fruit-harvesting systems to phenotyping tasks, the proposed Depth Object Detector (DOD) is trained and evaluated using the Microsoft Common Objects in Context dataset and the MinneApple dataset for object and fruit detection, respectively. The DOD is benchmarked against current state-of-the-art models. The results demonstrate the proposed method's efficiency for operation on embedded systems, with a favorable balance between accuracy and speed, making it well suited for real-time applications on edge devices in the context of the Internet of things.

engineering, electrical & electronic,chemistry, analytical,instruments & instrumentation
NS-MD: Near-Sensor Motion Detection with Energy Harvesting Image Sensor for Always-On Visual Perception.

Maimaiti Nazhamaiti,Han Xu,Zheyu Liu,Yixuan Chen,Qi Wei,Xing Wu,Fei Qiao

DOI: https://doi.org/10.1109/tcsii.2021.3087840

2021-01-01

Abstract:With the rapid development of Internet-of-Things (IoT), self-powered visual perception systems attract lots of attention. This brief presents a near-sensor motion detection architecture for the energy harvesting image sensor targeting always-on monitoring applications. Reconfigurable pixel structures are designed to harvest solar energy from the focal plane and conduct motion detection simultaneously. The near-sensor motion detection architecture combining analog-domain edge detection and digital-domain frame differencing technique is proposed to reduce the power consumption on data conversion significantly. A prototype chip of $32\times 32$ array size is fabricated with the standard $0.18~\mu \text{m}$ CMOS process. The measurement results show that the chip prototype achieves motion detection at a frame rate of 1894 fps consuming $16.43~\mu \text{W}$ power. Furthermore, the chip harvests $1.263~\mu \text{W}$ power under 100kLux illumination, and the analytical computation result shows that self-powered motion detection can be achieved at a frame rate of 58 fps.
An ultrawide field-of-view pinhole compound eye using hemispherical nanowire array for robot vision

Yu Zhou,Zhibo Sun,Yucheng Ding,Zhengnan Yuan,Xiao Qiu,Yang Bryan Cao,Zhu'an Wan,Zhenghao Long,Swapnadeep Poddar,Shivam Kumar,Wenhao Ye,Chak Lam Jonathan Chan,Daquan Zhang,Beitao Ren,Qianpeng Zhang,Hoi-Sing Kwok,Mitch Guijun Li,Zhiyong Fan

DOI: https://doi.org/10.1126/scirobotics.adi8666

2024-05-15

Abstract:Garnering inspiration from biological compound eyes, artificial vision systems boasting a vivid range of diverse visual functional traits have come to the fore recently. However, most of these artificial systems rely on transformable electronics, which suffer from the complexity and constrained geometry of global deformation, as well as potential mismatches between optical and detector units. Here, we present a unique pinhole compound eye that combines a three-dimensionally printed honeycomb optical structure with a hemispherical, all-solid-state, high-density perovskite nanowire photodetector array. The lens-free pinhole structure can be designed and fabricated with an arbitrary layout to match the underlying image sensor. Optical simulations and imaging results matched well with each other and substantiated the key characteristics and capabilities of our system, which include an ultrawide field of view, accurate target positioning, and motion tracking function. We further demonstrate the potential of our unique compound eye for advanced robotic vision by successfully completing a moving target tracking mission.
Ultralow Power In-Sensor Neuronal Computing with Oscillatory Retinal Neurons for Frequency-Multiplexed, Parallel Machine Vision

Ragib Ahsan,Hyun Uk Chae,Seyedeh Atiyeh Abbasi Jalal,Zezhi Wu,Jun Tao,Subrata Das,Hefei Liu,Jiang-Bin Wu,Stephen B Cronin,Han Wang,Constantine Sideris,Rehan Kapadia

DOI: https://doi.org/10.1021/acsnano.4c09055

IF: 17.1

2024-08-27

ACS Nano

Abstract:In-sensor and near-sensor computing architectures enable multiply accumulate operations to be carried out directly at the point of sensing. In-sensor architectures offer dramatic power and speed improvements over traditional von Neumann architectures by eliminating multiple analog-to-digital conversions, data storage, and data movement operations. Current in-sensor processing approaches rely on tunable sensors or additional weighting elements to perform linear functions such as multiply accumulate operations as the sensor acquires data. This work implements in-sensor computing with an oscillatory retinal neuron device that converts incident optical signals into voltage oscillations. A computing scheme is introduced based on the frequency shift of coupled oscillators that enables parallel, frequency multiplexed, nonlinear operations on the inputs. An experimentally implemented 3 × 3 focal plane array of coupled neurons shows that functions approximating edge detection, thresholding, and segmentation occur in parallel. An example of inference on handwritten digits from the MNIST database is also experimentally demonstrated with a 3 × 3 array of coupled neurons feeding into a single hidden layer neural network, approximating a liquid-state machine. Finally, the equivalent energy consumption to carry out image processing operations, including peripherals such as the Fourier transform circuits, is projected to be <20 fJ/OP, possibly reaching as low as 15 aJ/OP.
Ultra-low Power In-Sensor Neuronal Computing with Oscillatory Retinal Neurons for Frequency-Multiplexed, Parallel Machine Vision

Ragib Ahsan,Hyun Uk Chae,Seyedeh Atiyeh Abbasi Jalal,Zezhi Wu,Jun Tao,Subrata Das,Hefei Liu,Jiang-Bin Wu,Stephen B. Cronin,Han Wang,Constantine Sideris,Rehan Kapadia

DOI: https://doi.org/10.1021/acsnano.4c09055

IF: 17.1

2024-01-01

ACS Nano

Abstract:In-sensor and near-sensor computing architectures enable multiply accumulate operations to be carried out directly at the point of sensing. In-sensor architectures offer dramatic power and speed improvements over traditional von Neumann architectures by eliminating multiple analog-to-digital conversions, data storage, and data movement operations. Current in-sensor processing approaches rely on tunable sensors or additional weighting elements to perform linear functions such as multiply accumulate operations as the sensor acquires data. This work implements in-sensor computing with an oscillatory retinal neuron device that converts incident optical signals into voltage oscillations. A computing scheme is introduced based on the frequency shift of coupled oscillators that enables parallel, frequency multiplexed, nonlinear operations on the inputs. An experimentally implemented 3 × 3 focal plane array of coupled neurons shows that functions approximating edge detection, thresholding, and segmentation occur in parallel. An example of inference on handwritten digits from the MNIST database is also experimentally demonstrated with a 3 × 3 array of coupled neurons feeding into a single hidden layer neural network, approximating a liquid-state machine. Finally, the equivalent energy consumption to carry out image processing operations, including peripherals such as the Fourier transform circuits, is projected to be <20 fJ/OP, possibly reaching as low as 15 aJ/OP.
1000× Faster Camera and Machine Vision with Ordinary Devices

Tiejun Huang,Yajing Zheng,Zhaofei Yu,Rui Chen,Yuan Li,Ruiqin Xiong,Lei Ma,Junwei Zhao,Siwei Dong,Lin Zhu,Jianing Li,Shanshan Jia,Yihua Fu,Boxin Shi,Si Wu,Yonghong Tian

DOI: https://doi.org/10.1016/j.eng.2022.01.012

IF: 12.834

2022-04-01

Engineering

Abstract:In digital cameras, we find a major limitation: the image and video form inherited from a film camera obstructs it from capturing the rapidly changing photonic world. Here, we present vidar, a bit sequence array where each bit represents whether the accumulation of photons has reached a threshold, to record and reconstruct the scene radiance at any moment. By employing only consumer-level complementary metal-oxide-semiconductor (CMOS) sensors and integrated circuits, we have developed a vidar camera that is 1000× faster than conventional cameras. By treating vidar as spike trains in biological vision, we have further developed a spiking neural network (SNN)-based machine vision system that combines the speed of the machine and the mechanism of biological vision, achieving high-speed object detection and tracking 1000× faster than human vision. We demonstrate the utility of the vidar camera and the super vision system in an assistant referee and target pointing system. Our study is expected to fundamentally revolutionize the image and video concepts and related industries, including photography, movies, and visual media, and to unseal a new SNN-enabled speed-free machine vision era.

engineering, multidisciplinary
1000x Faster Camera and Machine Vision with Ordinary Devices

Tiejun Huang,Yajing Zheng,Zhaofei Yu,Rui Chen,Yuan Li,Ruiqin Xiong,Lei Ma,Junwei Zhao,Siwei Dong,Lin Zhu,Jianing Li,Shanshan Jia,Yihua Fu,Boxin Shi,Si Wu,Yonghong Tian

DOI: https://doi.org/10.1016/j.eng.2022.01.012

2022-01-24

Abstract:In digital cameras, we find a major limitation: the image and video form inherited from a film camera obstructs it from capturing the rapidly changing photonic world. Here, we present vidar, a bit sequence array where each bit represents whether the accumulation of photons has reached a threshold, to record and reconstruct the scene radiance at any moment. By employing only consumer-level CMOS sensors and integrated circuits, we have developed a vidar camera that is 1,000x faster than conventional cameras. By treating vidar as spike trains in biological vision, we have further developed a spiking neural network-based machine vision system that combines the speed of the machine and the mechanism of biological vision, achieving high-speed object detection and tracking 1,000x faster than human vision. We demonstrate the utility of the vidar camera and the super vision system in an assistant referee and target pointing system. Our study is expected to fundamentally revolutionize the image and video concepts and related industries, including photography, movies, and visual media, and to unseal a new spiking neural network-enabled speed-free machine vision era.

Computer Vision and Pattern Recognition
Birdlike broadband neuromorphic visual sensor arrays for fusion imaging

Pengshan Xie,Yunchao Xu,Jingwen Wang,Dengji Li,Yuxuan Zhang,Zixin Zeng,Boxiang Gao,Quan Quan,Bowen Li,You Meng,Weijun Wang,Yezhan Li,Yan Yan,Yi Shen,Jia Sun,Johnny C Ho

DOI: https://doi.org/10.1038/s41467-024-52563-4

2024-09-27

Abstract:Wearable visual bionic devices, fueled by advancements in artificial intelligence, are making remarkable progress. However, traditional silicon vision chips often grapple with high energy losses and challenges in emulating complex biological behaviors. In this study, we constructed a van der Waals P3HT/GaAs nanowires P-N junction by carefully directing the arrangement of organic molecules. Combined with a Schottky junction, this facilitated multi-faceted birdlike visual enhancement, including broadband non-volatile storage, low-light perception, and a near-zero power consumption operating mode in both individual devices and 5 × 5 arrays on arbitrary substrates. Specifically, we realized over 5 bits of in-memory sensing and computing with both negative and positive photoconductivity. When paired with two imaging modes (visible and UV), our reservoir computing system demonstrated up to 94% accuracy for color recognition. It achieved motion and UV grayscale information extraction (displayed with sunscreen), leading to fusion visual imaging. This work provides a promising co-design of material and device for a broadband and highly biomimetic optoelectronic neuromorphic system.
Light‐operated On‐chip Autonomous Vision Using Low‐dimensional Material Systems

Sumeet Walia

DOI: https://doi.org/10.1002/admt.202101494

IF: 6.8

2022-02-10

Advanced Materials Technologies

Abstract:Human‐brain inspired machine vision can revolutionise new technologies across sectors. Monolithic devices that are able to achieve image capture, processing, and storage with ultra‐low energy requirements can result in smart automation and enhance industrial output (both quality and quantity). This requires tapping into emerging novelties in materials physics, optical materials, and neuromorphic hardware. In this perspective, the author discusses the role of electrophotoactive low‐dimensional materials and how their unique intrinsic properties can be harnessed for futuristic machine vision systems in a major way. The author comments on some of the latest developments in 2D, 1D, and 0D material systems that have enabled smart devices and low‐energy neuromorphics. The perspective further critically assesses the challenges that the research community still needs to overcome and potential promising pathways to create miniaturized, highly efficient machine vision technologies that can transform healthcare, transport, and advanced manufacturing. This perspective discusses the role of electrophotoactive low‐dimensional materials and how their unique intrinsic properties can be harnessed for futuristic machine vision systems. The perspective further critically assesses the challenges that the research community still needs to overcome and potential promising pathways to create miniaturized, highly efficient machine vision technologies that can transform healthcare, transport, and advanced manufacturing.

materials science, multidisciplinary
A 2.17μw@120fps Ultra-Low-Power Dual-Mode CMOS Image Sensor with Senputing Architecture

Ziwei Li,Han Xu,Zheyu Liu,Li Luo,Qi Wei,Fei Qiao

DOI: https://doi.org/10.1109/asp-dac52403.2022.9712591

2022-01-01

Abstract:This paper proposes an ultra-low-power CMOS Image Sensor (CIS) chip based on sensing-with-computing (Senputing) architecture to reduce the power bottleneck of vision system. This Senputing chip achieves BNN 1st-layer convolution in analog domain with ultra-low power consumption. It has two working modes, Normal-Sensor (NS) mode and Direct- Photocurrent-Computation (DPC) mode. The prototype measurement results under 65nm CMOS process on MNIST classification task shows that the power of feature map computation is 2.17μW with 120fps frame rates and 98.1% accuracy. The computation efficiency reaches to 11.49TOPs/W, which is 14.8× higher than state-of-art works.
Artificial intelligence-based spatio-temporal vision sensors: applications and prospects

Zhixue Li,Hongwei Sun

DOI: https://doi.org/10.3389/fmats.2023.1269992

2023-12-07

Frontiers in Materials

Abstract:Introduction: Advances in machine vision and mobile electronics will be accelerated by the creation of sophisticated optoelectronic vision sensors that allow for sophisticated picture recognition of visual information and data pre-processing. Several new types of vision sensors have been devised in the last decade to solve these drawbacks, one of which is neuromorphic vision sensors, which have exciting qualities such as high temporal resolution, broad dynamic range, and low energy consumption. Neuromorphic sensors are inspired by the working principles of biological sensory neurons and would be useful in telemedicine, health surveillance, security monitoring, automatic driving, intelligent robots, and other applications of the Internet of Things. Methods: This paper provides a comprehensive review of various state-of-the-art AI vision sensors and frameworks. Results: The fundamental signal processing techniques deployed and the associated challenges were discussed. Discussion: Finally, the role of vision sensors in computer vision is also discussed.

materials science, multidisciplinary

Ultralow‐Power Machine Vision with Self‐Powered Sensor Reservoir

Senputing: An Ultra-Low-Power Always-On Vision Perception Chip Featuring the Deep Fusion of Sensing and Computing

A Multimodal Sensing CMOS Imager Based on Dual‐Focus Imaging(Adv. Sci. 14/2023)

An Adaptive Power Management Unit for Self-Sustainable Smart Vision Sensor

Low-Power Computer Vision: Status, Challenges, and Opportunities.

Low-Power Computer Vision: Status, Challenges, Opportunities

A 5.9μw Ultra-Low-Power Dual-Resolution CIS Chip of Sensing-with-Computing for Always-on Intelligent Visual Devices

An Ultra-low Power TinyML System for Real-time Visual Processing at Edge

Bioinspired and Low-Power 2D Machine Vision with Adaptive Machine Learning and Forgetting

Millimeter-Scale Ultra-Low-Power Imaging System for Intelligent Edge Monitoring

Application of Machine Vision Techniques in Low-Cost Devices to Improve Efficiency in Precision Farming

NS-MD: Near-Sensor Motion Detection with Energy Harvesting Image Sensor for Always-On Visual Perception.

An ultrawide field-of-view pinhole compound eye using hemispherical nanowire array for robot vision

Ultralow Power In-Sensor Neuronal Computing with Oscillatory Retinal Neurons for Frequency-Multiplexed, Parallel Machine Vision

Ultra-low Power In-Sensor Neuronal Computing with Oscillatory Retinal Neurons for Frequency-Multiplexed, Parallel Machine Vision

1000× Faster Camera and Machine Vision with Ordinary Devices

1000x Faster Camera and Machine Vision with Ordinary Devices

Birdlike broadband neuromorphic visual sensor arrays for fusion imaging

Light‐operated On‐chip Autonomous Vision Using Low‐dimensional Material Systems

A 2.17μw@120fps Ultra-Low-Power Dual-Mode CMOS Image Sensor with Senputing Architecture

Artificial intelligence-based spatio-temporal vision sensors: applications and prospects