Address-Event Based Stereo Vision with Bio-Inspired Silicon Retina Imagers

Jürgen Kogler,Christoph Sulzbachner,Martin Humenberger,Florian Eibensteiner,Jrgen Kogler

DOI: https://doi.org/10.5772/12941

2011-01-08

Abstract:Address-Event Based Stereo Vision with Bio-Inspired Silicon Retina Imagers | InTechOpen, Published on: 2011-01-08. Authors: Jürgen Kogler, Christoph Sulzbachner, Martin Humenberger, et

What problem does this paper attempt to address?

Neuromorphic Stereo Vision: A Survey of Bio-Inspired Sensors and Algorithms

Lea Steffen,Daniel Reichard,Jakob Weinland,Jacques Kaiser,Arne Roennau,Rüdiger Dillmann

DOI: https://doi.org/10.3389/fnbot.2019.00028

IF: 3.493

2019-05-28

Frontiers in Neurorobotics

Abstract:Any visual sensor, whether artificial or biological, maps the 3D-world on a 2D-representation. The missing dimension is depth and most species use stereo vision to recover it. Stereo vision implies multiple perspectives and matching, hence it obtains depth from a pair of images. Algorithms for stereo vision are also used prosperously in robotics. Although, biological systems seem to compute disparities effortless, artificial methods suffer from high energy demands and latency. The crucial part is the correspondence problem; finding the matching points of two images. The development of event-based cameras, inspired by the retina, enables the exploitation of an additional physical constraint—time. Due to their asynchronous course of operation, considering the precise occurrence of spikes, Spiking Neural Networks take advantage of this constraint. In this work, we investigate sensors and algorithms for event-based stereo vision leading to more biologically plausible robots. Hereby, we focus mainly on binocular stereo vision.

robotics,computer science, artificial intelligence,neurosciences
Event-Based Neuromorphic Vision for Autonomous Driving: A Paradigm Shift for Bio-Inspired Visual Sensing and Perception.

Guang Chen,Hu Cao,Jorg Conradt,Huajin Tang,Florian Rohrbein,Alois Knoll

DOI: https://doi.org/10.1109/msp.2020.2985815

IF: 15.204

2020-01-01

IEEE Signal Processing Magazine

Abstract:As a bio-inspired and emerging sensor, an event-based neuromorphic vision sensor has a different working principle compared to the standard frame-based cameras, which leads to promising properties of low energy consumption, low latency, high dynamic range (HDR), and high temporal resolution. It poses a paradigm shift to sense and perceive the environment by capturing local pixel-level light intensity changes and producing asynchronous event streams. Advanced technologies for the visual sensing system of autonomous vehicles from standard computer vision to event-based neuromorphic vision have been developed. In this tutorial-like article, a comprehensive review of the emerging technology is given. First, the course of the development of the neuromorphic vision sensor that is derived from the understanding of biological retina is introduced. The signal processing techniques for event noise processing and event data representation are then discussed. Next, the signal processing algorithms and applications for event-based neuromorphic vision in autonomous driving and various assistance systems are reviewed. Finally, challenges and future research directions are pointed out. It is expected that this article will serve as a starting point for new researchers and engineers in the autonomous driving field and provide a bird's-eye view to both neuromorphic vision and autonomous driving research communities.
Human-Computer Interaction System Based on Binocular Stereo Vision

Liao Guangjun,Hu Yueming,Li Zhifu,Wu Xuedong

2012-01-01

Abstract:Binocular stereo vision is an important aspect in computer vision field. In this paper, it is made a systematic research, which is the system of human-computer interaction based on bincular stereo vision. In order to improve the anti-interference performance the hardware system worked in the condition of near infrared ; to inhance the real time operation , the level of difficulty with stereo matching algorithm is simplified through special hardware design; based on the ICP algorithm for tracking tools, the flexibility and convenience of system are increasingly. The application in surgical navigation system is discussed. In the prototype system, how to unified virtual coordinates and real coordinates has been presented; the problem on how to tracking patient undergoing in real-time is solved. The result has higher precision and practicability, which provide a good platform for virtual reality.
Comparison between Human and Silicon Retina 2

Hauptseminar Neuroengineering,Marcello Mulas

Abstract:This paper provides a personal perspective on human vision and event-based vision sensors, algorithms, and applications over the period of last decade. The comparison between the standards of biology and technology has been the major focus throughout. Some recent advancements in the field are also briefly described.When Mahowald and Mead built the first silicon retina with asynchronous digital output around 1992 1 , conventional CMOS active pixel sensors (APS) were still research chips. It required the investment by industry of about a billion dollars to bring CMOS APS to high volume production. So it is no surprise that while the imager community has been consumed by the megapixel race to make nice photos, cameras that mimic more closely how the eye works have taken a long time to come to a useful form. These “silicon retinas” are much more complex at the pixel level than APS cameras and they pay the price in terms of fill factor and pixel size; machine vision cameras with capability of synchronous global electronic shutter are about 5um. Silicon retina pixels are roughly 10 times the area of a machine vision camera pixel. So why are silicon retinas still interesting? What was the nessesity to mimic the biological eye? Mostly because of the high cost at the system level of processing the highly redundant data from conventional cameras, and the fixed latencies imposed by the frame intervals. High performance activitydriven event-based sensors could greatly benefit applications in real time robotics, where just as in nature, latency and power are very important 2 3 . Still is this the closest that technology brought the field towards biology? What important ascpect are missing to make a bionic eye? These are the few aspects that are througly discussed throughout this paper. 1 Mahowald, M.A.: An Analog VLSI System for Stereoscopic Vision. Kluwer, Boston (1994) 2 Lichtsteiner, P., Posch, C., Delbruck, T.: A 128×128 120 dB 15 μs latency asynchronous temporal contrast vision sensor. IEEE Journal of Solid-State Circuits 43(2), 566–576 (2008) 3 . Liu, S.C., van Schaik, A., Minch, B.A., Delbruck, T.: Event-based 64-channel binaural silicon cochlea with Q enhancement mechanisms. In: IEEE ISCAS 2009, pp. 2426–2429 (2010) Comparison between Human and Silicon Retina 5

Computer Science,Engineering
A hemispherical electronic eye camera based on compressible silicon optoelectronics

Heung Cho Ko,Mark P. Stoykovich,Jizhou Song,Viktor Malyarchuk,Won Mook Choi,Chang-Jae Yu,Joseph B. Geddes III,Jianliang Xiao,Shuodao Wang,Yonggang Huang,John A. Rogers

DOI: https://doi.org/10.1038/nature07113

IF: 64.8

2008-01-01

Nature

Abstract:Looking Good: An electronic camera that shapes up like a human eye The electronic eye camera shown on the cover is a notable advance in optoelectronics. Even with the latest technologies it is difficult to produce a device to match the feats of imaging achieved by the human eye. Its hemispherical detector provides a wide field of view and low aberrations, using simple, single-component optics. Conventional optoelectronics materials exist only on the planar surfaces of rigid semiconductor wafers and cannot adopt spherical shapes. Now a multidisciplinary team based at the University of Illinois at Urbana-Champaign and Northwestern University, Evanston, has created an electronic eye-like camera based on single-crystalline silicon technology. Two novel fabrication steps make this possible. First, the optoelectronic circuits are ultra-thin in unusual, two-dimensionally compressible configurations; second, specially designed elastomeric elements transfer these planar layouts into hemispherical geometries. In addition to eye-like cameras, these strategies should make it possible to integrate planar device technologies onto the surfaces of complex curvilinear objects, for use in health monitoring devices, 'smart' prosthetics and elsewhere.
Real-Time Gesture Interface Based on Event-Driven Processing From Stereo Silicon Retinas

Jun Haeng Lee,Tobi Delbruck,Michael Pfeiffer,Paul K. J. Park,Chang-Woo Shin,Hyunsurk Ryu,Byung Chang Kang

DOI: https://doi.org/10.1109/tnnls.2014.2308551

IF: 14.255

2014-12-01

IEEE Transactions on Neural Networks and Learning Systems

Abstract:We propose a real-time hand gesture interface based on combining a stereo pair of biologically inspired event-based dynamic vision sensor (DVS) silicon retinas with neuromorphic event-driven postprocessing. Compared with conventional vision or 3-D sensors, the use of DVSs, which output asynchronous and sparse events in response to motion, eliminates the need to extract movements from sequences of video frames, and allows significantly faster and more energy-efficient processing. In addition, the rate of input events depends on the observed movements, and thus provides an additional cue for solving the gesture spotting problem, i.e., finding the onsets and offsets of gestures. We propose a postprocessing framework based on spiking neural networks that can process the events received from the DVSs in real time, and provides an architecture for future implementation in neuromorphic hardware devices. The motion trajectories of moving hands are detected by spatiotemporally correlating the stereoscopically verged asynchronous events from the DVSs by using leaky integrate-and-fire (LIF) neurons. Adaptive thresholds of the LIF neurons achieve the segmentation of trajectories, which are then translated into discrete and finite feature vectors. The feature vectors are classified with hidden Markov models, using a separate Gaussian mixture model for spotting irrelevant transition gestures. The disparity information from stereovision is used to adapt LIF neuron parameters to achieve recognition invariant of the distance of the user to the sensor, and also helps to filter out movements in the background of the user. Exploiting the high dynamic range of DVSs, furthermore, allows gesture recognition over a 60-dB range of scene illuminance. The system achieves recognition rates well over 90% under a variety of variable conditions with static and dynamic backgrounds with naïve users.

computer science, artificial intelligence, theory & methods,engineering, electrical & electronic, hardware & architecture
Recent Advances in Bio-Inspired Vision Sensor: A Review

Xiaoyu Zhong,Zhiguo Yu,Xiaofeng Gu

DOI: https://doi.org/10.1142/s0218126624300083

2024-07-12

Journal of Circuits Systems and Computers

Abstract:Journal of Circuits, Systems and Computers, Ahead of Print. Event-based cameras, also known as biologically inspired visual sensors, are capable of capturing real-time scene changes efficiently. Unlike traditional frame-based cameras, event cameras solely report triggered pixel-level brightness changes which are referred to as events. Event-based cameras show many advantages such as high temporal resolution, low latency, and high dynamic range, making them very attractive in robotics and computer vision, especially in challenging scenarios that are too demanding for traditional cameras. In this paper, we provide a comprehensive overview of the emerging field of event-based vision, focusing on the operation principle, sampling mechanisms, and algorithms that take advantage of their superior features. We also delve into the various tasks for which event cameras are utilized, such as object tracking, optical flow estimation, 3D reconstruction, SLAM, image reconstruction, and recognition. Additionally, we highlight the challenges and future opportunities for event cameras, seeking a more efficient way for machines to perceive and interact with the world.

engineering, electrical & electronic,computer science, hardware & architecture
Arthropod Eye-Inspired Digital Camera With Unique Imaging Characteristics

Jianliang Xiao,Young Min Song,Yizhu Xie,Viktor Malyarchuk,Inhwa Jung,Ki-Joong Choi,Zhuangjian Liu,Hyunsung Park,Chaofeng Lu,Rak-Hwan Kim,Rui Li,Kenneth B. Crozier,Yonggang Huang,John A. Rogers

DOI: https://doi.org/10.1117/12.2050820

2014-01-01

Abstract:In nature, arthropods have a remarkably sophisticated class of imaging systems, with a hemispherical geometry, a wide-angle field of view, low aberrations, high acuity to motion and an infinite depth of field. There are great interests in building systems with similar geometries and properties due to numerous potential applications. However, the established semiconductor sensor technologies and optics are essentially planar, which experience great challenges in building such systems with hemispherical, compound apposition layouts. With the recent advancement of stretchable optoelectronics, we have successfully developed strategies to build a fully functional artificial apposition compound eye camera by combining optics, materials and mechanics principles. The strategies start with fabricating stretchable arrays of thin silicon photodetectors and elastomeric optical elements in planar geometries, which are then precisely aligned and integrated, and elastically transformed to hemispherical shapes. This imaging device demonstrates nearly full hemispherical shape (about 160 degrees), with densely packed artificial ommatidia. The number of ommatidia (180) is comparable to those of the eyes of fire ants and bark beetles. We have illustrated key features of operation of compound eyes through experimental imaging results and quantitative ray-tracing-based simulations. The general strategies shown in this development could be applicable to other compound eye devices, such as those inspired by moths and lacewings (refracting superposition eyes), lobster and shrimp (reflecting superposition eyes), and houseflies (neural superposition eyes).
Automatic Tool Landmark Detection for Stereo Vision in Robot-Assisted Retinal Surgery

Thomas Probst,Kevis-Kokitsi Maninis,Ajad Chhatkuli,Mouloud Ourak,Emmanuel Vander Poorten,Luc Van Gool

DOI: https://doi.org/10.48550/arXiv.1709.05665

2017-11-21

Abstract:Computer vision and robotics are being increasingly applied in medical interventions. Especially in interventions where extreme precision is required they could make a difference. One such application is robot-assisted retinal microsurgery. In recent works, such interventions are conducted under a stereo-microscope, and with a robot-controlled surgical tool. The complementarity of computer vision and robotics has however not yet been fully exploited. In order to improve the robot control we are interested in 3D reconstruction of the anatomy and in automatic tool localization using a stereo microscope. In this paper, we solve this problem for the first time using a single pipeline, starting from uncalibrated cameras to reach metric 3D reconstruction and registration, in retinal microsurgery. The key ingredients of our method are: (a) surgical tool landmark detection, and (b) 3D reconstruction with the stereo microscope, using the detected landmarks. To address the former, we propose a novel deep learning method that detects and recognizes keypoints in high definition images at higher than real-time speed. We use the detected 2D keypoints along with their corresponding 3D coordinates obtained from the robot sensors to calibrate the stereo microscope using an affine projection model. We design an online 3D reconstruction pipeline that makes use of smoothness constraints and performs robot-to-camera registration. The entire pipeline is extensively validated on open-sky porcine eye sequences. Quantitative and qualitative results are presented for all steps.

Computer Vision and Pattern Recognition,Robotics
Bio-inspired hemispherical compound eye camera

jianliang xiao,young min song,y xie,viktor malyarchuk,inhwa jung,k choi,zhuangjian liu,hyunsung park,chaofeng lu,rakhwan kim,rui li,kenneth b crozier,yonggang huang,john a rogers

DOI: https://doi.org/10.1117/12.2041112

2014-01-01

Abstract:Compound eyes in arthropods demonstrate distinct imaging characteristics from human eyes, with wide angle field of view, low aberrations, high acuity to motion and infinite depth of field. Artificial imaging systems with similar geometries and properties are of great interest for many applications. However, the challenges in building such systems with hemispherical, compound apposition layouts cannot be met through established planar sensor technologies and conventional optics. We present our recent progress in combining optics, materials, mechanics and integration schemes to build fully functional artificial compound eye cameras. Nearly full hemispherical shapes (about 160 degrees) with densely packed artificial ommatidia were realized. The number of ommatidia (180) is comparable to those of the eyes of fire ants and bark beetles. The devices combine elastomeric compound optical elements with deformable arrays of thin silicon photodetectors, which were fabricated in the planar geometries and then integrated and elastically transformed to hemispherical shapes. Imaging results and quantitative ray-tracing-based simulations illustrate key features of operation. These general strategies seem to be applicable to other compound eye devices, such as those inspired by moths and lacewings (refracting superposition eyes), lobster and shrimp (reflecting superposition eyes), and houseflies (neural superposition eyes).
IRIS: Integrated Retinal Functionality in Image Sensors

Zihan Yin,Abdullah-Al Kaiser,Lamine Ousmane Camara,Mark Camarena,Maryam Parsa,Ajey Jacob,Gregory Schwartz,Akhilesh Jaiswal,Md Abdullah-Al Kaiser

DOI: https://doi.org/10.3389/fnins.2023.1241691

IF: 4.3

2023-09-01

Frontiers in Neuroscience

Abstract:Neuromorphic image sensors draw inspiration from the biological retina to implement visual computations in electronic hardware. Gain control in phototransduction and temporal differentiation at the first retinal synapse inspired the first generation of neuromorphic sensors, but processing in downstream retinal circuits, much of which has been discovered in the past decade, has not been implemented in image sensor technology. We present a technology-circuit co-design solution that implements two motion computations—object motion sensitivity and looming detection—at the retina's output that could have wide applications for vision-based decision-making in dynamic environments. Our simulations on Globalfoundries 22 nm technology node show that the proposed retina-inspired circuits can be fabricated on image sensing platforms in existing semiconductor foundries by taking advantage of the recent advances in semiconductor chip stacking technology. Integrated Retinal Functionality in Image Sensors (IRIS) technology could drive advances in machine vision applications that demand energy-efficient and low-bandwidth real-time decision-making.

neurosciences
Hyperacute Edge and Bar Detection in a Bioinspired Optical Position Sensing Device

S. Viollet,R. Juston,L. Kerhuel,N. Franceschini

DOI: https://doi.org/10.1109/TMECH.2013.2265983

2014-06-01

IEEE/ASME Transactions on Mechatronics

Abstract:We present an improved bioinspired optical position sensing device, in which insect-based retinal microscanning movements are used to detect and locate contrasting objects such as edges or bars. The active microvibrations imposed upon the retina endow the sensor with hyperacuity. For the sake of clarity, this is demonstrated here for a two-pixel sensor, but the same principle could be applied to all pairs of neighboring photosensors in a focal plane array. The sensor is able to detect an edge or a bar present within its small field of view ( 4°) and locate it with a resolution ( 0.025°) 160-fold finer than the static resolution imposed by the pixel spacing. The sensor features the novel ability to establish whether it is actually facing an edge or a bar, based on the phase difference between the sinusoidally modulated signals of its two photoreceptors. The visual processing algorithm involves simple linear filtering and purely arithmetic operations requiring few computational resources. The complete theoretical framework is presented here, including an analytical model for the microscanning sensor. This high-performance, low-cost angular position sensing device could have many applications in fields such as metrology, astronomy, robotics, automotive design, and aerospace.

Engineering,Biology
A Real-Time Image Optimization Strategy Based on Global Saliency Detection for Artificial Retinal Prostheses.

Heng Li,Tingting Han,Jing Wang,Zhuofan Lu,Xiaofei Cao,Yao Chen,Liming Li,Chuanqing Zhou,Xinyu Chai

DOI: https://doi.org/10.1016/j.ins.2017.06.014

IF: 8.1

2017-01-01

Information Sciences

Abstract:Current retinal prostheses can only generate low-resolution visual percepts constituted of inadequate phosphenes which are elicited by a limited number of stimulating electrodes and with unruly color and restricted grayscale. Fortunately, for most retinal prostheses, an external camera and a video processing unit are employed to be essential components, and allow image processing to improve visual perception for recipients. At present, there have been some studies that use a variety of sophisticated image processing algorithms to improve prosthetic vision perception. However, most of them cannot achieve real-time processing due to the complexity of the algorithms and the limitation of platform processing power. This greatly curbs the practical application of these algorithms on the retinal prostheses. In this study, we propose a real-time image processing strategy based on a novel bottom-up saliency detection algorithm, aiming to detect and enhance foreground objects in a scene. Results demonstrate by verification of conducting two eye-hand-coordination visual tasks that under simulated prosthetic vision, our proposed strategy has noticeable advantages in terms of accuracy, efficiency, and head motion range. The study aims to help develop image processing modules in retinal prostheses, and is hoped to provide more benefit towards prosthesis recipients.
An Event-Driven Stereo System for Real-Time 3-D 360° Panoramic Vision

Stephan Schraml,Ahmed Nabil Belbachir,Horst Bischof

DOI: https://doi.org/10.1109/tie.2015.2477265

IF: 7.7

2016-01-01

IEEE Transactions on Industrial Electronics

Abstract:A new multiperspective stereo concept for real-time 3-D panoramic vision is presented in this paper. The main contribution is a novel event-driven stereo approach enabling 3-D 360° high-dynamic-range panoramic vision for real-time application in a natural environment. This approach makes use of a sparse visual code generated by a rotating pair of dynamic vision line sensors. The use of this system allows panoramic images to be generated by the transformation of events, eliminating the need to capture a large set of images. It thereby increases the acquisition speed, which improves accuracy in dynamic scenes. This paper focuses on its 3-D reconstruction and performance analysis using such a rotating multiperspective vision system. First, a theoretical analysis of the stereo matching accuracy is performed. Second, a depth error formulation is developed, which takes motion into consideration and reveals the leverage of scene dynamics on depth estimation. In this paper, disparity is measured in time units, which allows accurate depth maps to be estimated from a moving sensor system. Third, a stereo matching workflow is presented using standard stereo image matching to assess the 3-D reconstruction accuracy. Finally, experimental results are reported on real-world sensor data, showing that the system allows for the 3-D reconstruction of high-resolution round views even under challenging illumination conditions.

automation & control systems,engineering, electrical & electronic,instruments & instrumentation
A bionic self-driven retinomorphic eye with ionogel photosynaptic retina

Xu Luo,Chen Chen,Zixi He,Min Wang,Keyuan Pan,Xuemei Dong,Zifan Li,Bin Liu,Zicheng Zhang,Yueyue Wu,Chaoyi Ban,Rong Chen,Dengfeng Zhang,Kaili Wang,Qiye Wang,Junyue Li,Gang Lu,Juqing Liu,Zhengdong Liu,Wei Huang

DOI: https://doi.org/10.1038/s41467-024-47374-6

IF: 16.6

2024-04-10

Nature Communications

Abstract:Abstract Bioinspired bionic eyes should be self-driving, repairable and conformal to arbitrary geometries. Such eye would enable wide-field detection and efficient visual signal processing without requiring external energy, along with retinal transplantation by replacing dysfunctional photoreceptors with healthy ones for vision restoration. A variety of artificial eyes have been constructed with hemispherical silicon, perovskite and heterostructure photoreceptors, but creating zero-powered retinomorphic system with transplantable conformal features remains elusive. By combining neuromorphic principle with retinal and ionoelastomer engineering, we demonstrate a self-driven hemispherical retinomorphic eye with elastomeric retina made of ionogel heterojunction as photoreceptors. The receptor driven by photothermoelectric effect shows photoperception with broadband light detection (365 to 970 nm), wide field-of-view (180°) and photosynaptic (paired-pulse facilitation index, 153%) behaviors for biosimilar visual learning. The retinal photoreceptors are transplantable and conformal to any complex surface, enabling visual restoration for dynamic optical imaging and motion tracking.

multidisciplinary sciences
A Spike-Based Neuromorphic Architecture of Stereo Vision

Nicoletta Risi,Alessandro Aimar,Elisa Donati,Sergio Solinas,Giacomo Indiveri

DOI: https://doi.org/10.3389/fnbot.2020.568283

IF: 3.493

2020-11-13

Frontiers in Neurorobotics

Abstract:The problem of finding stereo correspondences in binocular vision is solved effortlessly in nature and yet it is still a critical bottleneck for artificial machine vision systems. As temporal information is a crucial feature in this process, the advent of event-based vision sensors and dedicated event-based processors promises to offer an effective approach to solving the stereo matching problem. Indeed, event-based neuromorphic hardware provides an optimal substrate for fast, asynchronous computation, that can make explicit use of precise temporal coincidences. However, although several biologically-inspired solutions have already been proposed, the performance benefits of combining event-based sensing with asynchronous and parallel computation are yet to be explored. Here we present a hardware spike-based stereo-vision system that leverages the advantages of brain-inspired neuromorphic computing by interfacing two event-based vision sensors to an event-based mixed-signal analog/digital neuromorphic processor. We describe a prototype interface designed to enable the emulation of a stereo-vision system on neuromorphic hardware and we quantify the stereo matching performance with two datasets. Our results provide a path toward the realization of low-latency, end-to-end event-based, neuromorphic architectures for stereo vision.

robotics,computer science, artificial intelligence,neurosciences
Hemispherical Retina Emulated by Plasmonic Optoelectronic Memristors with All-Optical Modulation for Neuromorphic Stereo Vision

Xuanyu Shan,Zhongqiang Wang,Jun Xie,Jiaqi Han,Ye Tao,Ya Lin,Xiaoning Zhao,Daniele Ielmini,Yichun Liu,Haiyang Xu

DOI: https://doi.org/10.1002/advs.202405160

Abstract:Binocular stereo vision relies on imaging disparity between two hemispherical retinas, which is essential to acquire image information in three dimensional environment. Therefore, retinomorphic electronics with structural and functional similarities to biological eyes are always highly desired to develop stereo vision perception system. In this work, a hemispherical optoelectronic memristor array based on Ag-TiO2 nanoclusters/sodium alginate film is developed to realize binocular stereo vision. All-optical modulation induced by plasmonic thermal effect and optical excitation in Ag-TiO2 nanoclusters is exploited to realize in-pixel image sensing and storage. Wide field of view (FOV) and spatial angle detection are experimentally demonstrated owing to the device arrangement and incident-angle-dependent characteristics in hemispherical geometry. Furthermore, depth perception and motion detection based on binocular disparity have been realized by constructing two retinomorphic memristive arrays. The results demonstrated in this work provide a promising strategy to develop all-optically controlled memristor and promote the future development of binocular vision system with in-sensor architecture.
Retinomorphic optoelectronic devices for intelligent machine vision.

Weilin Chen,Zhang Zhang,Gang Liu

DOI: https://doi.org/10.1016/j.isci.2021.103729

IF: 5.8

2022-01-01

iScience

Abstract:Biological visual system can efficiently handle optical information within the retina and visual cortex of the brain, which suggests an alternative approach for the upgrading of the current low-intelligence, large energy consumption, and complex circuitry of the artificial vision system for high-performance edge computing applications. In recent years, retinomorphic machine vision based on the integration of optoelectronic image sensors and processors has been regarded as a promising candidate to improve this phenomenon. This novel intelligent machine vision technology can perform information preprocessing near or even within the sensor in the front end, thereby reducing the transmission of redundant raw data and improving the efficiency of the back-end processor for high-level computing tasks. In this contribution, we try to present a comprehensive review on the recent progress achieved in this emergent field.
Object Motion Sensitivity: A Bio-inspired Solution to the Ego-motion Problem for Event-based Cameras

Shay Snyder,Hunter Thompson,Md Abdullah-Al Kaiser,Gregory Schwartz,Akhilesh Jaiswal,Maryam Parsa

2023-04-15

Abstract:Neuromorphic (event-based) image sensors draw inspiration from the human-retina to create an electronic device that can process visual stimuli in a way that closely resembles its biological counterpart. These sensors process information significantly different than the traditional RGB sensors. Specifically, the sensory information generated by event-based image sensors are orders of magnitude sparser compared to that of RGB sensors. The first generation of neuromorphic image sensors, Dynamic Vision Sensor (DVS), are inspired by the computations confined to the photoreceptors and the first retinal synapse. In this work, we highlight the capability of the second generation of neuromorphic image sensors, Integrated Retinal Functionality in CMOS Image Sensors (IRIS), which aims to mimic full retinal computations from photoreceptors to output of the retina (retinal ganglion cells) for targeted feature-extraction. The feature of choice in this work is Object Motion Sensitivity (OMS) that is processed locally in the IRIS sensor. Our results show that OMS can accomplish standard computer vision tasks with similar efficiency to conventional RGB and DVS solutions but offers drastic bandwidth reduction. This cuts the wireless and computing power budgets and opens up vast opportunities in high-speed, robust, energy-efficient, and low-bandwidth real-time decision making.

Computer Vision and Pattern Recognition,Neural and Evolutionary Computing
2D Semiconductor‐Based Optoelectronics for Artificial Vision

Na Li,Songge Zhang,Yalin Peng,Xiuzhen Li,Yangkun Zhang,Congli He,Guangyu Zhang

DOI: https://doi.org/10.1002/adfm.202305589

IF: 19

2023-09-10

Advanced Functional Materials

Abstract:This comprehensive article provides an in‐depth review of recent advancements in 2D semiconductor‐based optoelectronics for artificial vision. The scope includes 2D semiconductor‐based photodetectors, optoelectronic memory, artificial synapses, and innovative applications inspired by retinal cells and vision systems. Furthermore, the review addresses critical technical challenges, outlines strategic approaches for development, and explores potential applications in detail. Artificial retina technologies aim to restore visual function by mimicking the natural processes of the eye. These biomimetic devices can convert light into electrical signals that the brain can interpret as visual information, bypassing damaged or non‐functional cells of the eye. To be effective, these devices should possess high sensitivity to light, high spatial resolution, biocompatibility, power efficiency, and so on. Recently, 2D semiconductor materials have appeared as a promising candidate for artificial retinal devices, thanks to their excellent optoelectronic properties, ultrathin body, flexible nature, and biocompatibility. Here the recent developments in the field of 2D semiconductors‐based optoelectronics for visual function recovery are reviewed and their potential applications are discussed. The photodetector, optoelectronic memory, and artificial synapse mechanisms and devices utilized in artificial systems that are based on 2D semiconductor materials are summarized. Additionally, a range of application scenarios for devices that are inspired by retinal cells and vision systems is explored. Finally, it is concluded with an overview of the critical technical challenges and strategies that must be addressed for the successful development of artificial retina technologies. It also highlights the potential for new applications in other fields, such as robotics and artificial intelligence.

materials science, multidisciplinary,chemistry, physical,physics, applied, condensed matter,nanoscience & nanotechnology

Address-Event Based Stereo Vision with Bio-Inspired Silicon Retina Imagers

Neuromorphic Stereo Vision: A Survey of Bio-Inspired Sensors and Algorithms

Event-Based Neuromorphic Vision for Autonomous Driving: A Paradigm Shift for Bio-Inspired Visual Sensing and Perception.

Human-Computer Interaction System Based on Binocular Stereo Vision

Comparison between Human and Silicon Retina 2

A hemispherical electronic eye camera based on compressible silicon optoelectronics

Real-Time Gesture Interface Based on Event-Driven Processing From Stereo Silicon Retinas

Recent Advances in Bio-Inspired Vision Sensor: A Review

Arthropod Eye-Inspired Digital Camera With Unique Imaging Characteristics

Automatic Tool Landmark Detection for Stereo Vision in Robot-Assisted Retinal Surgery

Bio-inspired hemispherical compound eye camera

IRIS: Integrated Retinal Functionality in Image Sensors

Hyperacute Edge and Bar Detection in a Bioinspired Optical Position Sensing Device

A Real-Time Image Optimization Strategy Based on Global Saliency Detection for Artificial Retinal Prostheses.

An Event-Driven Stereo System for Real-Time 3-D 360° Panoramic Vision

A bionic self-driven retinomorphic eye with ionogel photosynaptic retina

A Spike-Based Neuromorphic Architecture of Stereo Vision

Hemispherical Retina Emulated by Plasmonic Optoelectronic Memristors with All-Optical Modulation for Neuromorphic Stereo Vision

Retinomorphic optoelectronic devices for intelligent machine vision.

Object Motion Sensitivity: A Bio-inspired Solution to the Ego-motion Problem for Event-based Cameras

2D Semiconductor‐Based Optoelectronics for Artificial Vision