Wireless steerable vision for live insects and insect-scale robots

Vikram Iyer,Ali Najafi,Johannes James,Sawyer Fuller,Shyamnath Gollakota

DOI: https://doi.org/10.1126/scirobotics.abb0839

IF: 25

2020-07-15

Science Robotics

Abstract:A mechanically steerable vision system that imitates insect head motion can be mounted on insects and small robots.

robotics

What problem does this paper attempt to address?

From insect vision to robot vision

N. Franceschini,J. M. Pichon,C. Blanes

DOI: https://doi.org/10.1098/rstb.1992.0106

1992-09-29

Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences

Abstract:Airborne insects are miniature w ing-flapping aircraft the visually guided manoeuvres of which depend on analogue, ‘fly-by-wire’ controls. The front-end of their visuomotor system consists of a pair of com pound eyes which are masterpieces of integrated optics and neural design. They rely on an array of passive sensors driving an orderly analogue neural network. We explored in concrete terms how motion-detecting neurons might possibly be used to solve navigational tasks involving obstacle avoidance in a creature whose wings are exquisitely guided by eyes with a poor spatial resolution. We designed, simulated, and built a complete terrestrial creature which moves about and avoids obstacles solely by evaluating the relative motion between itself and the environment. The compound eye uses an array of elementary motion detectors (emds) as smart, passive ranging sensors. Like its physiological counterpart, the visuomotor system is based on analogue, continuous-time processing and does not make use of conventional computers. It uses hardly any memory to adjust the robot’s heading in real time via a local and intermittent visuomotor feedback loop. This paper shows that the understanding of some invertebrate sensory-motor systems has now reached a level able to provide valuable design hints. Our approach brings into prominence the mutual constraints in the designs of a sensory and a motor system, in both living and non-living ambulatory creatures.
A wearable wireless platform for visually stimulating small flying insects

K Mann,T L Massey,S Guha,J P van Kleef,M M Maharbiz

DOI: https://doi.org/10.1109/EMBC.2014.6943923

Abstract:Linking neurons and muscles to their roles in behavior requires not only the ability to measure their response during unrestrained movement but also the ability to stimulate them and observe the behavioral results. Current wireless stimulation technologies can be carried by rodent-sized animals and very large insects. However, the mass and volume of these devices make them impractical for studying smaller animals like insects. Here we present a battery-powered electronics platform suitable to be carried on a flying locust (2.7 g). The device has an IR-based (infrared) receiver, can deliver optical or electrical stimulation, occupies a volume of 0.1 cm(3), and weighs ~280 mg. We show the device is capable of powering two white SMD light emitting diodes (LEDs) for ~4 min and can be recharged in ~20 min. We demonstrate that our system shows no crosstalk with an IR-based Vicon tracking system. The entire package is made from commercial off-the-shelf components and requires no microfabrication.
Target Detection and Localization for Flapping-Wing Robots Based on Biological Eagle-Eye Vision

Qiang Fu,Shutai Wang,Jin Wang,Xiaoyang Wu,Wei He

DOI: https://doi.org/10.1109/tmech.2024.3424983

2024-01-01

Abstract:In recent years, the development of flapping-wing robots (FWRs) has made rapid progress. However, the vision-based perception system of FWRs is still in its infancy, which seriously limits the application of FWRs in military reconnaissance and civil monitoring. To this end, this article designs an eagle-eye-based vision system for target detection, tracking and localization, which fits well with the perception requirements of FWRs. First, inspired by the double-fovea structure of eagle eye, a two-degree-of-freedom pan-tilt equipped with two different focal length cameras is established. Second, the aerial photography characteristics of FWRs and the lateral inhibition mechanism are employed to improve the accuracy of target detection. Third, based on the information of the two different focal length cameras, a cooperative control method is proposed for the pan-tilt to achieve target tracking and the light gradient boosting machine algorithm is adopted to improve the accuracy of target localization. Finally, flight experiments under various scenarios are conducted by using our self-developed FWR named USTB-Hawk, demonstrating the effectiveness of the proposed vision system.
Millisecond insect tracking system

T. Thang Vo-Doan,Andrew D. Straw

DOI: https://doi.org/10.48550/arXiv.2002.12100

2020-02-27

Abstract:Animals such as insects have provided a rich source of inspiration for designing robots. For example, animals navigate to goals via efficient coordination of individual motor actions, and demonstrate natural solutions to problems also faced by engineers. Recording individual body part positions during large scale movement would therefore be useful. Such multi-scale observations, however, are challenging. With video, for example, there is typically a trade-off between the volume over which an animal can be recorded and spatial resolution within the volume. Even with high pixel-count cameras, motion blur can be a challenge when using available light. Here we present a new approach for tracking animals, such as insects, with an optical system that bypasses this tradeoff by actively pointing a telephoto video camera at the animal. This system is based around high-speed pan-tilt mirrors which steer an optical path shared by a quadrant photodiode and a high-resolution, high-speed telephoto video recording system. The mirror is directed to lock on to the image of a 25-milligram retroreflector worn by the animal. This system allows high-magnification videography with reduced motion blur over a large tracking volume. With our prototype, we obtained millisecond order closed-loop latency and recorded videos of flying insects in a tracking volume extending to an axial distance of 3 meters and horizontally and vertically by 40 degrees. The system offers increased capabilities compared to other video recording solutions and may be useful for the study of animal behavior and the design of bio-inspired robots.

Quantitative Methods
Real-time and Precise Insect Flight Control System Based on Virtual Reality

Nenggan Zheng,Mengjie Jin,Hui Hong,Lei Huang,Zonghua Gu,Hong Li

DOI: https://doi.org/10.1049/el.2016.3048

2017-01-01

Electronics Letters

Abstract:A non-invasive and automatic control method based on real-time virtual reality and reinforcement learning is reported to modulate insects' flight behaviour. The online configurable LED display system is proposed to present the visual stimulus within ultra-low latency. State-action-reward-state-action (SARSA) coupled with sequential K-means clustering is applied to generate optimal control sequence automatically to ensure the satisfactory control performance. By evaluating the control results of 15 bumblebees, it shows that the proposed method achieves to modulate the abdominal waving pattern quickly (less than six steps) with small deviation (<0.03).
Living IoT: A Flying Wireless Platform on Live Insects

Vikram Iyer,Rajalakshmi Nandakumar,Anran Wang,Sawyer Fuller,Shyamnath Gollakota

DOI: https://doi.org/10.1145/3300061.3300136

2018-12-22

Abstract:Sensor networks with devices capable of moving could enable applications ranging from precision irrigation to environmental sensing. Using mechanical drones to move sensors, however, severely limits operation time since flight time is limited by the energy density of current battery technology. We explore an alternative, biology-based solution: integrate sensing, computing and communication functionalities onto live flying insects to create a mobile IoT platform. Such an approach takes advantage of these tiny, highly efficient biological insects which are ubiquitous in many outdoor ecosystems, to essentially provide mobility for free. Doing so however requires addressing key technical challenges of power, size, weight and self-localization in order for the insects to perform location-dependent sensing operations as they carry our IoT payload through the environment. We develop and deploy our platform on bumblebees which includes backscatter communication, low-power self-localization hardware, sensors, and a power source. We show that our platform is capable of sensing, backscattering data at 1 kbps when the insects are back at the hive, and localizing itself up to distances of 80 m from the access points, all within a total weight budget of 102 mg.

Networking and Internet Architecture
Synthetic Neural Vision System Design for Motion Pattern Recognition in Dynamic Robot Scenes

Qinbing Fu,Cheng Hu,Pengcheng Liu,Shigang Yue

DOI: https://doi.org/10.48550/arXiv.1904.07180

2019-04-16

Abstract:Insects have tiny brains but complicated visual systems for motion perception. A handful of insect visual neurons have been computationally modeled and successfully applied for robotics. How different neurons collaborate on motion perception, is an open question to date. In this paper, we propose a novel embedded vision system in autonomous micro-robots, to recognize motion patterns in dynamic robot scenes. Here, the basic motion patterns are categorized into movements of looming (proximity), recession, translation, and other irrelevant ones. The presented system is a synthetic neural network, which comprises two complementary sub-systems with four spiking neurons -- the lobula giant movement detectors (LGMD1 and LGMD2) in locusts for sensing looming and recession, and the direction selective neurons (DSN-R and DSN-L) in flies for translational motion extraction. Images are transformed to spikes via spatiotemporal computations towards a switch function and decision making mechanisms, in order to invoke proper robot behaviors amongst collision avoidance, tracking and wandering, in dynamic robot scenes. Our robot experiments demonstrated two main contributions: (1) This neural vision system is effective to recognize the basic motion patterns corresponding to timely and proper robot behaviors in dynamic scenes. (2) The arena tests with multi-robots demonstrated the effectiveness in recognizing more abundant motion features for collision detection, which is a great improvement compared with former studies.

Neural and Evolutionary Computing,Artificial Intelligence,Robotics
Steering Motion Control of a Snake Robot Via a Biomimetic Approach

Wenjuan Ouyang,Wenyu Liang,Chenzui Li,Hui Zheng,Qinyuan Ren,Ping Li

DOI: https://doi.org/10.1631/fitee.1800554

IF: 2.526

2019-01-01

Frontiers of Information Technology & Electronic Engineering

Abstract:We propose a biomimetic approach for steering motion control of a snake robot. Inspired by a vertebrate biological motor system paradigm, a hierarchical control scheme is adopted. In the control scheme, an artificial central pattern generator (CPG) is employed to generate serpentine locomotion in the robot. This generator outputs the coordinated desired joint angle commands to each lower-level effector controller, while the locomotion can be controlled through CPG modulation by a higher-level motion controller. The motion controller consists of a cerebellar model articulation controller (CMAC) and a proportional-derivative (PD) controller. Because of the fast learning ability of the CMAC, the proposed motion controller can drive the robot to track the desired orientation and adapt to unexpected perturbations. The PD controller is employed to expedite the convergence speed of the motion controller. Finally, both numerical studies and experiments proved that the proposed approach can help the snake robot achieve good tracking performance and adaptability in a varying environment.
Visually Guided Swarm Motion Coordination via Insect-inspired Small Target Motion Reactions

Md Arif Billah,Imraan A. Faruque

2024-05-08

Abstract:Despite progress developing experimentally-consistent models of insect in-flight sensing and feedback for individual agents, a lack of systematic understanding of the multi-agent and group performance of the resulting bio-inspired sensing and feedback approaches remains a barrier to robotic swarm implementations. This study introduces the small-target motion reactive (STMR) swarming approach by designing a concise engineering model of the small target motion detector (STMD) neurons found in insect lobula complexes. The STMD neuron model identifies the bearing angle at which peak optic flow magnitude occurs, and this angle is used to design an output feedback switched control system. A theoretical stability analysis provides bi-agent stability and state boundedness in group contexts. The approach is simulated and implemented on ground vehicles for validation and behavioral studies. The results indicate despite having the lowest connectivity of contemporary approaches (each agent instantaneously regards only a single neighbor), collective group motion can be achieved. STMR group level metric analysis also highlights continuously varying polarization and decreasing heading variance.

Systems and Control
An autonomous robot inspired by insect neurophysiology pursues moving features in natural environments

Zahra M Bagheri,Benjamin S Cazzolato,Steven Grainger,David C O'Carroll,Steven D Wiederman

DOI: https://doi.org/10.1088/1741-2552/aa776c

Abstract:Objective: Many computer vision and robotic applications require the implementation of robust and efficient target-tracking algorithms on a moving platform. However, deployment of a real-time system is challenging, even with the computational power of modern hardware. Lightweight and low-powered flying insects, such as dragonflies, track prey or conspecifics within cluttered natural environments, illustrating an efficient biological solution to the target-tracking problem. Approach: We used our recent recordings from 'small target motion detector' neurons in the dragonfly brain to inspire the development of a closed-loop target detection and tracking algorithm. This model exploits facilitation, a slow build-up of response to targets which move along long, continuous trajectories, as seen in our electrophysiological data. To test performance in real-world conditions, we implemented this model on a robotic platform that uses active pursuit strategies based on insect behaviour. Main results: Our robot performs robustly in closed-loop pursuit of targets, despite a range of challenging conditions used in our experiments; low contrast targets, heavily cluttered environments and the presence of distracters. We show that the facilitation stage boosts responses to targets moving along continuous trajectories, improving contrast sensitivity and detection of small moving targets against textured backgrounds. Moreover, the temporal properties of facilitation play a useful role in handling vibration of the robotic platform. We also show that the adoption of feed-forward models which predict the sensory consequences of self-movement can significantly improve target detection during saccadic movements. Significance: Our results provide insight into the neuronal mechanisms that underlie biological target detection and selection (from a moving platform), as well as highlight the effectiveness of our bio-inspired algorithm in an artificial visual system.
A 116.4 Mg Wireless Ultra-Lightweight Microstimulator for Cyborg Bees

Hui Hong,Haochuan Wang,Chenglong Zhu,Wenkai Jin,Xingyang Ye,Xiwang Dai,Qian Ma,Nenggan Zheng

DOI: https://doi.org/10.1109/tie.2024.3454200

IF: 7.7

2024-01-01

IEEE Transactions on Industrial Electronics

Abstract:Cyborg insects are considered to be viable substitutes for micro air vehicles (MAVs) with the wireless electric microstimulator. The design of a wireless stimulator with lightweight, small-size, and low-power that can be comfortably carried by insects in free motion has proved critically challenging. Here, a novel wireless microstimulator with an ultra-lightweight of 116.4 mg and a compact size of 121.8 mm 3 is reported. It combines semiactive radio frequency identification (RFID) technology and millimeter-scale miniaturized antenna with heterogeneous integration process, thus meeting the stringent requirements of cyborg bees. The semiactive RFID technique uses radio-frequency energy harvesting to supply the power of wireless communication and then greatly reduce the energy requirement of the battery. To satisfy the radio-frequency energy harvesting requirement and realize conjugate matching of semiactive RFID chip and antenna at the same time, a millimeter-scale miniaturized antenna with compact spiral-structure is designed. A high-flexible heterogeneous integration process that combines the MCU die, antenna, and RFID chip together in the same ultra-thin substrate is proposed to solve the integration density limitation problem of the microsystem. Compared with the lightest reported microstimulator of cyborg insects, the weight reduced by almost 60%. The behavior control experiments are established to verify the control functionalities of this wireless stimulator. Experimental results demonstrate that the proposed wireless ultra-lightweight microstimulator is capable of outputting stimulus waveforms into the bee’s brain, and controlling the crawling deflection of cyborg bees.
Arthropod Eye-Inspired Digital Camera With Unique Imaging Characteristics

Jianliang Xiao,Young Min Song,Yizhu Xie,Viktor Malyarchuk,Inhwa Jung,Ki-Joong Choi,Zhuangjian Liu,Hyunsung Park,Chaofeng Lu,Rak-Hwan Kim,Rui Li,Kenneth B. Crozier,Yonggang Huang,John A. Rogers

DOI: https://doi.org/10.1117/12.2050820

2014-01-01

Abstract:In nature, arthropods have a remarkably sophisticated class of imaging systems, with a hemispherical geometry, a wide-angle field of view, low aberrations, high acuity to motion and an infinite depth of field. There are great interests in building systems with similar geometries and properties due to numerous potential applications. However, the established semiconductor sensor technologies and optics are essentially planar, which experience great challenges in building such systems with hemispherical, compound apposition layouts. With the recent advancement of stretchable optoelectronics, we have successfully developed strategies to build a fully functional artificial apposition compound eye camera by combining optics, materials and mechanics principles. The strategies start with fabricating stretchable arrays of thin silicon photodetectors and elastomeric optical elements in planar geometries, which are then precisely aligned and integrated, and elastically transformed to hemispherical shapes. This imaging device demonstrates nearly full hemispherical shape (about 160 degrees), with densely packed artificial ommatidia. The number of ommatidia (180) is comparable to those of the eyes of fire ants and bark beetles. We have illustrated key features of operation of compound eyes through experimental imaging results and quantitative ray-tracing-based simulations. The general strategies shown in this development could be applicable to other compound eye devices, such as those inspired by moths and lacewings (refracting superposition eyes), lobster and shrimp (reflecting superposition eyes), and houseflies (neural superposition eyes).
Visual guidance based on optic flow: a biorobotic approach

Nicolas Franceschini

DOI: https://doi.org/10.1016/j.jphysparis.2004.06.002

Abstract:This paper addresses some basic questions as to how vision links up with action and serves to guide locomotion in both biological and artificial creatures. The thorough knowledge gained during the past five decades on insects' sensory-motor abilities and the neuronal substrates involved has provided us with a rich source of inspiration for designing tomorrow's self-guided vehicles and micro-vehicles, which will be able to cope with unforeseen events on the ground, under water, in the air, in space, on other planets, and inside the human body. Insects can teach us some useful tricks for designing agile autonomous robots. Since constructing a "biorobot" first requires exactly formulating the biological principles presumably involved, it gives us a unique opportunity of checking the soundness and robustness of these principles by bringing them face to face with the real physical world. "Biorobotics" therefore goes one step beyond computer simulation. It leads to experimenting with real physical robots which have to pass the stringent test of the real world. Biorobotics provide us with a new tool, which can help neurobiologists and neuroethologists to identify and investigate worthwhile issues in the field of sensory-motor control. Here we describe some of the visually guided terrestrial and aerial robots we have developed since 1985 on the basis of our biological findings. All these robots behave in response to the optic flow, i.e., they work by measuring the slip speed of the retinal image. Optic flow is sensed on-board by miniature electro-optical velocity sensors. The very principle of these sensors was based on studies in which we recorded the responses of single identified neurons to single photoreceptor stimulation in a model visual system: the fly's compound eye.
A self-sustaining wireless sensing and flight control device for beetles

Xin Huang,Wenhao Zhao,Meisong Yuan,Kaixuan Sun,Bo Yang

DOI: https://doi.org/10.1063/5.0228102

IF: 1.697

2024-09-01

AIP Advances

Abstract:Insect-electronics hybrid robots integrate live insects with small electronic backpacks. These backpacks guide insect movement and sense environmental data. Due to size and payload constraints, high-capacity batteries are impractical for prolonged energy supply. This study proposes a self-sustaining wireless sensing and flight control device with an energy management module and a Bluetooth slave module. This setup enables solar energy harvesting, wireless flight control of beetles, and the acquisition of image and attitude information. The device achieves directional flight control with a 72.5% success rate in turning. Operation duration increases by 46.6% in image-only mode and 50.9% in combined image and attitude angle mode under 20 000 lux indoor illuminance. Outdoor experiments demonstrate continuous operation at one image per second under favorable illuminance conditions, with a 92% increase in operation time when capturing images and yaw angles. This microelectronic device enhances the durability of insect robots during extended field missions, providing valuable insights into long-term environmental monitoring.

materials science, multidisciplinary,physics, applied,nanoscience & nanotechnology
High Resolution Outdoor Videography of Insects Using Fast Lock-On Tracking

T. Thang Vo-Doan,Victor V Titov,Michael JM Harrap,Stephan Lochner,Andrew D Straw

DOI: https://doi.org/10.1101/2023.12.20.572558

2024-08-22

Abstract:Insects have significant global impacts on ecology, economy, and health and yet our understanding of their behavior remains limited. Bees, for example, use vision and a tiny brain to find flowers and return home, but understanding how they perform these impressive tasks has been hampered by limitations in recording technology. Here we present Fast Lock-On (FLO) tracking. This method moves an image sensor to remain focused on a retroreflective marker affixed to an insect. Using paraxial infrared illumination, simple image processing can localize the sensor location of the insect in a few milliseconds. When coupled with a feedback system to steer a high magnification optical system to remain focused on the insect, a high spatial-temporal resolution trajectory can be gathered over a large region. As the basis for several robotic systems, we show FLO is a versatile idea which can be employed in combination with other components. We demonstrate that the optical path can be split and used for recording high-speed video. Furthermore, by flying a FLO system on a quadcopter drone, we track a flying honey bee and anticipate tracking insects in the wild over kilometer scales. Such systems have the capability of providing higher resolution information about insects behaving in natural environments and as such will be helpful in revealing the biomechanical and neuroethological mechanisms used by insects in natural settings.

Animal Behavior and Cognition
Bioinspired engineering of exploration systems for NASA and DoD

Sarita Thakoor,Javaan Chahl,M V Srinivasan,L Young,Frank Werblin,Butler Hine,Steven Zornetzer

DOI: https://doi.org/10.1162/106454602321202426

Abstract:A new approach called bioinspired engineering of exploration systems (BEES) and its value for solving pressing NASA and DoD needs are described. Insects (for example honeybees and dragonflies) cope remarkably well with their world, despite possessing a brain containing less than 0.01% as many neurons as the human brain. Although most insects have immobile eyes with fixed focus optics and lack stereo vision, they use a number of ingenious, computationally simple strategies for perceiving their world in three dimensions and navigating successfully within it. We are distilling selected insect-inspired strategies to obtain novel solutions for navigation, hazard avoidance, altitude hold, stable flight, terrain following, and gentle deployment of payload. Such functionality provides potential solutions for future autonomous robotic space and planetary explorers. A BEES approach to developing lightweight low-power autonomous flight systems should be useful for flight control of such biomorphic flyers for both NASA and DoD needs. Recent biological studies of mammalian retinas confirm that representations of multiple features of the visual world are systematically parsed and processed in parallel. Features are mapped to a stack of cellular strata within the retina. Each of these representations can be efficiently modeled in semiconductor cellular nonlinear network (CNN) chips. We describe recent breakthroughs in exploring the feasibility of the unique blending of insect strategies of navigation with mammalian visual search, pattern recognition, and image understanding into hybrid biomorphic flyers for future planetary and terrestrial applications. We describe a few future mission scenarios for Mars exploration, uniquely enabled by these newly developed biomorphic flyers.
Bio-inspired reconfigurable stereo vision for robotics using omnidirectional cameras

Suchang Chen,Dongliang Fan,Huijuan Feng,Jian S Dai

2024-10-11

Abstract:This work introduces a novel bio-inspired reconfigurable stereo vision system for robotics, leveraging omnidirectional cameras and a novel algorithm to achieve flexible visual capabilities. Inspired by the adaptive vision of various species, our visual system addresses traditional stereo vision limitations, i.e., immutable camera alignment with narrow fields of view, by introducing a reconfigurable stereo vision system to robotics. Our key innovations include the reconfigurable stereo vision strategy that allows dynamic camera alignment, a robust depth measurement system utilizing a nonrectified geometrical method combined with a deep neural network for feature matching, and a geometrical compensation technique to enhance visual accuracy. Implemented on a metamorphic robot, this vision system demonstrates its great adaptability to various scenarios by switching its configurations of 316° monocular with 79° binocular field for fast target seeking and 242° monocular with 150° binocular field for detailed close inspection.

Robotics
Finding the gap: neuromorphic motion-vision in dense environments

Thorben Schoepe,Ella Janotte,Moritz B. Milde,Olivier J. N. Bertrand,Martin Egelhaaf,Elisabetta Chicca

DOI: https://doi.org/10.1038/s41467-024-45063-y

IF: 16.6

2024-01-27

Nature Communications

Abstract:Abstract Animals have evolved mechanisms to travel safely and efficiently within different habitats. On a journey in dense terrains animals avoid collisions and cross narrow passages while controlling an overall course. Multiple hypotheses target how animals solve challenges faced during such travel. Here we show that a single mechanism enables safe and efficient travel. We developed a robot inspired by insects. It has remarkable capabilities to travel in dense terrain, avoiding collisions, crossing gaps and selecting safe passages. These capabilities are accomplished by a neuromorphic network steering the robot toward regions of low apparent motion. Our system leverages knowledge about vision processing and obstacle avoidance in insects. Our results demonstrate how insects might safely travel through diverse habitats. We anticipate our system to be a working hypothesis to study insects’ travels in dense terrains. Furthermore, it illustrates that we can design novel hardware systems by understanding the underlying mechanisms driving behaviour.

multidisciplinary sciences
Bio-inspired hemispherical compound eye camera

jianliang xiao,young min song,y xie,viktor malyarchuk,inhwa jung,k choi,zhuangjian liu,hyunsung park,chaofeng lu,rakhwan kim,rui li,kenneth b crozier,yonggang huang,john a rogers

DOI: https://doi.org/10.1117/12.2041112

2014-01-01

Abstract:Compound eyes in arthropods demonstrate distinct imaging characteristics from human eyes, with wide angle field of view, low aberrations, high acuity to motion and infinite depth of field. Artificial imaging systems with similar geometries and properties are of great interest for many applications. However, the challenges in building such systems with hemispherical, compound apposition layouts cannot be met through established planar sensor technologies and conventional optics. We present our recent progress in combining optics, materials, mechanics and integration schemes to build fully functional artificial compound eye cameras. Nearly full hemispherical shapes (about 160 degrees) with densely packed artificial ommatidia were realized. The number of ommatidia (180) is comparable to those of the eyes of fire ants and bark beetles. The devices combine elastomeric compound optical elements with deformable arrays of thin silicon photodetectors, which were fabricated in the planar geometries and then integrated and elastically transformed to hemispherical shapes. Imaging results and quantitative ray-tracing-based simulations illustrate key features of operation. These general strategies seem to be applicable to other compound eye devices, such as those inspired by moths and lacewings (refracting superposition eyes), lobster and shrimp (reflecting superposition eyes), and houseflies (neural superposition eyes).
The long-acting walking control of a cockroach bio-bot for vision-aided pipeline navigation

Songsong Ma,Shen Liu,Songlin Yang,Yuansheng Chen,Yao Li,Bing Li

DOI: https://doi.org/10.1109/EMBC40787.2023.10340200

Abstract:Many small bionic crawling robots have been invented for search and rescue missions in narrow spaces. However, their locomotion capability is far from that of insects of the same size. Transforming a cockroach into a bio-bot has been a hot topic in the past decade. Herein, we modified this insect to perform surveillance work in dark confined environments. The synergistic electrical stimulation for turning control was proposed by alternating electrical stimulation of the cerci and antennae every 5 trials. The result showed that this method was able to control cockroaches turning steadily 117 times. An electronic backpack was designed, which was capable of transmitting images in real time, and a light emitting diode (LED) was installed on the backpack providing a light source for the camera. Thus, a vision-aided navigation system was formed for dark confined environments, e.g. pipelines. With a host computer software, the operator controlled the bio-bot to pass through a completely dark and closed pipeline. The electronic backpack and the host computer were connected via transmission control protocol (TCP), which allows the operator to manipulate the bio-robot remotely. This technology can be applied in pipeline surveillance in the future.

Wireless steerable vision for live insects and insect-scale robots

From insect vision to robot vision

A wearable wireless platform for visually stimulating small flying insects

Target Detection and Localization for Flapping-Wing Robots Based on Biological Eagle-Eye Vision

Millisecond insect tracking system

Real-time and Precise Insect Flight Control System Based on Virtual Reality

Living IoT: A Flying Wireless Platform on Live Insects

Synthetic Neural Vision System Design for Motion Pattern Recognition in Dynamic Robot Scenes

Steering Motion Control of a Snake Robot Via a Biomimetic Approach

Visually Guided Swarm Motion Coordination via Insect-inspired Small Target Motion Reactions

An autonomous robot inspired by insect neurophysiology pursues moving features in natural environments

A 116.4 Mg Wireless Ultra-Lightweight Microstimulator for Cyborg Bees

Arthropod Eye-Inspired Digital Camera With Unique Imaging Characteristics

Visual guidance based on optic flow: a biorobotic approach

A self-sustaining wireless sensing and flight control device for beetles

High Resolution Outdoor Videography of Insects Using Fast Lock-On Tracking

Bioinspired engineering of exploration systems for NASA and DoD

Bio-inspired reconfigurable stereo vision for robotics using omnidirectional cameras

Finding the gap: neuromorphic motion-vision in dense environments

Bio-inspired hemispherical compound eye camera

The long-acting walking control of a cockroach bio-bot for vision-aided pipeline navigation