Abstract:Tracking microrobots is challenging, considering their minute size and high speed. As the field progresses towards developing microrobots for biomedical applications and conducting mechanistic studies in physiologically relevant media (e.g., collagen), this challenge is exacerbated by the dense surrounding environments with feature size and shape comparable to microrobots. Herein, we report Motion Enhanced Multi-level Tracker (MEMTrack), a robust pipeline for detecting and tracking microrobots using synthetic motion features, deep learning-based object detection, and a modified Simple Online and Real-time Tracking (SORT) algorithm with interpolation for tracking. Our object detection approach combines different models based on the object's motion pattern. We trained and validated our model using bacterial micro-motors in collagen (tissue phantom) and tested it in collagen and aqueous media. We demonstrate that MEMTrack accurately tracks even the most challenging bacteria missed by skilled human annotators, achieving precision and recall of 77% and 48% in collagen and 94% and 35% in liquid media, respectively. Moreover, we show that MEMTrack can quantify average bacteria speed with no statistically significant difference from the laboriously-produced manual tracking data. MEMTrack represents a significant contribution to microrobot localization and tracking, and opens the potential for vision-based deep learning approaches to microrobot control in dense and low-contrast settings. All source code for training and testing MEMTrack and reproducing the results of the paper have been made publicly available <a class="link-external link-https" href="https://github.com/sawhney-medha/MEMTrack" rel="external noopener nofollow">this https URL</a>.

High-resolution and High-Speed 3D Tracking of Microrobots Using a Fluorescent Light Field Microscope

Fluorescence Interference Structured Illumination Microscopy for 3D Morphology Imaging with High Axial Resolution

Real-Time Super-Resolution Imaging By High-Speed Fluorescence Emission Difference Microscopy

3D Navigation Control of Untethered Magnetic Microrobot in Centimeter-Scale Workspace Based on Field-of-View Tracking Scheme

X-ray Fluoroscopy Guided Localization and Steering of Medical Microrobots through Virtual Enhancement

High-resolution 3D reconstruction of microtubule structures by quantitative multi-angle total internal reflection fluorescence microscopy

MEMTRACK: A Deep Learning-Based Approach to Microrobot Tracking in Dense and Low-Contrast Environments

Submillimeter Magnetic Microrobot Tracking Using an Integrated Ultrasound and Photoacoustic Imaging System

Photoacoustic Imaging to Track Magnetic-manipulated Micro-Robots in Deep Tissue

Three-dimensional Tracking of GLUT4 Vesicles in TIRF Microscopy

Lagrangian 3D tracking of fluorescent microscopic objects in motion

Real-time imaging and tracking of microrobots in tissues using ultrasound phase analysis

Fast 3D cell tracking with wide-field fluorescence microscopy through deep learning

3D particle tracking using a dual-objective fluorescent reflection system with spherical aberration

Optical Flow and Acoustic Phase Analysis Comparison in Ultrasound-Based Microrobot Tracking

Multi-Object Portion Tracking in 4D Fluorescence Microscopy Imagery with Deep Feature Maps

Label-free 3D tracking of biological micro-objects using RGB pinholes

Mirror-enhanced Scanning Light-Field Microscopy for Long-Term High-Speed 3D Imaging with Isotropic Resolution

Amoebic liver abscess in patients presenting with jaundice. A report of 5 cases.

Micro-rocket robot with all-optic actuating and tracking in blood

Field-dependent deep learning enables high-throughput whole-cell 3D super-resolution imaging