Abstract:Optical microrobots are activated by a laser in a liquid medium using optical tweezers. To create visual control loops for robotic automation, this work describes a deep learning-based method for orientation estimation of optical microrobots, focusing on detecting 3-D rotational movements and localizing microrobots and trapping points (TPs). We integrated and fine-tuned You Only Look Once (YOLOv7) and Deep Simple Online Real-time Tracking (DeepSORT) algorithms, improving microrobot and TP detection accuracy by % and %, respectively, at the 0.95 Intersection over Union (IoU) threshold in our test set. Additionally, it increased mean average precision (mAP) by 3% at the 0.5:0.95 IoU threshold during training. Our results showed a 99% success rate in trapping events with no false-positive detection. We introduced a model that employs EfficientNet as a feature extractor combined with custom convolutional neural networks (CNNs) and feature fusion layers. To demonstrate its generalization ability, we evaluated the model on an independent in-house dataset comprising 4,757 image frames, where microrobots executed simultaneous rotations across all three axes. Our method provided mean rotation angle errors of , , and for X (yaw), Y (roll), and Z (pitch) axes, respectively. Compared to pre-trained models, our model provided the lowest error in the Y and Z axes while offering competitive results for X-axis. Finally, we demonstrated the explainability and transparency of the model's decision-making process. Our work contributes to the field of microrobotics by providing an efficient 3-axis orientation estimation pipeline, with a clear focus on automation.

Depth Estimation of Optically Transparent Laser-Driven Microrobots

Data-Driven Microscopic Pose and Depth Estimation for Optical Microrobot Manipulation

Depth Estimation of Optically Transparent Microrobots Using Convolutional and Recurrent Neural Networks

Three-Dimensional Pose Estimation of Optically Transparent Microrobots

Calibration-free Deep Optics for Depth Estimation with Precise Simulation

Three-dimensional optical microrobot orientation estimation and tracking using deep learning

Gaze contingent control for optical micromanipulation.

MICROMANIPULATION ROBOT DEPTH EXTRACTION BASED ON MICRO IMAGE DEFOCUS FEATURE

High-resolution and High-Speed 3D Tracking of Microrobots Using a Fluorescent Light Field Microscope

Three-Dimensional Tracking at Micro-scale Using a Single Optical Microscope

X-ray Fluoroscopy Guided Localization and Steering of Medical Microrobots through Virtual Enhancement

Robotic Cell Manipulation Using Optical Tweezers with Unknown Trapping Stiffness and Limited FOV

3D Navigation Control of Untethered Magnetic Microrobot in Centimeter-Scale Workspace Based on Field-of-View Tracking Scheme

Autonomous object harvesting using synchronized optoelectronic microrobots

Abstract—Optoelectronic tweezer-driven microrobots (OETdMs) are a versatile micromanipulation technology based on the application of light induced dielectrophoresis to move

KineDepth: Utilizing Robot Kinematics for Online Metric Depth Estimation

Depth Estimation Method for Monocular Camera Defocus Images in Microscopic Scenes

Adaptive Autonomous Navigation of Multiple Optoelectronic Microrobots in Dynamic Environments

Depth measurement in the direction of optical axis based on defocused micro-manipulation system

Observer-Based Optical Manipulation of Biological Cells With Robotic Tweezers

LucidGrasp: Robotic Framework for Autonomous Manipulation of Laboratory Equipment with Different Degrees of Transparency via 6D Pose Estimation