Abstract:Microrobotic contact manipulation enables automated and precise cell capture, positioning and screening and has potential in biomedical engineering and disease detection. However, when using an optical microscope for visual positioning of targets, the poor clarity, limited cell-background contrast and lack of global 3D information of the cells in the field of view hinder global strategy making and automation, thereby affecting the accuracy and efficiency of manipulation. Here, we propose the 3D locating of multiple biological targets based on digital holography. Global–local combined visual feedback is developed for overall spatial locating and partial locating in a liquid-phase bright-field environment. By applying a filtering-based planar locating algorithm and maximum-area-based depth detection algorithm, the 3D global distribution of micro-targets in the field of view is periodically updated with a high detection rate. By applying a planar locating algorithm based on a convolutional neural network and a depth detection algorithm based on a gradient descent, the 3D fast locating of targets is performed precisely. Experiments show that the detection rate of the global positioning is 95.1%, the mean average precision of the local planar positioning is 90.53%, and the deviation of the local depth positioning is 1.22 $\mu$ m. When capturing cells, this method reaches an average speed of 7.4 cells/min and a collection rate of 90.5%. We anticipate that our method will support the research in cell-based bioengineering including cell screening and early disease diagnosis. Note to Practitioners—Automated cell manipulation is one of the most significant techniques in cell-based biomedical applications. This paper introduces a three-dimensional multi-target visual positioning method for automated cell manipulation. Combining with holographic imaging technique, the global information for the cells in the limited field of view is provided with improved imaging clarity and extended depth of field. The visual recognition algorithm can screen and extract the rare cells in the cell population, which will support the research in the field of early disease diagnosis and biomedicine. The research outcome provides an effective and precise solution to achieve biological targets positioning, capture, and screening within 3D liquid-phase bright-field environment.

Simultaneous Depth Estimation and Localization for Cell Manipulation Based on Deep Learning.

Single-shot Autofocusing in Light Sheet Fluorescence Microscopy with Multiplexed Structured Illumination and Deep Learning

Digital Holography Based Three-Dimensional Multi-Target Locating for Automated Cell Micromanipulation

Extraction of Key Foreground Information from Visual Feedback Images for Contact Micromanipulation in Liquid Environment

Field-dependent deep learning enables high-throughput whole-cell 3D super-resolution imaging

A Deep-Learning-based Workflow to Deal with the Defocusing Problem in High-Throughput Experiments

Fast 3D cell tracking with wide-field fluorescence microscopy through deep learning

Deep-Learning-Based Polar-Body Detection For Automatic Cell Manipulation

Automatic Detection of Three Cell Types in a Microscope Image Based on Deep Learning

Learn to segment single cells with deep distance estimator and deep cell detector

Autofocus System for Imaging Multiple Cells Across Thick Liquid Layers in Differrent Focal Planes

Precision Alignment in Cell Microinjection Based on Hybrid Triple-View Micro-Vision

Extraction of Depth Information and Image Processing in Manipulator Cell Injection

Robust deep learning optical autofocus system applied to automated multiwell plate single molecule localization microscopy

Deep Learning-Based Single-Shot Autofocus Method for Digital Microscopy

A Robot-Assisted Cell Manipulation System with an Adaptive Visual Servoing Method

A Deep Local Patch Matching Network for Cell Tracking in Microscopy Image Sequences Without Registration

MICROMANIPULATION ROBOT DEPTH EXTRACTION BASED ON MICRO IMAGE DEFOCUS FEATURE

Cell Tracking Using Deep Neural Networks with Multi-Task Learning

Deep Learning-Enhanced Dual-Module Large-Throughput Microinjection System for Adherent Cells

Particle Field Positioning with a Commercial Microscope Based on a Developed CNN and the Depth-from-defocus Method