3D Single Molecule Localization Microscopy Via Deep Learning

Xiaoxu Li,Runzhao Yang,Yuchen Guo,Hui Qiao,Qionghai Dai

DOI: https://doi.org/10.1364/microscopy.2020.mw2a.4

2020-01-01

Abstract:We propose a deep-learning based approach to estimate 3D locations of multiple emitters over a large axial range with high accuracy. Localization performance is validated by several experiments on public benchmarks.

What problem does this paper attempt to address?

Sub-Diffraction Dark Spot Localization Microscopy

Chuankang Li,Yuzhu Li,Zhengyi Zhan,Yuhang Li,Xin Liu,Yong Liu,Xiang Hao,Cuifang Kuang,Xu Liu

DOI: https://doi.org/10.1364/PRJ.429933

IF: 7.6

2021-01-01

Photonics Research

Abstract:Single molecular localization microscopy (SMLM) is a useful tool in biological observation with sub-10-nm resolution. However, SMLM is incapable of discerning two molecules within the diffraction-limited region unless with the help of a stochastic on-off switching scheme which yet entails time-consuming processes. Here, we produce a novel kind of focal spot pattern, called sub-diffraction dark spot (SDS), to localize molecules within the sub-diffraction region of interest. In our proposed technique nominated as sub-diffracted dark spot localization microscopy (SDLM), multiple molecules within the diffraction-limited region could be distinguished without the requirement of stochastic fluorescent switches. We have numerically investigated some related impacts of SDLM, such as detection circle diameter, collected photon number, background noise, and SDS size. Simulative localization framework has been implemented on randomly distributed and specifically structured samples. In either two- or three-dimensional case, SDLM is evidenced to have similar to 2 nmlocalization accuracy. (C) 2021 Chinese Laser Press
Field-dependent deep learning enables high-throughput whole-cell 3D super-resolution imaging

Shuang Fu,Wei Shi,Tingdan Luo,Yingchuan He,Lulu Zhou,Jie Yang,Zhichao Yang,Jiadong Liu,Xiaotian Liu,Zhiyong Guo,Chengyu Yang,Chao Liu,Zhen-li Huang,Jonas Ries,Mingjie Zhang,Peng Xi,Dayong Jin,Yiming Li

DOI: https://doi.org/10.1038/s41592-023-01775-5

IF: 48

2023-02-25

Nature Methods

Abstract:Single-molecule localization microscopy in a typical wide-field setup has been widely used for investigating subcellular structures with super resolution; however, field-dependent aberrations restrict the field of view (FOV) to only tens of micrometers. Here, we present a deep-learning method for precise localization of spatially variant point emitters (FD-DeepLoc) over a large FOV covering the full chip of a modern sCMOS camera. Using a graphic processing unit-based vectorial point spread function (PSF) fitter, we can fast and accurately model the spatially variant PSF of a high numerical aperture objective in the entire FOV. Combined with deformable mirror-based optimal PSF engineering, we demonstrate high-accuracy three-dimensional single-molecule localization microscopy over a volume of ~180 × 180 × 5 μm 3 , allowing us to image mitochondria and nuclear pore complexes in entire cells in a single imaging cycle without hardware scanning; a 100-fold increase in throughput compared to the state of the art.

biochemical research methods
Deep learning-based spectroscopic single-molecule localization microscopy

Sunil Kumar Gaire,Ali Daneshkhah,Ethan Flowerday,Ruyi Gong,Jane Frederick,Vadim Backman

DOI: https://doi.org/10.1117/1.jbo.29.6.066501

IF: 3.758

2024-05-26

Journal of Biomedical Optics

Abstract:SignificanceSpectroscopic single-molecule localization microscopy (sSMLM) takes advantage of nanoscopy and spectroscopy, enabling sub-10 nm resolution as well as simultaneous multicolor imaging of multi-labeled samples. Reconstruction of raw sSMLM data using deep learning is a promising approach for visualizing the subcellular structures at the nanoscale.AimDevelop a novel computational approach leveraging deep learning to reconstruct both label-free and fluorescence-labeled sSMLM imaging data.ApproachWe developed a two-network-model based deep learning algorithm, termed DsSMLM, to reconstruct sSMLM data. The effectiveness of DsSMLM was assessed by conducting imaging experiments on diverse samples, including label-free single-stranded DNA (ssDNA) fiber, fluorescence-labeled histone markers on COS-7 and U2OS cells, and simultaneous multicolor imaging of synthetic DNA origami nanoruler.ResultsFor label-free imaging, a spatial resolution of 6.22 nm was achieved on ssDNA fiber; for fluorescence-labeled imaging, DsSMLM revealed the distribution of chromatin-rich and chromatin-poor regions defined by histone markers on the cell nucleus and also offered simultaneous multicolor imaging of nanoruler samples, distinguishing two dyes labeled in three emitting points with a separation distance of 40 nm. With DsSMLM, we observed enhanced spectral profiles with 8.8% higher localization detection for single-color imaging and up to 5.05% higher localization detection for simultaneous two-color imaging.ConclusionsWe demonstrate the feasibility of deep learning-based reconstruction for sSMLM imaging applicable to label-free and fluorescence-labeled sSMLM imaging data. We anticipate our technique will be a valuable tool for high-quality super-resolution imaging for a deeper understanding of DNA molecules' photophysics and will facilitate the investigation of multiple nanoscopic cellular structures and their interactions.

radiology, nuclear medicine & medical imaging,biochemical research methods,optics
High-axial-resolution Single-Molecule Localization under Dense Excitation with a Multi-Channel Deep U-Net.

Weihang Zhang,Zhihong Zhang,Liheng Bian,Haoqian Wang,Jinli Suo,Qionghai Dai

DOI: https://doi.org/10.1364/ol.441536

IF: 3.6

2021-01-01

Optics Letters

Abstract:Single-molecule localization microscopy (SMLM) can bypass the diffraction limit of optical microscopes and greatly improve the resolution in fluorescence microscopy. By introducing the point spread function (PSF) engineering technique, we can customize depth varying PSF to achieve higher axial resolution. However, most existing 3D single-molecule localization algorithms require excited fluorescent molecules to be sparse and captured at high signal-to-noise ratios, which results in a long acquisition time and precludes SMLM's further applications in many potential fields. To address this problem, we propose a novel 3D single-molecular localization method based on a multi-channel neural network based on U-Net. By leveraging the deep network's great advantages in feature extraction, the proposed network can reliably discriminate dense fluorescent molecules with overlapped PSFs and corrupted by sensor noise. Both simulated and real experiments demonstrate its superior performance in PSF engineered microscopes with short exposure and dense excitations, which holds great potential in fast 3D super-resolution microscopy.
DeepSTORM3D: dense 3D localization microscopy and PSF design by deep learning

Elias Nehme,Daniel Freedman,Racheli Gordon,Boris Ferdman,Lucien E. Weiss,Onit Alalouf,Tal Naor,Reut Orange,Tomer Michaeli,Yoav Shechtman

DOI: https://doi.org/10.1038/s41592-020-0853-5

IF: 48

2020-06-15

Nature Methods

Abstract:An outstanding challenge in single-molecule localization microscopy is the accurate and precise localization of individual point emitters in three dimensions in densely labeled samples. One established approach for three-dimensional single-molecule localization is point-spread-function (PSF) engineering, in which the PSF is engineered to vary distinctively with emitter depth using additional optical elements. However, images of dense emitters, which are desirable for improving temporal resolution, pose a challenge for algorithmic localization of engineered PSFs, due to lateral overlap of the emitter PSFs. Here we train a neural network to localize multiple emitters with densely overlapping Tetrapod PSFs over a large axial range. We then use the network to design the optimal PSF for the multi-emitter case. We demonstrate our approach experimentally with super-resolution reconstructions of mitochondria and volumetric imaging of fluorescently labeled telomeres in cells. Our approach, DeepSTORM3D, enables the study of biological processes in whole cells at timescales that are rarely explored in localization microscopy.

biochemical research methods
Digital-SMLM for precisely localizing emitters within the diffraction limit

Zhe Jia,Lingxiao Zhou,Haoyu Li,Jielei Ni,Danni Chen,Dongfei Guo,Bo Cao,Gang Liu,Guotao Liang,Qianwen Zhou,Xiaocong Yuan,Yanxiang Ni

DOI: https://doi.org/10.1515/nanoph-2023-0936

IF: 7.5

2024-06-06

Nanophotonics

Abstract:Precisely pinpointing the positions of emitters within the diffraction limit is crucial for quantitative analysis or molecular mechanism investigation in biomedical research but has remained challenging unless exploiting single molecule localization microscopy (SMLM). Via integrating experimental spot dataset with deep learning, we develop a new approach, Digital-SMLM, to accurately predict emitter numbers and positions for sub-diffraction-limit spots with an accuracy of up to 98 % and a root mean square error as low as 14 nm. Digital-SMLM can accurately resolve two emitters at a close distance, e.g. 30 nm. Digital-SMLM outperforms Deep-STORM in predicting emitter numbers and positions for sub-diffraction-limited spots and recovering the ground truth distribution of molecules of interest. We have validated the generalization capability of Digital-SMLM using independent experimental data. Furthermore, Digital-SMLM complements SMLM by providing more accurate event number and precise emitter positions, enabling SMLM to closely approximate the natural state of high-density cellular structures.

optics,physics, applied,materials science, multidisciplinary,nanoscience & nanotechnology
Localization of Ultra-dense Emitters with Neural Networks

Armin Abdehkakha,Craig Snoeyink

2023-05-08

Abstract:Single-Molecule Localization Microscopy (SMLM) has expanded our ability to visualize subcellular structures but is limited in its temporal resolution. Increasing emitter density will improve temporal resolution, but current analysis algorithms struggle as emitter images significantly overlap. Here we present a deep convolutional neural network called LUENN which utilizes a unique architecture that rejects the isolated emitter assumption; it can smoothly accommodate emitters that range from completely isolated to co-located. This architecture, alongside an accurate estimator of location uncertainty, extends the range of usable emitter densities by a factor of 6 to over 31 emitters per micrometer-squared with reduced penalty to localization precision and improved temporal resolution. Apart from providing uncertainty estimation, the algorithm improves usability in laboratories by reducing imaging times and easing requirements for successful experiments.

Signal Processing,Computer Vision and Pattern Recognition,Machine Learning,Data Analysis, Statistics and Probability,Fluid Dynamics,Optics,Computation
Deep Learning for Precise Axial Localization of Trapped Microspheres in Reflective Optical Systems.

Mengdi Guo,Zuzeng Lin,Cong Zhai,Yulu Chen,Xiaowei Zhou,Zhongyan Chai,Tong Guo,Chunguang Hu

DOI: https://doi.org/10.1364/oe.487866

IF: 3.8

2023-01-01

Optics Express

Abstract:High-precision axial localization measurement is an important part of micro-nanometer optical measurement, but there have been issues such as low calibration efficiency, poor accuracy, and cumbersome measurement, especially in reflected light illumination systems, where the lack of clarity of imaging details leads to the low accuracy of commonly used methods. Herein, we develop a trained residual neural network coupled with a convenient data acquisition strategy to address this challenge. Our method improves the axial localization precision of microspheres in both reflective illumination systems and transmission illumination systems. Using this new localization method, the reference position of the trapped microsphere can be extracted from the identification results, namely the "positioning point" among the experimental groups. This point relies on the unique signal characteristics of each sample measurement, eliminates systematic repeatability errors when performing identification across samples, and improves the localization precision of different samples. This method has been verified on both transmission and reflected illumination optical tweezers platforms. We will bring greater convenience to measurements in solution environments and will provide higher-order guarantees for force spectroscopy measurements in scenarios such as microsphere-based super-resolution microscopy and the surface mechanical properties of adherent flexible materials and cells.
Enhancing fluorescence microscopy performance with Richardson-Lucy based deep learning

Yue Li,Yijun Su,Huafeng Liu,Yicong Wu,Hari Shroff

DOI: https://doi.org/10.1364/3d.2022.jw5b.3

2022-01-01

Abstract:We present a deep learning method which incorporates the Richardson-Lucy formula into a fully convolutional network and demonstrate its performance on cells, tissues, and embryos imaged with widefield-, confocal- light sheet-, and super-resolution microscopy.
Localizing Axial Dense Emitters Based on Single-Helix Point Spread Function and Deep Learning

Yihong Ji,Danni Chen,Hanzhe Wu,Gan Xiang,Heng Li,Bin Yu,Junle Qu

DOI: https://doi.org/10.1109/jphot.2024.3476514

IF: 2.4

2024-11-02

IEEE Photonics Journal

Abstract:The point-by point 3D scanning strategy adopted in Stimulated Emission Depletion Microscopy (STED) is time-consuming. The 3D scanning can be replaced with a 2D scanning in the non-diffracting Bessel-Bessel STED (BB-STED). In order to extract the excited emitters' axial information in BB-STED, we propose to encode axial information by using a detection optical path with single-helix PSF, and then predict the depths of the emitters with deep learning. Simulation demonstrated that, for dense emitters in a depth range of 4 μm, an axial precision of ∼35 nm can be achieved. Our method also works for experimental data, and an axial precision of ∼63 nm can be achieved.

engineering, electrical & electronic,optics,physics, applied
DeepLFM: Deep Learning-based 3D Reconstruction for Light Field Microscopy

Xiaoxu Li,Hui Qiao,Jiamin Wu,Zhi Lu,Tao Yan,Ruxin Zhang,Xu Zhang,Qionghai Dai

DOI: https://doi.org/10.1364/ntm.2019.nm3c.2

2019-01-01

Abstract:We propose a high-resolution 3D reconstruction method for light field microscopy via deep learning. Experimental results on K562 cells verify its superior performance, which exhibit less artifacts especially near the native object plane.
Deep Learning for Super-Resolution Localization Microscopy

Tianyang Zhou,Jianwen Luo,Xin Liu

DOI: https://doi.org/10.1117/12.2500832

2018-01-01

Abstract:Super-resolution localization microscopy techniques (e.g., STORM or PALM), breaks the optics diffraction limit, making possible the observation of sub-cellular structures in vivo. However, long acquisition time is required to maintain a desired high spatial resolution. To overcome the limitation, an effective method is to increase the density of activated emitters in each frame. The high-density emitters will cause them to overlap, which makes it difficult to accurately resolve each emitter location. Although some methods have been proposed to identify the overlapped emitters, these methods are computationally intensive and parameter dependent. To address these problems, in this paper, we proposed a novel method based on convolutional neural networks (CNN) for super-resolution localization microscopy, termed as DL-SRLM. DL-SRLM is capable of learning the nonlinear mapping between a camera frame (i.e., the experimentally acquired low-resolution image) and the true locations of emitters in the corresponding image region (i.e., the recovered super-resolution image). As a result, the method provides the possibility to faster resolve the high-density emitters, without requiring the parameters. To evaluate the performance of DL-SRLM, a series of simulations with varying emitter densities, signal-to-noise ratios (SNRs), and point spread functions (PSFs) were performed. The results show that DL-SRLM can accurately resolve the locations of high-density emitters, even if when the raw measurement data contained noise or was generated by using inaccurate PSF. In addition, DL-SRLM greatly improve the computational speed (~ 15 ms/frame) compared with the current methods while avoiding the effect of the parameters on the super-resolution imaging performance.
Enhancing the data processing speed of a deep-learning-based three-dimensional single molecule localization algorithm (FD-DeepLoc) with a combination of feature compression and pipeline programming

Shuhao Guo,Jiaxun Lin,Yingjun Zhang,Zhen-Li Huang

DOI: https://doi.org/10.1142/s1793545824500251

IF: 2.396

2024-10-18

Journal of Innovative Optical Health Sciences

Abstract:Journal of Innovative Optical Health Sciences, Ahead of Print. Three-dimensional (3D) single molecule localization microscopy (SMLM) plays an important role in biomedical applications, but its data processing is very complicated. Deep learning is a potential tool to solve this problem. As the state of art 3D super-resolution localization algorithm based on deep learning, FD-DeepLoc algorithm reported recently still has a gap with the expected goal of online image processing, even though it has greatly improved the data processing throughput. In this paper, a new algorithm Lite-FD-DeepLoc is developed on the basis of FD-DeepLoc algorithm to meet the online image processing requirements of 3D SMLM. This new algorithm uses the feature compression method to reduce the parameters of the model, and combines it with pipeline programming to accelerate the inference process of the deep learning model. The simulated data processing results show that the image processing speed of Lite-FD-DeepLoc is about twice as fast as that of FD-DeepLoc with a slight decrease in localization accuracy, which can realize real-time processing of [math] pixels size images. The results of biological experimental data processing imply that Lite-FD-DeepLoc can successfully analyze the data based on astigmatism and saddle point engineering, and the global resolution of the reconstructed image is equivalent to or even better than FD-DeepLoc algorithm.

radiology, nuclear medicine & medical imaging,optics
Deep learning-driven adaptive optics for single-molecule localization microscopy

Peiyi Zhang,Donghan Ma,Xi Cheng,Andy P Tsai,Yu Tang,Hao-Cheng Gao,Li Fang,Cheng Bi,Gary E Landreth,Alexander A Chubykin,Fang Huang

DOI: https://doi.org/10.1038/s41592-023-02029-0

Abstract:The inhomogeneous refractive indices of biological tissues blur and distort single-molecule emission patterns generating image artifacts and decreasing the achievable resolution of single-molecule localization microscopy (SMLM). Conventional sensorless adaptive optics methods rely on iterative mirror changes and image-quality metrics. However, these metrics result in inconsistent metric responses and thus fundamentally limit their efficacy for aberration correction in tissues. To bypass iterative trial-then-evaluate processes, we developed deep learning-driven adaptive optics for SMLM to allow direct inference of wavefront distortion and near real-time compensation. Our trained deep neural network monitors the individual emission patterns from single-molecule experiments, infers their shared wavefront distortion, feeds the estimates through a dynamic filter and drives a deformable mirror to compensate sample-induced aberrations. We demonstrated that our method simultaneously estimates and compensates 28 wavefront deformation shapes and improves the resolution and fidelity of three-dimensional SMLM through >130-µm-thick brain tissue specimens.
3D Localization of Point Source Based on Light Field Imaging and Deep Learning

Shizhu Yuan,Yao Hu,Rui Cao,Chengqiang Xu,Qun Hao,Xuemin Cheng

DOI: https://doi.org/10.1117/12.2541587

2020-01-01

Abstract:3D localization of point source is widely used in many fields, such as bioimaging and autonomous driving fields. However, the localization is hard to perform under scattering conditions because of the diffuse effect of the scattering. We propose a novel method for 3D localization of point source under scattering conditions based on light field imaging and deep learning by only one shot. First, we introduce the description of the point source in a light field wise and how to localize a point source by its light field. On the basis, we elaborate on the effect of scattering on a light field and how to retrieve the location of a point source from a light field with scattering. Then, the effect of aberration on a light field will be introduced. We also build an artificial scene and a deep learning framework to perform a 3D localization practically, and the feasibility and accuracy of our method have been evaluated.
DeepCEL0 for 2D Single Molecule Localization in Fluorescence Microscopy

Pasquale Cascarano,Maria Colomba Comes,Andrea Sebastiani,Arianna Mencattini,Elena Loli Piccolomini,Eugenio Martinelli

DOI: https://doi.org/10.48550/arXiv.2107.02281

2021-07-06

Abstract:In fluorescence microscopy, Single Molecule Localization Microscopy (SMLM) techniques aim at localizing with high precision high density fluorescent molecules by stochastically activating and imaging small subsets of blinking emitters. Super Resolution (SR) plays an important role in this field since it allows to go beyond the intrinsic light diffraction limit. In this work, we propose a deep learning-based algorithm for precise molecule localization of high density frames acquired by SMLM techniques whose $\ell_{2}$-based loss function is regularized by positivity and $\ell_{0}$-based constraints. The $\ell_{0}$ is relaxed through its Continuous Exact $\ell_{0}$ (CEL0) counterpart. The arising approach, named DeepCEL0, is parameter-free, more flexible, faster and provides more precise molecule localization maps if compared to the other state-of-the-art methods. We validate our approach on both simulated and real fluorescence microscopy data.

Machine Learning,Image and Video Processing,Numerical Analysis
Model-free machine learning-based 3D single molecule localisation microscopy

Miguel d'Arcangues Boland,Seth Flaxman,Jonathan P.E. Lightley,Edwin Garcia,Sunil Kumar,Chris Dunsby,Mark A. A. Neil,Paul P. W. French,Edward A.K. Cohen

DOI: https://doi.org/10.1101/2024.10.14.618179

2024-10-15

Abstract:Single molecule localisation microscopy (SMLM) can provide two-dimensional super-resolved image datafrom conventional fluorescence microscopes, while three dimensional (3D) SMLM usually involves amodification of the microscope, e.g. to engineer a predictable axial variation in the point spread function. Herewe demonstrate a 3D SMLM approach (we call easyZloc) utilising a lightweight Convolutional NeuralNetwork that is generally applicable, including with standard (unmodified) fluorescence microscopes, andwhich we consider may be practically useful in a high throughput SMLM workflow. We demonstrate thereconstruction of nuclear pore complexes with comparable performance to previously reported methods butwith a significant reduction in computational power and execution time. 3D reconstructions of the nuclearenvelope and a tubulin sample over a larger axial range are also shown.

Biophysics
Deep-learning Light-Field Microscopy with Improved Resolution and Reconstruction Speed

Guo Li,Zhaoqiang Wang,Hao Zhang,Lanxin Zhu,Peng Fei

DOI: https://doi.org/10.1364/translational.2020.jw5a.1

2020-01-01

Abstract:We demonstrate a deep-learning based light-field microscopy method, which achieves greatly improved spatial resolution and video-rate reconstruction throughput, thereby allowing sustained volumetric recording of fast biological dynamics at high spatiotemporal resolution.
Fast 3D cell tracking with wide-field fluorescence microscopy through deep learning

Kan Liu,Hui Qiao,Jiamin Wu,Haoqian Wang,Lu Fang,Qionghai Dai

DOI: https://doi.org/10.48550/arXiv.1805.05139

2018-05-15

Abstract:Tracking cells in 3D at high speed continues to attract extensive attention for many biomedical applications, such as monitoring immune cell migration and observing tumor metastasis in flowing blood vessels. Here, we propose a deep convolutional neural networks (CNNs) based method to retrieve the 3D locations of the fluorophores from a single 2D image captured by a conventional wide-field fluorescence microscope without any hardware modification. The reported method converts the challenging 3D localization from an ill-posed model-based fitting problem, especially with dense samples and low signal-to-noise ratio, to a solvable multi-label classification problem through two cascaded CNNs, where deep learning technique has a great advantage over other algorithms. Compared with traditional kernel-fitting methods, the proposed method achieves more accurate and robust localization of multiple objects across a much larger axial range, which is validated by both simulation and experimental results on 3D distributed fluorescent beads. Moreover, in vivo 3D tracking of multiple blood cells in zebrafish at 100 fps further verifies the feasibility of our framework.

Optics

3D Single Molecule Localization Microscopy Via Deep Learning

Sub-Diffraction Dark Spot Localization Microscopy

Field-dependent deep learning enables high-throughput whole-cell 3D super-resolution imaging

Deep learning-based spectroscopic single-molecule localization microscopy

High-axial-resolution Single-Molecule Localization under Dense Excitation with a Multi-Channel Deep U-Net.

DeepSTORM3D: dense 3D localization microscopy and PSF design by deep learning

Digital-SMLM for precisely localizing emitters within the diffraction limit

Localization of Ultra-dense Emitters with Neural Networks

Deep Learning for Precise Axial Localization of Trapped Microspheres in Reflective Optical Systems.

Enhancing fluorescence microscopy performance with Richardson-Lucy based deep learning

Localizing Axial Dense Emitters Based on Single-Helix Point Spread Function and Deep Learning

DeepLFM: Deep Learning-based 3D Reconstruction for Light Field Microscopy

Deep Learning for Super-Resolution Localization Microscopy

Enhancing the data processing speed of a deep-learning-based three-dimensional single molecule localization algorithm (FD-DeepLoc) with a combination of feature compression and pipeline programming

Deep learning-driven adaptive optics for single-molecule localization microscopy

3D Localization of Point Source Based on Light Field Imaging and Deep Learning

DeepCEL0 for 2D Single Molecule Localization in Fluorescence Microscopy

Model-free machine learning-based 3D single molecule localisation microscopy

Deep-learning Light-Field Microscopy with Improved Resolution and Reconstruction Speed

Fast 3D cell tracking with wide-field fluorescence microscopy through deep learning