Abstract:Fluorescence lifetime imaging (FLI) has been receiving increased attention in recent years as a powerful diagnostic technique in biological and medical research. However, existing FLI systems often suffer from a tradeoff between processing speed, accuracy, and robustness. In this paper, we propose a robust approach that enables fast FLI with no degradation of accuracy. The approach is based on a SPAD TCSPC system coupled to a recurrent neural network (RNN) that accurately estimates the fluorescence lifetime directly from raw timestamps without building histograms, thereby drastically reducing transfer data volumes and hardware resource utilization, thus enabling FLI acquisition at video rate. We train two variants of the RNN on a synthetic dataset and compare the results to those obtained using center-of-mass method (CMM) and least squares fitting (LS fitting). Results demonstrate that two RNN variants, gated recurrent unit (GRU) and long short-term memory (LSTM), are comparable to CMM and LS fitting in terms of accuracy, while outperforming them in background noise by a large margin. To explore the ultimate limits of the approach, we derived the Cramer-Rao lower bound of the measurement, showing that RNN yields lifetime estimations with near-optimal precision. Moreover, our FLI model, which is purely trained on synthetic datasets, works well with never-seen-before, real-world data. To demonstrate real-time operation, we have built a FLI microscope based on Piccolo, a 32x32 SPAD sensor developed in our lab. Four quantized GRU cores, capable of processing up to 4 million photons per second, are deployed on a Xilinx Kintex-7 FPGA. Powered by the GRU, the FLI setup can retrieve real-time fluorescence lifetime images at up to 10 frames per second. The proposed FLI system is promising and ideally suited for biomedical applications.

SPIDERweb: a Neural Network approach to spectral phase interferometry

SPIDERweb: a neural network approach to spectral phase interferometry

Untrained Neural Network Enabling Fast and Universal Structured-Illumination Microscopy

Single-frame Noisy Interferogram Phase Retrieval Using an End-to-end Deep Learning Network with Physical Information Constraints

Phase-sensitive optical pulse characterization on a chip via Spectral Phase Interferometry for Direct Electric-Field Reconstruction (SPIDER)

Channeled imaging spectropolarimeter reconstruction by neural networks

Neural-network-powered pulse reconstruction from one-dimensional interferometric cross-correlation traces

One-shot Phase Retrieval Method for Interferometry Using a Hypercolumns Convolutional Neural Network

Real-time X-ray Phase-contrast Imaging Using SPINNet -- A Speckle-based Phase-contrast Imaging Neural Network

Real-time phase-retrieval and wavefront sensing enabled by an artificial neural network

3D-2D Neural Nets for Phase Retrieval in Noisy Interferometric Imaging

Coupling a Recurrent Neural Network to SPAD TCSPC Systems for Real-time Fluorescence Lifetime Imaging

A Sparse Model-Based Network for Interferometric Phase Denoising.

Spectral splitting and concentration of broadband light using neural networks

Sensing analysis of self-mixing and Michelson interferometry with neural-network-based phase extraction

Learned Interferometric Imaging for the SPIDER Instrument

Wavelet Transform Analysis for Phase Reconstruction of Spectral Shearing Interferometry of Ultrashort Optical Pulses

Deep neural networks for efficient phase demodulation in wavelength shifting interferometry

Coordinate-based Neural Network for Fourier Phase Retrieval

Opto-intelligence Spectrometer Using Diffractive Neural Networks