Abstract:We present a novel deep learning approach to reconstruct confocal microscopy stacks from single light field images. To perform the reconstruction, we introduce the LFMNet, a novel neural network architecture inspired by the U-Net design. It is able to reconstruct with high-accuracy a 112x112x57.6$\mu m^3$ volume (1287x1287x64 voxels) in 50ms given a single light field image of 1287x1287 pixels, thus dramatically reducing 720-fold the time for confocal scanning of assays at the same volumetric resolution and 64-fold the required storage. To prove the applicability in life sciences, our approach is evaluated both quantitatively and qualitatively on mouse brain slices with fluorescently labelled blood vessels. Because of the drastic reduction in scan time and storage space, our setup and method are directly applicable to real-time in vivo 3D microscopy. We provide analysis of the optical design, of the network architecture and of our training procedure to optimally reconstruct volumes for a given target depth range. To train our network, we built a data set of 362 light field images of mouse brain blood vessels and the corresponding aligned set of 3D confocal scans, which we use as ground truth. The data set will be made available for research purposes.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is to reconstruct a confocal microscope stack in a single light - field image, so as to significantly reduce the imaging time, lower the storage requirements, and improve the ability of three - dimensional real - time imaging. Specifically, the authors proposed a deep - learning - based method, LFMNet, to reconstruct a volume of 112×112×57.6 µm³ (1287×1287×64 voxels) from a single light - field image with high precision, and the processing time is only 50 milliseconds. Compared with traditional confocal scanning, this method can reduce the scanning time by 720 times and the storage requirements by 64 times at the same volume resolution.
The main contributions of the paper include:
1. **Design of LFMNet**: This is a novel convolutional neural network architecture inspired by the U - Net design. It can efficiently reconstruct a high - quality three - dimensional volume from a single light - field image.
2. **Dataset construction**: The authors constructed a dataset containing 362 light - field images and their corresponding confocal scans for training and validating the model. The images and volumes in these datasets have been aligned to ensure the effectiveness of training.
3. **Optical configuration analysis**: Through a detailed analysis of different optical configurations, the optimal light - field microscope settings were determined to optimize the quality of 3D reconstruction.
4. **Real - time imaging ability**: By significantly reducing the acquisition and reconstruction time, this method is suitable for real - time in - vivo 3D microscopic imaging, especially in applications that require rapid capture of biological processes.
### Main problems
- **Long imaging time**: Traditional confocal microscopes need to scan voxel by voxel, resulting in an overly long imaging time.
- **High storage requirements**: The large amount of data generated by confocal scanning requires a large amount of storage space.
- **Not suitable for fast in - vivo imaging**: Due to spatio - temporal distortion, traditional confocal microscopes are not suitable for imaging rapidly moving samples.
### Solutions
- **Use of light - field microscopes**: Light - field microscopes can acquire 3D information in a single shot, but with lower resolution.
- **Combination with deep learning**: By training LFMNet, a high - resolution 3D volume can be efficiently reconstructed from a single light - field image.
- **Optimization of optical configuration**: Through a detailed analysis of the optical configuration, the optimal light - field microscope settings are found to improve the reconstruction quality.
### Formula presentation
- **Fisher information matrix of the point spread function (PSF)**:
\[
F(p)=\begin{bmatrix}
F_{x_px_p}(p,b)&F_{x_py_p}(p,b)&F_{x_pz_p}(p,b)\\
F_{y_px_p}(p,b)&F_{y_py_p}(p,b)&F_{y_pz_p}(p,b)\\
F_{z_px_p}(p,b)&F_{z_py_p}(p,b)&F_{z_pz_p}(p,b)
\end{bmatrix}
\]
where
\[
F_{ij}(p,b)=\int\left(\frac{\partial^2\ln\hat{h}(x,y,p,b)}{\partial i\partial j}\right)\hat{h}(x,y,p,b)\,dx\,dy
\]
\(i, j\in\{x_p,y_p,z_p\}\), \(\hat{h}(x,y,p,b)\) is the representation of the normalized light - field PSF in the image coordinates \(x, y\), and the distance from the MLA to the sensor is \(b\).
Through these methods, the paper has successfully addressed the deficiencies of traditional confocal microscopes in imaging time and storage requirements, providing a new solution for fast 3D imaging in the life sciences field.