Convolutional Neural Network Transformer (CNNT) for Fluorescence Microscopy image Denoising with Improved Generalization and Fast Adaptation

Azaan Rehman,Alexander Zhovmer,Ryo Sato,Yosuke Mukoyama,Jiji Chen,Alberto Rissone,Rosa Puertollano,Harshad Vishwasrao,Hari Shroff,Christian A. Combs,Hui Xue

2024-04-07

Abstract:Deep neural networks have been applied to improve the image quality of fluorescence microscopy imaging. Previous methods are based on convolutional neural networks (CNNs) which generally require more time-consuming training of separate models for each new imaging experiment, impairing the applicability and generalization. Once the model is trained (typically with tens to hundreds of image pairs) it can then be used to enhance new images that are like the training data. In this study, we proposed a novel imaging-transformer based model, Convolutional Neural Network Transformer (CNNT), to outperform the CNN networks for image denoising. In our scheme we have trained a single CNNT based backbone model from pairwise high-low SNR images for one type of fluorescence microscope (instance structured illumination, iSim). Fast adaption to new applications was achieved by fine-tuning the backbone on only 5-10 sample pairs per new experiment. Results show the CNNT backbone and fine-tuning scheme significantly reduces the training time and improves the image quality, outperformed training separate models using CNN approaches such as - RCAN and Noise2Fast. Here we show three examples of the efficacy of this approach on denoising wide-field, two-photon and confocal fluorescence data. In the confocal experiment, which is a 5 by 5 tiled acquisition, the fine-tuned CNNT model reduces the scan time form one hour to eight minutes, with improved quality.

Quantitative Methods

What problem does this paper attempt to address?

The paper aims to address the issue of image denoising in fluorescence microscopy imaging and improve the model's generalization ability and rapid adaptation to new experiments.

To achieve this goal, the authors propose a novel Convolutional Neural Network Transformer (CNNT) based on the Transformer architecture. Compared to traditional Convolutional Neural Networks (CNNs), CNNT retains the spatial invariance of CNNs while introducing the attention mechanism from the Transformer architecture, thereby better capturing long-range correlations in images. Additionally, the authors designed a two-stage training method: first, a diverse dataset is used to train a general "backbone model"; then, in new experiments, only a small amount of data is needed to fine-tune this backbone model, allowing for quick adaptation to different microscope types, cell types, and imaging protocols.

Specifically, the study demonstrates the application of CNNT on wide-field, two-photon, and confocal microscopy imaging data. Experimental results show that CNNT not only significantly improves image quality but also greatly reduces training time compared to traditional CNN models (such as 3D-RCAN) and self-supervised models (such as Noise2Fast) trained from scratch. For example, in confocal experiments, the fine-tuned CNNT model can reduce the scanning time from 1 hour to 8 minutes while maintaining or even improving image quality.

In summary, by introducing CNNT and a two-stage training strategy, this paper effectively addresses the limitations of traditional CNN models in terms of generalization ability, rapid adaptability, and training efficiency, providing a more efficient and high-quality solution for fluorescence microscopy image processing.