Generative interpolation and restoration of images using deep learning for improved 3D tissue mapping

Saurabh Joshi,André Forjaz,Kyu Sang Han,Yu Shen,Vasco Queiroga,Daniel Xenes,Jordan Matelsk,Brock Wester,Arrate Munoz Barrutia,Ashley L. Kiemen,Pei-Hsun Wu,Denis Wirtz

DOI: https://doi.org/10.1101/2024.03.07.583909

2024-03-28

Abstract:The development of novel imaging platforms has improved our ability to collect and analyze large three-dimensional (3D) biological imaging datasets. Advances in computing have led to an ability to extract complex spatial information from these data, such as the composition, morphology, and interactions of multi-cellular structures, rare events, and integration of multi-modal features combining anatomical, molecular, and transcriptomic (among other) information. Yet, the accuracy of these quantitative results is intrinsically limited by the quality of the input images, which can contain missing or damaged regions, or can be of poor resolution due to mechanical, temporal, or financial constraints. In applications ranging from intact imaging (e.g. light-sheet microscopy and magnetic resonance imaging) to sectioning based platforms (e.g. serial histology and serial section transmission electron microscopy), the quality and resolution of imaging data has become paramount. Here, we address these challenges by leveraging frame interpolation for large image motion (FILM), a generative AI model originally developed for temporal interpolation, for spatial interpolation of a range of 3D image types. Comparative analysis demonstrates the superiority of FILM over traditional linear interpolation to produce functional synthetic images, due to its ability to better preserve biological information including microanatomical features and cell counts, as well as image quality, such as contrast, variance, and luminance. FILM repairs tissue damages in images and reduces stitching artifacts. We show that FILM can decrease imaging time by synthesizing skipped images. We demonstrate the versatility of our method with a wide range of imaging modalities (histology, tissue-clearing/light-sheet microscopy, magnetic resonance imaging, serial section transmission electron microscopy), species (human, mouse), healthy and diseased tissues (pancreas, lung, brain), staining techniques (IHC, H&E), and pixel resolutions (8 nm, 2 µm, 1mm). Overall, we demonstrate the potential of generative AI in improving the resolution, throughput, and quality of biological image datasets, enabling improved 3D imaging.

Bioinformatics

What problem does this paper attempt to address?

This paper mainly explores the use of deep learning techniques to solve the problems of image generation, interpolation, and restoration in 3D biological tissue mapping. The study presents a generative AI model called FILM (Frame Interpolation for Large Image Motion), which was originally used for temporal interpolation in videos and is now applied to spatial interpolation to enhance the quality and resolution of various types of 3D images.

In 3D bioimaging, image quality issues often arise due to mechanical, temporal, and financial constraints, or due to damage or insufficient resolution. The FILM model improves the details of microanatomical structures, reduces damage and stitching errors, and reduces the loss of details caused by insufficient resolution by generating intermediate images to compensate for lost information.

Compared to traditional linear interpolation, FILM demonstrates superiority in preserving biological information such as microanatomical features and cell count, as well as contrast, variance, and brightness. It can repair tissue damage, reduce stitching artifacts, and even reduce imaging time through synthetic images. The paper showcases the applications of FILM in different imaging modalities (such as tissue staining, light-sheet microscopy, magnetic resonance imaging, and transmission electron microscopy), species (human and mouse), healthy and diseased tissues (pancreas, lung, brain), and different pixel resolutions.

In addition, the paper introduces the data source of the dataset, sample acquisition, image segmentation methods, and evaluation metrics, including Pearson correlation coefficient and Haralick texture features, to quantitatively compare the performance of FILM with other interpolation methods. Experimental results demonstrate the potential of FILM in restoring and enhancing 3D biological image data, improving resolution, throughput, and image quality, and helping to improve 3D imaging analysis.