Generative Adversarial Networks Accurately Reconstruct Pan-Cancer Histology from Pathologic, Genomic, and Radiographic Latent Features

Frederick M. Howard,Hanna M. Hieromnimon,Siddhi Ramesh,James Dolezal,Sara Kochanny,Qianchen Zhang,Brad Feiger,Joseph Peterson,Cheng Fan,Charles M. Perou,Jasmine Vickery,Megan Sullivan,Kimberly Cole,Galina Khramtsova,Alexander T. Pearson

DOI: https://doi.org/10.1101/2024.03.22.586306

2024-03-25

Abstract:Artificial intelligence models have been increasingly used in the analysis of tumor histology to perform tasks ranging from routine classification to identification of novel molecular features. These approaches distill cancer histologic images into high-level features which are used in predictions, but understanding the biologic meaning of such features remains challenging. We present and validate a custom generative adversarial network – HistoXGAN – capable of reconstructing representative histology using feature vectors produced by common feature extractors. We evaluate HistoXGAN across 29 cancer subtypes and demonstrate that reconstructed images retain information regarding tumor grade, histologic subtype, and gene expression patterns. We leverage HistoXGAN to illustrate the underlying histologic features for deep learning models for actionable mutations, identify model reliance on histologic batch effect in predictions, and demonstrate accurate reconstruction of tumor histology from radiographic imaging for a ‘virtual biopsy’.

Cancer Biology

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is to accurately reconstruct pan - cancer histological images from the latent features of pathology, genomics and radiology through Generative Adversarial Networks (GANs). Specifically, the paper proposes a customized Generative Adversarial Network named HistoXGAN, which can use the feature vectors generated by common feature extractors to reconstruct representative histological images. The goals of the paper include:

1. **Improve the accuracy of image reconstruction**: By comparing the performance of HistoXGAN with existing methods in reconstructed images, verify the superiority of HistoXGAN in maintaining key information such as tumor grading, histological subtypes and gene expression patterns.

2. **Enhance the interpretability of the model**: Use HistoXGAN to reveal the histological features behind the actionable mutations identified by the deep - learning model, evaluate whether the model predictions are affected by histological batch effects, and demonstrate the ability to accurately reconstruct tumor histology from radiological data, achieving "virtual biopsy".

3. **Discover new biological insights**: Through the images generated by HistoXGAN, researchers can more intuitively understand the biological characteristics of tumors, such as the histological manifestations of PIK3CA mutations and homologous recombination deficiency (HRD) status, thus providing new insights for targeted therapies of these molecular pathways.

4. **Solve the problem of cross - modal data**: Explore how to reconstruct histological images from other forms of data (such as gene sequencing or MRI imaging), which helps to develop more accurate cross - modal auto - encoders and further promote the application of "virtual biopsy".

In summary, this paper aims to not only improve the quality of image reconstruction through the innovative tool of HistoXGAN, but also enhance the ability to interpret the prediction results of deep - learning models, while providing a new perspective for tumor biology research.