Abstract:As advances in large language models (LLMs) and multimodal techniques continue to mature, the development of general-purpose multimodal large language models (MLLMs) has surged, with significant applications in natural image interpretation. However, the field of pathology has largely remained untapped in this regard, despite the growing need for accurate, timely, and personalized diagnostics. To bridge the gap in pathology MLLMs, we present the PathAsst in this study, which is a generative foundation AI assistant to revolutionize diagnostic and predictive analytics in pathology. To develop PathAsst, we collect over 142K high-quality pathology image-text pairs from a variety of reliable sources, including PubMed, comprehensive pathology textbooks, reputable pathology websites, and private data annotated by pathologists. Leveraging the advanced capabilities of ChatGPT/GPT-4, we generate over 180K instruction-following samples. Furthermore, we devise additional instruction-following data, specifically tailored for the invocation of the pathology-specific models, allowing the PathAsst to effectively interact with these models based on the input image and user intent, consequently enhancing the model's diagnostic capabilities. Subsequently, our PathAsst is trained based on Vicuna-13B language model in coordination with the CLIP vision encoder. The results of PathAsst show the potential of harnessing the AI-powered generative foundation model to improve pathology diagnosis and treatment processes. We are committed to open-sourcing our meticulously curated dataset, as well as a comprehensive toolkit designed to aid researchers in the extensive collection and preprocessing of their own datasets. Resources can be obtained at https://github.com/superjamessyx/Generative-Foundation-AI-Assistant-for-Pathology.

PathUp: Patch-wise Timestep Tracking for Multi-class Large Pathology Image Synthesising Diffusion Model

Space and Level Cooperation Framework for Pathological Cancer Grading

Exploit Spatially Resolved Transcriptomic Data to Infer Cellular Features from Pathology Imaging Data}

PathoDuet: Foundation Models for Pathological Slide Analysis of H&E and IHC Stains

PathTR: Context-Aware Memory Transformer for Tumor Localization in Gigapixel Pathology Images.

Generating Progressive Images from Pathological Transitions via Diffusion Model

Unpaired Stain Transfer Using Pathology-Consistent Constrained Generative Adversarial Networks

A pathology foundation model for cancer diagnosis and prognosis prediction

PathAsst: Redefining Pathology through Generative Foundation AI Assistant for Pathology

Controllable and Efficient Multi-Class Pathology Nuclei Data Augmentation using Text-Conditioned Diffusion Models

Towards a general-purpose foundation model for computational pathology

Automatic Whole Slide Pathology Image Diagnosis Framework Via Unit Stochastic Selection and Attention Fusion

Automatic Generation of Pathological Benchmark Dataset from Hyperspectral Images of Double Stained Tissues

Prompt-driven Healthy/Diseased Image Pairs Enabling Pixel-level Chest X-ray Pathology Localization

Latent Diffusion Models with Image-Derived Annotations for Enhanced AI-Assisted Cancer Diagnosis in Histopathology

Micro‐morphological feature visualization, auto‐classification, and evolution quantitative analysis of tumors by using SR‐PCT

Understanding the Mechanism of Deep Learning Framework for Lesion Detection in Pathological Images with Breast Cancer

A General-Purpose Self-Supervised Model for Computational Pathology

A whole-slide foundation model for digital pathology from real-world data

Whole Slide Images are 2D Point Clouds: Context-Aware Survival Prediction using Patch-based Graph Convolutional Networks

A Deep Reinforcement Learning Framework for Rapid Diagnosis of Whole Slide Pathological Images