SkinGPT-4: An Interactive Dermatology Diagnostic System with Visual Large Language Model

Juexiao Zhou,Xiaonan He,Liyuan Sun,Jiannan Xu,Xiuying Chen,Yuetan Chu,Longxi Zhou,Xingyu Liao,Bin Zhang,Xin Gao
2023-06-08
Abstract:Skin and subcutaneous diseases rank high among the leading contributors to the global burden of nonfatal diseases, impacting a considerable portion of the population. Nonetheless, the field of dermatology diagnosis faces three significant hurdles. Firstly, there is a shortage of dermatologists accessible to diagnose patients, particularly in rural regions. Secondly, accurately interpreting skin disease images poses a considerable challenge. Lastly, generating patient-friendly diagnostic reports is usually a time-consuming and labor-intensive task for dermatologists. To tackle these challenges, we present SkinGPT-4, which is the world's first interactive dermatology diagnostic system powered by an advanced visual large language model. SkinGPT-4 leverages a fine-tuned version of MiniGPT-4, trained on an extensive collection of skin disease images (comprising 52,929 publicly available and proprietary images) along with clinical concepts and doctors' notes. We designed a two-step training process to allow SkinGPT to express medical features in skin disease images with natural language and make accurate diagnoses of the types of skin diseases. With SkinGPT-4, users could upload their own skin photos for diagnosis, and the system could autonomously evaluate the images, identifies the characteristics and categories of the skin conditions, performs in-depth analysis, and provides interactive treatment recommendations. Meanwhile, SkinGPT-4's local deployment capability and commitment to user privacy also render it an appealing choice for patients in search of a dependable and precise diagnosis of their skin ailments. To demonstrate the robustness of SkinGPT-4, we conducted quantitative evaluations on 150 real-life cases, which were independently reviewed by certified dermatologists, and showed that SkinGPT-4 could provide accurate diagnoses of skin diseases.
Image and Video Processing,Computer Vision and Pattern Recognition,Machine Learning
What problem does this paper attempt to address?
### Problems the Paper Aims to Solve The paper aims to address three major challenges in the field of dermatological diagnosis: 1. **Shortage of Dermatologists**: Especially in rural areas, there is a lack of professional dermatologists to diagnose patients. 2. **Difficulty in Accurately Interpreting Skin Disease Images**: Accurately identifying features of skin diseases from images is a significant challenge. 3. **Time-Consuming and Labor-Intensive Generation of Patient-Friendly Diagnostic Reports**: Dermatologists often need to spend a considerable amount of time and effort writing patient-friendly diagnostic reports. To address these issues, the research team proposed **SkinGPT-4**, the world's first interactive dermatological diagnosis system based on an advanced visual Large Language Model (LLM). Specifically: - **Dataset and Training**: SkinGPT-4 uses a fine-tuned version of MiniGPT-4 and is trained on a dataset containing 52,929 public and proprietary skin disease images. Additionally, it incorporates clinical concepts and doctor notes for two-stage training. - **Features and Advantages**: - Users can upload their own skin photos for diagnosis. - The system can automatically evaluate the images, identify features and categories of skin conditions, and provide in-depth analysis and interactive treatment suggestions. - SkinGPT-4's local deployment capability and its protection of user privacy make it a reliable diagnostic tool. - **Evaluation Results**: Quantitative evaluation of 150 real cases shows that SkinGPT-4 can provide accurate skin disease diagnoses. Although SkinGPT-4 cannot completely replace doctors, it can enhance users' understanding of their medical conditions, improve communication between patients and doctors, expedite the diagnostic process for dermatologists, and potentially promote healthcare equity in underdeveloped areas.