Abstract:With an estimated 3 billion people globally lacking access to dermatological care, technological solutions leveraging artificial intelligence (AI) have been proposed to improve access. Diagnostic AI algorithms, however, require high-quality datasets to allow development and testing, particularly those that enable evaluation of both unimodal and multimodal approaches. Currently, the majority of dermatology AI algorithms are built and tested on proprietary, siloed data, often from a single site and with only a single image type (i.e., clinical or dermoscopic). To address this, we developed and released the Melanoma Research Alliance Multimodal Image Dataset for AI-based Skin Cancer (MIDAS) dataset, the largest publicly available, prospectively-recruited, paired dermoscopic- and clinical image-based dataset of biopsy-proven and dermatopathology-labeled skin lesions. We explored model performance on real-world cases using four previously published state-of-the-art (SOTA) models and compared model-to-clinician diagnostic performance. We also assessed algorithm performance using clinical photography taken at different distances from the lesion to assess its influence across diagnostic categories. We prospectively enrolled 796 patients through an IRB-approved protocol with informed consent representing 1290 unique lesions and 3830 total images (including dermoscopic and clinical images taken at 15-cm and 30-cm distance). Images represented the diagnostic diversity of lesions seen in general dermatology, with malignant, benign, and inflammatory lesions that included melanocytic nevi (22%; n=234), invasive cutaneous melanomas (4%; n=46), and melanoma in situ (4%; n=47). When evaluating SOTA models using the MIDAS dataset, we observed performance reduction across all models compared to their previously published performance metrics, indicating challenges to generalizability of current SOTA algorithms. As a comparative baseline, the dermatologists performing biopsies were 79% accurate with their top-1 diagnosis at differentiating a malignant from benign lesion. For malignant lesions, algorithms performed better on images acquired at 15-cm compared to 30-cm distance while dermoscopic images yielded higher sensitivity compared to clinical images. Improving our understanding of the strengths and weaknesses of AI diagnostic algorithms is critical as these tools advance towards widespread clinical deployment. While many algorithms may report high performance metrics, caution should be taken due to the potential for overfitting to localized datasets. MIDAS's robust, multimodal, and diverse dataset allows researchers to evaluate algorithms on our real-world images and better assess their generalizability.

PAD-UFES-20: a skin lesion dataset composed of patient data and clinical images collected from smartphones

The SLICE-3D dataset: 400,000 skin lesion image crops extracted from 3D TBP for skin cancer detection

BCN20000: Dermoscopic Lesions in the Wild

A protocol for annotation of total body photography for machine learning to analyze skin phenotype and lesion classification

Multimodal Image Dataset for AI-based Skin Cancer (MIDAS) Benchmarking

DERM12345: A Large, Multisource Dermatoscopic Skin Lesion Dataset with 40 Subclasses

DERM12345: A Large, Multisource Dermatoscopic Skin Lesion Dataset with 38 Subclasses

The impact of patient clinical information on automated skin cancer detection

Skin Cancer Diagnostics with an All-Inclusive Smartphone Application

A Smartphone based Application for Skin Cancer Classification Using Deep Learning with Clinical Images and Lesion Information

A Patient-Centric Dataset of Images and Metadata for Identifying Melanomas Using Clinical Context

Implementation of artificial intelligence algorithms for melanoma screening in a primary care setting

Beyond Visual Image: Automated Diagnosis of Pigmented Skin Lesions Combining Clinical Image Features with Patient Data

Optimizing Skin Lesion Classification via Multimodal Data and Auxiliary Task Integration

Skin cancer diagnosis using NIR spectroscopy data of skin lesions in vivo using machine learning algorithms

A Survey on Deep Learning for Skin Lesion Segmentation

Malignancy Prediction and Lesion Identification from Clinical Dermatological Images

Clinical Skin Lesion Diagnosis Using Representations Inspired By Dermatologist Criteria

Learning from Dermoscopic Images in Association with Clinical Metadata for Skin Lesion Segmentation and Classification

Skin Lesion Analysis toward Melanoma Detection: A Challenge at the International Symposium on Biomedical Imaging (ISBI) 2016, hosted by the International Skin Imaging Collaboration (ISIC)