Comparison of Humans Versus Mobile Phone-Powered Artificial Intelligence for the Diagnosis and Management of Skin Cancer: A Multinational Prospective Diagnostic Clinical Trial
Scott W. Menzies,Christoph Sinz,Michelle Menzies,Serigne N. Lo,William Yolland,Johann Lingohr,Majid Razmara,Philipp Tschandl,Pascale Guitera,Richard A. Scolyer,Florentina Boltz,Liliane Borik-Heil,Hsien Herbert Chan,David Chromy,David J. Coker,Helena Collgros,Maryam Eghtedari,Marina Corral Forteza,Emily Forward,Bruna Gallo,Stephanie Geisler,Matthew Gibson,Amelie Hampel,Genevieve Ho,Laura Junez,Philipp Kienzl,Arthur Martin,Fergal J. Moloney,Amanda Regio Pereira,Julia Maria Ressler,Susanne Richter,Katarina Silic,Thomas Silly,Michael Skoll,Julia Tittes,Philipp Weber,Wolfgang P. Weninger,Doris Weiss,Ping Woo-Sampson,Catherine Zilberg,Harald Kittler
DOI: https://doi.org/10.2139/ssrn.4282949
2022-01-01
SSRN Electronic Journal
Abstract:Background: Early diagnosis of skin cancer requires specialised medical expertise, which is limited. However, there is hope that mobile phone-powered artificial intelligence (AI) can bridge this gap, but it is unclear how this technology will perform in a realistic clinical scenario. Methods: We conducted a multinational prospective clinical trial to compare the accuracy of expert and novice diagnostic and management decisions with two types of mobile phone-powered AI incorporating a simple optical attachment; a novel 7-class algorithm and the ISIC algorithm previously tested in a large online reader study. The diagnostic study included 172 pigmented lesions (84 malignant) from 124 patients and the management study included 5696 lesions (18 malignant) from 66 patients, with dismiss, biopsy, or 3-month monitor as possible decisions. Decisions to monitor were considered equivalent to dismissing (scenario A) or equivalent to biopsy of malignant lesions (scenario B). Findings: The 7-class algorithm was equivalent to specialist diagnosis (absolute accuracy difference 1·2%,95%CI -6·9to9·2%) and significantly superior to the novice (21·5%,95%CI 13·1to30·0%). The ISIC algorithm was significantly inferior to the specialist (-11·6%,95%CI -20·3to-2·97%) but significantly superior to the novice (8.7%,95%CI -0·5to18·0%). Compared with specialists the best 7-class management was significantly inferior (absolute difference in correct management decision -0·5%,95%CI -0·7to-0·24% in Scenario A and -0·4%,95%CI -0·8to-0·05% for Scenario B). Compared with the novices’ management the 7-class was significantly inferior (-0·4%,95%CI -0·6to-0·15%) for Scenario A but significantly superior (+0·4%,95%CI 0·0to0·85%) for Scenario B. Interpretation: A new mobile phone-powered AI technology is a simple, practical and accurate intervention for the diagnosis of pigmented skin cancer, although its usage for management decisions requires more careful execution. An algorithm that was superior in experimental studies was significantly inferior to specialists in a real-world scenario, suggesting that caution is needed when extrapolating results of experimental studies to clinical practice.Trial Registration Details: The trial was registered at the Australian New Zealand Clinical Trials Registry ACTRN12620000695909 (Universal trial no.U1111-1251-8995).Funding Information: MetaOptima Technology Inc. Vancouver, Canada.Declaration of Interests: Scott Menzies is on the Scientific Advisory Board of SciBase AB, Stockholm, Sweden and previously (within 1 year) for Kāhu (MoleMap, Auckland, New Zealand), both which produce competing diagnostic devices for skin cancer. William Yolland, Johann Lingohr, and Majid Razmara are employees of the trial sponsor MetaOptima Technology Inc. (Vancouver, Canada). Pascale Guitera has received fees for professional services from MetaOptima Technology Inc. Richard Scolyer has received fees for professional services from MetaOptima Technology Inc., F. Hoffmann-La Roche Ltd, Evaxion, Provectus Biopharmaceuticals Australia, Qbiotics, Novartis, Merck Sharp & Dohme, NeraCare, AMGEN Inc., Bristol-Myers Squibb, Myriad Genetics, GlaxoSmithKline. Serigne Lo’s institute has received payment for consultancy services. Harald Kittler received speaker fees from Fotofinder, equipment from Fotofinder, Heine, Derma Medical, Casio, and 3Gen, and license fees from MetaOptima Technology Inc, Heine, and Casio. Matthew Gibson has received a conference travel grant from Sun Pharmaceutical Industries Limited. David Chromy served as a speaker and/or advisory board member for AbbVie, Gilead Science, ViiV Healthcare and MSD, and received travel support from AbbVie, MSD, ViiV Healthcare and Gilead Science. Philipp Tschandl has received fees for professional services from Silverchair, an unrestricted 1-year postdoc grant from MetaOptima Technology Inc (2017), speaker fees from Novartis, Lilly and FotoFinder, and unrestricted grants from Lilly. Philipp Weber has received travel support from Pfizer. Julia Maria Ressler has received speakers honoraria by Bristol-Myers Squibb, Roche, Amgen and Novartis and travel support by Roche, Bristol-Myers Squibb and Sanofi.Liliane Borik-Heil: no conflict of interests to declare.Julia Tittes has received speaker fees from Lilly and Novartis, travel support from Almirall and AbbVie, and an unrestricted grant from Lilly. Hsien Herbert Chan serves as a medical advisor to Metasense.Ethics Approval Statement: The study was conducted in accordance with the National Statement on Ethical Conduct in Human Research (2007), the CPMP/ICH Note for Guidance on Good Clinical Practice and consistent with the principles that have their origin in the Declaration of Helsinki. Compliance with these standards provides assurance that the rights, safety and well-being of trial participants are respected. The trial was approved by the Research Ethics and Governance Office of the Sydney Local Health District (Royal Prince Alfred Hospital Protocol No X19-0066 & 2019/ETH00468).