Prasanna Porwal,Samiksha Pachade,Manesh Kokare,Girish Deshmukh,Jaemin Son,Woong Bae,Lihong Liu,Jianzong Wang,Xinhui Liu,Liangxin Gao,TianBo Wu,Jing Xiao,Fengyan Wang,Baocai Yin,Yunzhi Wang,Gopichandh Danala,Linsheng He,Yoon Ho Choi,Yeong Chan Lee,Sang-Hyuk Jung,Zhongyu Li,Xiaodan Sui,Junyan Wu,Xiaolong Li,Ting Zhou,Janos Toth,Agnes Bara,Avinash Kori,Sai Saketh Chennamsetty,Mohammed Safwan,Varghese Alex,Xingzheng Lyu,Li Cheng,Qinhao Chu,Pengcheng Li,Xin Ji,Sanyuan Zhang,Yaxin Shen,Ling Dai,Oindrila Saha,Rachana Sathish,Tania Melo,Teresa Araujo,Balazs Harangi,Bin Sheng,Ruogu Fang,Debdoot Sheet,Andras Hajdu,Yuanjie Zheng,Ana Maria Mendonca,Shaoting Zhang,Aurelio Campilho,Bin Zheng,Dinggang Shen,Luca Giancardo,Gwenole Quellec,Fabrice Meriaudeau

Abstract:Diabetic Retinopathy (DR) is the most common cause of avoidable vision loss, predominantly affecting the working-age population across the globe. Screening for DR, coupled with timely consultation and treatment, is a globally trusted policy to avoid vision loss. However, implementation of DR screening programs is challenging due to the scarcity of medical professionals able to screen a growing global diabetic population at risk for DR. Computer-aided disease diagnosis in retinal image analysis could provide a sustainable approach for such large-scale screening effort. The recent scientific advances in computing capacity and machine learning approaches provide an avenue for biomedical scientists to reach this goal. Aiming to advance the state-of-the-art in automatic DR diagnosis, a grand challenge on "Diabetic Retinopathy - Segmentation and Grading" was organized in conjunction with the IEEE International Symposium on Biomedical Imaging (ISBI - 2018). In this paper, we report the set-up and results of this challenge that is primarily based on Indian Diabetic Retinopathy Image Dataset (IDRiD). There were three principal sub-challenges: lesion segmentation, disease severity grading, and localization of retinal landmarks and segmentation. These multiple tasks in this challenge allow to test the generalizability of algorithms, and this is what makes it different from existing ones. It received a positive response from the scientific community with 148 submissions from 495 registrations effectively entered in this challenge. This paper outlines the challenge, its organization, the dataset used, evaluation methods and results of top-performing participating solutions. The top-performing approaches utilized a blend of clinical information, data augmentation, and an ensemble of models. These findings have the potential to enable new developments in retinal image analysis and image-based DR screening in particular.

Is the winner really the best? A critical analysis of common research practice in biomedical image analysis competitions.

Why is the Winner the Best?

BIAS: Transparent reporting of biomedical image analysis challenges

Postoperative graft incorporation after aortic reconstruction--comparison between computerised tomography and Tc-99m-HMPAO labelled leucocyte imaging.

How can we learn (more) from challenges? A statistical approach to driving future algorithm development

Understanding metric-related pitfalls in image analysis validation

Challenge Results Are Not Reproducible

Beyond the Leaderboard: Insight and Deployment Challenges to Address Research Problems

The Medical Segmentation Decathlon

AI COMPETITIONS AS INFRASTRUCTURES: EXAMINING POWER RELATIONS ON KAGGLE AND GRAND CHALLENGE IN AI-DRIVEN MEDICAL IMAGING

Metrics reloaded: Recommendations for image analysis validation

AI Competitions and Benchmarks: The life cycle of challenges and benchmarks

Her2 Challenge Contest: A Detailed Assessment of Automated Her2 Scoring Algorithms in Whole Slide Images of Breast Cancer Tissues

Community challenges in biomedical text mining over 10 years: success, failure and the future

Ten years of image analysis and machine learning competitions in dementia

A comprehensive analysis of classification methods in gastrointestinal endoscopy imaging

IDRiD: Diabetic Retinopathy - Segmentation and Grading Challenge.

Sources of performance variability in deep learning-based polyp detection

BACH: Grand challenge on breast cancer histology images

A Framework for Challenge Design: Insight and Deployment Challenges to Address Medical Image Analysis Problems