External validation of inpatient neonatal mortality prediction models in high-mortality settings
Timothy Tuti,Gary Collins,Mike English,Clinical Information Network,Jalemba Aluvaala,George Mbevi,John Wainaina,Livingstone Mumelo,Edith Gicheha,Naomi Muinga,Muthoni Ogola,Laura Oyiengo,Caroline Mwangi,Fred Were,Juma Vitalis,Nyumbile Bonface,Roselyne Malangachi,Christine Manyasi,Catherine Mutinda,David Kibiwott Kimutai,Rukia Aden,Caren Emadau,Elizabeth Atieno Jowi,Cecilia Muithya,Charles Nzioki,Supa Tunje,Penina Musyoka,Wagura Mwangi,Agnes Mithamo,Magdalene Kuria,Esther Njiru,Mwangi Ngina,Penina Mwangi,Rachel Inginia,Melab Musabi,Emma Namulala,Grace Ochieng,Lydia Thuranira,Felicitas Makokha,Josephine Ojigo,Beth Maina,Mary Waiyego,Bernadette Lusweti,Angeline Ithondeka,Julie Barasa,Meshack Liru,Elizabeth Kibaru,Alice Nkirote Nyaribari,Joyce Akuka,Joyce Wangari,Amilia Ngoda,Aggrey Nzavaye Emenwa,Dolphine Mochache,Patricia Nafula Wesakania,George Lipesa,Jane Mbungu,Marystella Mutenyo,Joyce Mbogho,Joan Baswetty,Ann Jambi,Josephine Aritho,Beatrice Njambi,Felisters Mucheke,Zainab Kioni,Lucy Kinyua,Margaret Kethi,Alice Oguda,Salome Nashimiyu Situma,Nancy Gachaja,Loise N Mwangi,Ruth Mwai,Irginia Wangari Muruga,Nancy Mburu,Celestine Muteshi,Abigael Bwire,Salome Okisa Muyale,Naomi Situma,Faith Mueni,Hellen Mwaura,Rosemary Mututa,Caroline Lavu,Joyce Oketch,Jane Hore Olum,Orina Nyakina,Faith Njeru,Rebecca Chelimo,Margaret Wanjiku Mwaura,Ann Wambugu,Epharus Njeri Mburu,Linda Awino Tindi,Jane Akumu,Ruth Otieno,Slessor Osok,Seline Kulubi,Susan Wanjala,Pauline Njeru,Rebbecca Mukami Mbogo,John Ollongo,Samuel Soita,Judith Mirenja,Mary Nguri,Margaret Waweru,Mary Akoth Oruko,Jeska Kuya,Caroline Muthuri,Esther Muthiani,Esther Mwangi,Joseph Nganga,Benjamin Tanui,Alfred Wanjau,Judith Onsongo,Peter Muigai,Arnest Namayi,Elizabeth Kosiom,Dorcas Cherop,Faith Marete,Johanness Simiyu,Collince Danga,Arthur Otieno Oyugi,Fredrick Keya Okoth
DOI: https://doi.org/10.1186/s12916-022-02439-5
2022-08-03
Abstract:Background: Two neonatal mortality prediction models, the Neonatal Essential Treatment Score (NETS) which uses treatments prescribed at admission and the Score for Essential Neonatal Symptoms and Signs (SENSS) which uses basic clinical signs, were derived in high-mortality, low-resource settings to utilise data more likely to be available in these settings. In this study, we evaluate the predictive accuracy of two neonatal prediction models for all-cause in-hospital mortality. Methods: We used retrospectively collected routine clinical data recorded by duty clinicians at admission from 16 Kenyan hospitals used to externally validate and update the SENSS and NETS models that were initially developed from the data from the largest Kenyan maternity hospital to predict in-hospital mortality. Model performance was evaluated by assessing discrimination and calibration. Discrimination, the ability of the model to differentiate between those with and without the outcome, was measured using the c-statistic. Calibration, the agreement between predictions from the model and what was observed, was measured using the calibration intercept and slope (with values of 0 and 1 denoting perfect calibration). Results: At initial external validation, the estimated mortality risks from the original SENSS and NETS models were markedly overestimated with calibration intercepts of - 0.703 (95% CI - 0.738 to - 0.669) and - 1.109 (95% CI - 1.148 to - 1.069) and too extreme with calibration slopes of 0.565 (95% CI 0.552 to 0.577) and 0.466 (95% CI 0.451 to 0.480), respectively. After model updating, the calibration of the model improved. The updated SENSS and NETS models had calibration intercepts of 0.311 (95% CI 0.282 to 0.350) and 0.032 (95% CI - 0.002 to 0.066) and calibration slopes of 1.029 (95% CI 1.006 to 1.051) and 0.799 (95% CI 0.774 to 0.823), respectively, while showing good discrimination with c-statistics of 0.834 (95% CI 0.829 to 0.839) and 0.775 (95% CI 0.768 to 0.782), respectively. The overall calibration performance of the updated SENSS and NETS models was better than any existing neonatal in-hospital mortality prediction models externally validated for settings comparable to Kenya. Conclusion: Few prediction models undergo rigorous external validation. We show how external validation using data from multiple locations enables model updating and improving their performance and potential value. The improved models indicate it is possible to predict in-hospital mortality using either treatments or signs and symptoms derived from routine neonatal data from low-resource hospital settings also making possible their use for case-mix adjustment when contrasting similar hospital settings.