Large Language Models versus Classical Machine Learning: Performance in COVID-19 Mortality Prediction Using High-Dimensional Tabular Data
Mohammadreza Ghaffarzadeh-Esfahani,Mahdi Ghaffarzadeh-Esfahani,Arian Salahi-Niri,Hossein Toreyhi,Zahra Atf,Amirali Mohsenzadeh-Kermani,Mahshad Sarikhani,Zohreh Tajabadi,Fatemeh Shojaeian,Mohammad Hassan Bagheri,Aydin Feyzi,Mohammadamin Tarighatpayma,Narges Gazmeh,Fateme Heydari,Hossein Afshar,Amirreza Allahgholipour,Farid Alimardani,Ameneh Salehi,Naghmeh Asadimanesh,Mohammad Amin Khalafi,Hadis Shabanipour,Ali Moradi,Sajjad Hossein Zadeh,Omid Yazdani,Romina Esbati,Moozhan Maleki,Danial Samiei Nasr,Amirali Soheili,Hossein Majlesi,Saba Shahsavan,Alireza Soheilipour,Nooshin Goudarzi,Erfan Taherifard,Hamidreza Hatamabadi,Jamil S Samaan,Thomas Savage,Ankit Sakhuja,Ali Soroush,Girish Nadkarni,Ilad Alavi Darazam,Mohamad Amin Pourhoseingholi,Seyed Amir Ahmad Safavi-Naini
2024-09-02
Abstract:Background: This study aimed to evaluate and compare the performance of classical machine learning models (CMLs) and large language models (LLMs) in predicting mortality associated with COVID-19 by utilizing a high-dimensional tabular dataset.
Materials and Methods: We analyzed data from 9,134 COVID-19 patients collected across four hospitals. Seven CML models, including XGBoost and random forest (RF), were trained and evaluated. The structured data was converted into text for zero-shot classification by eight LLMs, including GPT-4 and Mistral-7b. Additionally, Mistral-7b was fine-tuned using the QLoRA approach to enhance its predictive capabilities.
Results: Among the CML models, XGBoost and RF achieved the highest accuracy, with F1 scores of 0.87 for internal validation and 0.83 for external validation. In the LLM category, GPT-4 was the top performer with an F1 score of 0.43. Fine-tuning Mistral-7b significantly improved its recall from 1% to 79%, resulting in an F1 score of 0.74, which was stable during external validation.
Conclusion: While LLMs show moderate performance in zero-shot classification, fine-tuning can significantly enhance their effectiveness, potentially aligning them closer to CML models. However, CMLs still outperform LLMs in high-dimensional tabular data tasks.
Machine Learning,Artificial Intelligence,Computation and Language