Artificial Intelligence and Public Health: Evaluating ChatGPT Responses to Vaccination Myths and Misconceptions
Giovanna Deiana,Marco Dettori,Antonella Arghittu,Antonio Azara,Giovanni Gabutti,Paolo Castiglia
DOI: https://doi.org/10.3390/vaccines11071217
2023-07-08
Vaccines
Abstract:Artificial intelligence (AI), such as ChatGPT, is the subject of intense debate regarding its possible applications, including in the health care context. This study evaluates Correctness, Clarity, and Exhaustiveness of the answers provided by ChatGPT on the topic of vaccination. The World Health Organization's 11 "myths and misconceptions" about vaccinations were administered to both the free (GPT-3.5) and paid version (GPT-4.0) of ChatGPT. The AI's responses were evaluated qualitatively and quantitatively, in reference to those provided by WHO, independently by two expert Raters. The agreement between the Raters was significant for both versions (p of K < 0.05). Overall, ChatGPT responses were easy to understand and 85.4% accurate although one of the questions was misinterpreted. Qualitatively, GPT-4.0 responses were superior to GPT-3.5 responses in Correctness, Clarity, and Exhaustiveness (Δ = 5.6%, 17.9%, 9.3% respectively). The study shows that, if appropriately questioned, AIs can represent a useful aid in the health care field. However, when consulted by non-expert users, without the support of expert medical advice, it is not free from the risk of eliciting misleading responses. Moreover, given the existing social divide in information access, the improved accuracy of answers from the paid version raises further ethical issues.
immunology,medicine, research & experimental