Performance of ChatGPT on NASS Clinical Guidelines for the Diagnosis and Treatment of Low Back Pain

Nancy Shrestha,Zekun Shen,Bashar Zaidat,Akiro H Duey,Justin E Tang,Wasil Ahmed,Timothy Hoang,Mateo Restrepo Mejia,Rami Rajjoub,Jonathan S Markowitz,Jun S Kim,Samuel K Cho,Akiro H. Duey,Justin E. Tang,Jonathan S. Markowitz,Jun S. Kim,Samuel K. Cho

DOI: https://doi.org/10.1097/brs.0000000000004915

IF: 3.241

2024-01-12

Spine

Abstract:Study Design. Comparative analysis. Objective. To evaluate ChatGPT’s ability to predict appropriate clinical recommendations based on the most recent clinical guidelines for the diagnosis and treatment of low back pain. Background. Low back pain is a very common and often debilitating condition that affects many people globally. Chat Generative Pre-trained Transformer (ChatGPT) is an artificial intelligence model that may be able to generate recommendations for low back pain. Methods. Using the North American Spine Society Evidence-Based Clinical Guidelines as the gold standard, 82 clinical questions relating to low back pain were entered into ChatGPT (GPT-3.5) independently. For each question, we recorded ChatGPT’s answer, then used a point - answer system— the point being the guideline recommendation and the answer being ChatGPT’s response— and asked ChatGPT if the point was mentioned in the answer to assess for accuracy. This response accuracy was repeated with one caveat— a prior prompt is given in ChatGPT to answer as an experienced orthopedic surgeon— for each question by guideline category. A two-sample proportion z-test was used to assess any differences between the pre- and post-prompt scenarios with alpha=0.05. Results. ChatGPT’s response was accurate 65% (72% post-prompt, P =0.41) for guidelines with clinical recommendations, 46% (58% post-prompt, P =0.11) for guidelines with insufficient or conflicting data, and 49% (16% post-prompt, P =0.003*) for guidelines with no adequate study to address the clinical question. For guidelines with insufficient or conflicting data, 44% (25% post-prompt, P =0.01*) of ChatGPT responses wrongly suggested that sufficient evidence existed. Conclusion. ChatGPT was able to produce a sufficient clinical guideline recommendation for low back pain, with overall improvements if initially prompted. However, it tended to wrongly suggest evidence and often failed to mention, especially post-prompt, when there is not enough evidence to adequately give an accurate recommendation.

clinical neurology,orthopedics