Can artificial intelligence models serve as patient information consultants in orthodontics?

Derya Dursun,Rumeysa Bilici Geçer

DOI: https://doi.org/10.1186/s12911-024-02619-8

IF: 3.298

2024-07-30

BMC Medical Informatics and Decision Making

Abstract:To evaluate the accuracy, reliability, quality, and readability of responses generated by ChatGPT-3.5, ChatGPT-4, Gemini, and Copilot in relation to orthodontic clear aligners.

medical informatics

What problem does this paper attempt to address?

How reliable is the artificial intelligence product large language model ChatGPT in orthodontics?

Kevser Kurt Demirsoy,Suleyman Kutalmış Buyuk,Tayyip Bicer

DOI: https://doi.org/10.2319/031224-207.1

2024-08-14

Abstract:Objectives: To evaluate the reliability of information produced by the artificial intelligence-based program ChatGPT in terms of accuracy and relevance, as assessed by orthodontists, dental students, and individuals seeking orthodontic treatment. Materials and methods: Frequently asked and curious questions in four basic areas related to orthodontics were prepared and asked in ChatGPT (Version 4.0), and answers were evaluated by three different groups (senior dental students, individuals seeking orthodontic treatment, orthodontists). Questions asked in these basic areas of orthodontics were about: clear aligners (CA), lingual orthodontics (LO), esthetic braces (EB), and temporomandibular disorders (TMD). The answers were evaluated by the Global Quality Scale (GQS) and Quality Criteria for Consumer Health Information (DISCERN) scale. Results: The total mean DISCERN score for answers on CA for students was 51.7 ± 9.38, for patients was 57.2 ± 10.73 and, for orthodontists was 47.4 ± 4.78 (P = .001). Comparison of GQS scores for LO among groups: students (3.53 ± 0.78), patients (4.40 ± 0.72), and orthodontists (3.63 ± 0.72) (P < .001). Intergroup comparison of ChatGPT evaluations about TMD was examined in terms of the DISCERN scale, with the highest value given in the patients group (57.83 ± 11.47) and lowest value in the orthodontist group (45.90 ± 11.84). When information quality evaluation about EB was examined, it GQS scores were >3 in all three groups (students: 3.50 ± 0.78; patients: 4.17 ± 0.87; orthodontists: 3.50 ± 0.82). Conclusions: ChatGPT has significant potential in terms of usability for patient information and education in the field of orthodontics if it is developed and necessary updates are made.
Content analysis of AI-generated (ChatGPT) responses concerning orthodontic clear aligners

Sarah Abu Arqub,Dalya Al-Moghrabi,Veerasathpurush Allareddy,Madhur Upadhyay,Nikhilesh Vaid,Sumit Yadav

DOI: https://doi.org/10.2319/071123-484.1

2024-01-10

The Angle Orthodontist

Abstract:ABSTRACT Objectives To assess the accuracy of ChatGPT answers concerning orthodontic clear aligners. Materials and Methods A cross-sectional content analysis of ChatGPT generated responses to queries related to clear aligner treatment (CAT) was undertaken. A total of 111 questions were generated by three orthodontists based on a set of predefined domains and subdomains. The artificial intelligence (AI)-generated (ChatGPT) answers were extracted and their accuracy was determined independently by five orthodontists. The accuracy of answers was assessed using a prepiloted four-point scale scoring rubric. Descriptive statistics were performed. Results The total mean accuracy score for the entire set was 2.6 ± 1.1. It was noted that 58% of the AI-generated answers were scored as objectively true, 18% were selected facts, 9% were minimal facts, and 15% were false. False claims included the ability of CAT to reduce the need for orthognathic surgery (4.0 ± 0.0), improve airway function (3.8 ± 0.5), achieve root parallelism (3.6 ± 0.5), alleviate sleep apnea (3.8 ± 0.5), and produce more stable results compared to fixed appliances (3.8 ± 0.5). Conclusions The overall level of accuracy of ChatGPT responses to questions concerning CAT was suboptimal and lacked citations to relevant literature. Ability of the software to offer current and precise information was limited. Therefore, clinicians and patients must be mindful of false claims and relevant facts omitted in the answers generated by ChatGPT.

dentistry, oral surgery & medicine
An evaluation of orthodontic information quality regarding artificial intelligence (AI) chatbot technologies: A comparison of ChatGPT and google BARD

Can Arslan,Kaan Kahya,Emre Cesur,Derya Germec Cakan

DOI: https://doi.org/10.2478/aoj-2024-0012

2024-01-01

Australasian Orthodontic Journal

Abstract:Abstract Introduction In recent times, chatbots have played an increasing and noteworthy role in the field of medical practice. The present research was conducted to evaluate the accuracy of the responses provided by ChatGPT and BARD, two of the most utilised chatbot programs, when interrogated regarding orthodontics. Materials and methods Twenty-four popular questions about conventional braces, clear aligners, orthognathic surgery, and orthodontic retainers were chosen for the study. When submitted to the ChatGPT and Google BARD platforms, an experienced orthodontist and an orthodontic resident rated the responses to the questions using a five-point Likert scale, with five indicating evidence-based information, four indicating adequate information, three indicating insufficient information, two indicating incorrect information, and one indicating no response. The results were recorded in Microsoft Excel for comparison and analysis. Results No correlation was found between the ChatGPT and Google BARD scores and word counts. However, a moderate to significant relationship was observed between the scores and several listed references. No significant association was found between the number of words and references, and a statistically significant difference was observed in both investigators’ numerical rating scales using the AI tools ( p = 0.014 and p = 0.030, respectively). Conclusion Generally, ChatGPT and BARD provide satisfactory responses to common orthodontic inquiries that patients might ask. ChatGPT’s answers marginally surpassed those of Google BARD in quality.

dentistry, oral surgery & medicine
Evaluating ChatGPT-4's performance as a digital health advisor for otosclerosis surgery

Samil Sahin,Burak Erkmen,Yaşar Kemal Duymaz,Furkan Bayram,Ahmet Mahmut Tekin,Vedat Topsakal

DOI: https://doi.org/10.3389/fsurg.2024.1373843

2024-06-05

Abstract:Purpose: This study aims to evaluate the effectiveness of ChatGPT-4, an artificial intelligence (AI) chatbot, in providing accurate and comprehensible information to patients regarding otosclerosis surgery. Methods: On October 20, 2023, 15 hypothetical questions were posed to ChatGPT-4 to simulate physician-patient interactions about otosclerosis surgery. Responses were evaluated by three independent ENT specialists using the DISCERN scoring system. The readability was evaluated using multiple indices: Flesch Reading Ease (FRE), Flesch-Kincaid Grade Level (FKGL), Gunning Fog Index (Gunning FOG), Simple Measure of Gobbledygook (SMOG), Coleman-Liau Index (CLI), and Automated Readability Index (ARI). Results: The responses from ChatGPT-4 received DISCERN scores ranging from poor to excellent, with an overall score of 50.7 ± 8.2. The readability analysis indicated that the texts were above the 6th-grade level, suggesting they may not be easily comprehensible to the average reader. There was a significant positive correlation between the referees' scores. Despite providing correct information in over 90% of the cases, the study highlights concerns regarding the potential for incomplete or misleading answers and the high readability level of the responses. Conclusion: While ChatGPT-4 shows potential in delivering health information accurately, its utility is limited by the level of readability of its responses. The study underscores the need for continuous improvement in AI systems to ensure the delivery of information that is both accurate and accessible to patients with varying levels of health literacy. Healthcare professionals should supervise the use of such technologies to enhance patient education and care.
A comparative analysis of AI-based chatbots: Assessing data quality in orthognathic surgery related patient information

Ebru Yurdakurban,Kübra Gülnur Topsakal,Gökhan Serhat Duran

DOI: https://doi.org/10.1016/j.jormas.2023.101757

Abstract:Introduction: The aim of the current study is to evaluate the quality, reliability, readability, and similarity of data provided by different AI-based chatbots in the field of orthognathic surgery. Materials and methods: Guidelines on orthognathic surgery were reviewed, and a list of questions for patients to ask chatbots was produced by two reasearchers. The questions were categorized into 'General Information and Procedure' and 'Results and Recovery', with 30 questions in each category. Five different scoring criteria were used to evaluate the chatbot responses to 60 questions: Ensuring Quality Information for Patients (EQIP) tool, Reliability Scoring System (adapted from DISCERN), Global Quality Scale (GQS), Simple Measure of Gobbledygook (SMOG) and Similarity Index. Results: The highest mean values were observed in OpenEvidence for EQIP tool, SMOG, and Similarity Index, while for Reliability and GQS assessment criteria, MediSearch showed the highest values. When evaluated in terms of reliability and quality, all three AI-based chatbots demonstrated high reliability and good quality; however, they required at least a college-level education for readability based on the SMOG index. Additionally, upon assessment of similarity, ChatGPT-4 displayed high originality, while OpenEvidence exhibited a high degree of similarity. Conclusion: AI-based chatbots with a variety of features have usually provided answers with high quality, reliability, and difficult readability to questions. Although the medical information in the field of orthognathic surgery provided through chatbots is of higher quality, it is recommended that individuals consult their healthcare professionals on this issue.
"Dr. AI Will See You Now": How Do ChatGPT-4 Treatment Recommendations Align With Orthopaedic Clinical Practice Guidelines?

Tanios Dagher,Emma P. Dwyer,Hayden P. Baker,Senthooran Kalidoss,Jason A. Strelzow

DOI: https://doi.org/10.1097/corr.0000000000003234

IF: 4.755

2024-09-10

Clinical Orthopaedics and Related Research

Abstract:Artificial intelligence (AI) is engineered to emulate tasks that have historically required human interaction and intellect, including learning, pattern recognition, decision-making, and problem-solving. Although AI models like ChatGPT-4 have demonstrated satisfactory performance on medical licensing exams, suggesting a potential for supporting medical diagnostics and decision-making, no study of which we are aware has evaluated the ability of these tools to make treatment recommendations when given clinical vignettes and representative medical imaging of common orthopaedic conditions. As AI continues to advance, a thorough understanding of its strengths and limitations is necessary to inform safe and helpful integration into medical practice.

surgery,orthopedics
Assessing the Accuracy of AI Models in Orthodontic Knowledge: A Comparative Study Between ChatGPT-4 and Google Bard

Sadia Naureen,Huma Ghazanfar Kiani

DOI: https://doi.org/10.29271/jcpsp.2024.07.761

Abstract:Objective: To compare the knowledge accuracy of ChatGPT-4 and Google Bard in response to knowledge-based questions related to orthodontic diagnosis and treatment modalities. Study design: Cross-sectional comparative study. Place and Duration of the Study: Department of Orthodontics, Rawal Institute of Health Sciences, Islamabad, Pakistan, from June 23rd to August 30th 2023. Methodology: A comprehensive content analysis was designed based on a mini implant-assisted rapid palatal expansion (MARPE), clear aligners (CA), and cone beam computed tomography (CBCT), involving 30 questions for each category (total = 90) derived from recent review articles. Questions were prepared and presented to two large language models (LLMs): Google Bard and ChatGPT-4. Two independent raters evaluated the accuracy of the responses using a scoring system ranging from one to five, by comparing the answers to a standard key. Statistical analyses, including the paired sample t-test, were used to assess the performance of the two language models. Results: GPT-4 demonstrated superior performance, outperforming Google Bard significantly in the MARPE, CBCT, and CA categories, and achieved a higher mean score. A p-value was found to be (p = 0.001) for MARPE and CBCT, while it was (p = 0.013) for CA. Overall, GPT-4 achieved a total score of 92.6%, surpassing Google Bard's which was 72%. Conclusion: GPT-4 is more efficient than Google Bard in providing accurate and up-to-date information regarding recent trends in orthodontic treatment modalities. Key words: Aligners, Cone beam computed tomography, ChatGPT-4, Google Bard, Mini implant-assisted rapid palatal expansion.
Assessing the Accuracy, Completeness, and Reliability of Artificial Intelligence-Generated Responses in Dentistry: A Pilot Study Evaluating the ChatGPT Model

Kelly F Molena,Ana P Macedo,Anum Ijaz,Fabrício K Carvalho,Maria Julia D Gallo,Francisco Wanderley Garcia de Paula E Silva,Andiara de Rossi,Luis A Mezzomo,Leda Regina F Mugayar,Alexandra M Queiroz

DOI: https://doi.org/10.7759/cureus.65658

2024-07-29

Cureus

Abstract:Background: Artificial intelligence (AI) can be a tool in the diagnosis and acquisition of knowledge, particularly in dentistry, sparking debates on its application in clinical decision-making. Objective: This study aims to evaluate the accuracy, completeness, and reliability of the responses generated by Chatbot Generative Pre-Trained Transformer (ChatGPT) 3.5 in dentistry using expert-formulated questions. Materials and methods: Experts were invited to create three questions, answers, and respective references according to specialized fields of activity. The Likert scale was used to evaluate agreement levels between experts and ChatGPT responses. Statistical analysis compared descriptive and binary question groups in terms of accuracy and completeness. Questions with low accuracy underwent re-evaluation, and subsequent responses were compared for improvement. The Wilcoxon test was utilized (α = 0.05). Results: Ten experts across six dental specialties generated 30 binary and descriptive dental questions and references. The accuracy score had a median of 5.50 and a mean of 4.17. For completeness, the median was 2.00 and the mean was 2.07. No difference was observed between descriptive and binary responses for accuracy and completeness. However, re-evaluated responses showed a significant improvement with a significant difference in accuracy (median 5.50 vs. 6.00; mean 4.17 vs. 4.80; p=0.042) and completeness (median 2.0 vs. 2.0; mean 2.07 vs. 2.30; p=0.011). References were more incorrect than correct, with no differences between descriptive and binary questions. Conclusions: ChatGPT initially demonstrated good accuracy and completeness, which was further improved with machine learning (ML) over time. However, some inaccurate answers and references persisted. Human critical discernment continues to be essential to facing complex clinical cases and advancing theoretical knowledge and evidence-based practice.
A Comparative Study of ChatGPT-3.5, ChatGPT-4.0, Gemini, Copilot, Chatsonic, and Perplexity

Ali Hakim Reyhan,Çağrı Mutaf,İrfan Uzun,Funda Yüksekyayla

DOI: https://doi.org/10.3390/jcm13216512

IF: 3.9

2024-10-31

Journal of Clinical Medicine

Abstract:Background: This study evaluates the ability of six popular chatbots; ChatGPT-3.5, ChatGPT-4.0, Gemini, Copilot, Chatsonic, and Perplexity, to provide reliable answers to questions concerning keratoconus. Methods: Chatbots responses were assessed using mDISCERN (range: 15–75) and Global Quality Score (GQS) (range: 1–5) metrics. Readability was evaluated using nine validated readability assessments. We also addressed the quality and accountability of websites from which the questions originated. Results: We analyzed 20 websites, 65% "Private practice or independent user" and 35% "Official patient education materials". The mean JAMA benchmark score was 1.40 ± 0.91 (0–4 points), indicating low accountability. Reliability, measured using mDISCERN, ranged from 42.91 ± 3.15 (ChatGPT-3.5) to 46.95 ± 3.53 (Copilot). The most frequent question was "What is keratoconus?" with 70% of websites providing relevant information. This received the highest mDISCERN score (49.30 ± 4.91) and a relatively high GQS score (3.40 ± 0.56) with an Automated Readability Level Calculator score of 13.17 ± 2.13. Moderate positive correlations were determined between the website numbers and both mDISCERN (r = 0.265, p = 0.25) and GQS (r = 0.453, p = 0.05) scores. The quality of information, assessed using the GQS, ranged from 3.02 ± 0.55 (ChatGPT-3.5) to 3.31 ± 0.64 (Gemini) (p = 0.34). The differences between the texts were statistically significant. Gemini emerged as the easiest to read, while ChatGPT-3.5 and Perplexity were the most difficult. Based on mDISCERN scores, Gemini and Copilot exhibited the highest percentage of responses in the "good" range (51–62 points). For the GQS, the Gemini model exhibited the highest percentage of responses in the "good" quality range with 40% of its responses scoring 4–5. Conclusions: While all chatbots performed well, Gemini and Copilot showed better reliability and quality. However, their readability often exceeded recommended levels. Continuous improvements are essential to match information with patients' health literacy for effective use in ophthalmology.

medicine, general & internal
The performance of artificial intelligence models in generating responses to general orthodontic questions: ChatGPT vs Google Bard

Baraa Daraqel,Khaled Wafaie,Hisham Mohammed,Li Cao,Samer Mheissen,Yang Liu,Leilei Zheng

DOI: https://doi.org/10.1016/j.ajodo.2024.01.012

IF: 2.711

2024-03-16

American Journal of Orthodontics and Dentofacial Orthopedics

Abstract:Introduction This study aimed to evaluate and compare the performance of 2 artificial intelligence (AI) models, Chat Generative Pretrained Transformer-3.5 (ChatGPT-3.5; OpenAI, San Francisco, Calif) and Google Bidirectional Encoder Representations from Transformers (Google Bard; Bard Experiment, Google, Mountain View, Calif), in terms of response accuracy, completeness, generation time, and response length when answering general orthodontic questions. Methods A team of orthodontic specialists developed a set of 100 questions in 10 orthodontic domains. One author submitted the questions to both ChatGPT and Google Bard. The AI-generated responses from both models were randomly assigned into 2 forms and sent to 5 blinded and independent assessors. The quality of AI-generated responses was evaluated using a newly developed tool for accuracy of information and completeness. In addition, response generation time and length were recorded. Results The accuracy and completeness of responses were high in both AI models. The median accuracy score was 9 (interquartile range [IQR]: 8-9) for ChatGPT and 8 (IQR: 8-9) for Google Bard (Median difference: 1; P <0.001). The median completeness score was similar in both models, with 8 (IQR: 8-9) for ChatGPT and 8 (IQR: 7-9) for Google Bard. The odds of accuracy and completeness were higher by 31% and 23% in ChatGPT than in Google Bard. Google Bard's response generation time was significantly shorter than that of ChatGPT by 10.4 second/question. However, both models were similar in terms of response length generation. Conclusions Both ChatGPT and Google Bard generated responses were rated with a high level of accuracy and completeness to the posed general orthodontic questions. However, acquiring answers was generally faster using the Google Bard model.

dentistry, oral surgery & medicine
Can ChatGPT help patients answer their otolaryngology questions?

Habib G. Zalzal,Ariel Abraham,Jenhao Cheng,Rahul K. Shah

DOI: https://doi.org/10.1002/lio2.1193

IF: 2.542

2023-12-09

Laryngoscope Investigative Otolaryngology

Abstract:Abstract Background Over the past year, the world has been captivated by the potential of artificial intelligence (AI). The appetite for AI in science, specifically healthcare is huge. It is imperative to understand the credibility of large language models in assisting the public in medical queries. Objective To evaluate the ability of ChatGPT to provide reasonably accurate answers to public queries within the domain of Otolaryngology. Methods Two board‐certified otolaryngologists (HZ, RS) inputted 30 text‐based patient queries into the ChatGPT‐3.5 model. ChatGPT responses were rated by physicians on a scale (accurate, partially accurate, incorrect), while a similar 3‐point scale involving confidence was given to layperson reviewers. Demographic data involving gender and education level was recorded for the public reviewers. Inter‐rater agreement percentage was based on binomial distribution for calculating the 95% confidence intervals and performing significance tests. Statistical significance was defined as p < .05 for two‐sided tests. Results In testing patient queries, both Otolaryngology physicians found that ChatGPT answered 98.3% of questions correctly, but only 79.8% (range 51.7%–100%) of patients were confident that the AI model was accurate in its responses (corrected agreement = 0.682; p < .001). Among the layperson responses, the corrected coefficient was of moderate agreement (0.571; p < .001). No correlation was noted among age, gender, or education level for the layperson responses. Conclusion ChatGPT is highly accurate in responding to questions posed by the public with regards to Otolaryngology from a physician standpoint. Public reviewers were not fully confident in believing the AI model, with subjective concerns related to less trust in AI answers compared to physician explanation. Larger evaluations with a representative public sample and broader medical questions should immediately be conducted by appropriate organizations, governing bodies, and/or governmental agencies to instill public confidence in AI and ChatGPT as a medical resource. Level of Evidence 4.

otorhinolaryngology
Large-Language Models in Orthodontics: Assessing Reliability and Validity of ChatGPT in Pretreatment Patient Education

Stratos Vassis,Harriet Powell,Emma Petersen,Asta Barkmann,Beatrice Noeldeke,Kasper D Kristensen,Peter Stoustrup

DOI: https://doi.org/10.7759/cureus.68085

2024-08-29

Cureus

Abstract:Background: Patients seeking orthodontic treatment may use large language models (LLMs) such as Chat-GPT for self-education, thereby impacting their decision-making process. This study assesses the reliability and validity of Chat-GPT prompts aimed at informing patients about orthodontic side effects and examines patients' perceptions of this information. Materials and methods: To assess reliability, n = 28 individuals were asked to generate information from GPT-3.5 and Generative Pretrained Transformer 4 (GPT-4) about side effects related to orthodontic treatment using both self-formulated and standardized prompts. Three experts evaluated the content generated based on these prompts regarding its validity. We asked a cohort of 46 orthodontic patients about their perceptions after reading an AI-generated information text about orthodontic side effects and compared it with the standard text from the postgraduate orthodontic program at Aarhus University. Results: Although the GPT-generated answers mentioned several relevant side effects, the replies were diverse. The experts rated the AI-generated content generally as "neither deficient nor satisfactory," with GPT-4 achieving higher scores than GPT-3.5. The patients perceived the GPT-generated information as more useful and more comprehensive and experienced less nervousness when reading the GPT-generated information. Nearly 80% of patients preferred the AI-generated information over the standard text. Conclusions: Although patients generally prefer AI-generated information regarding the side effects of orthodontic treatment, the tested prompts fall short of providing thoroughly satisfactory and high-quality education to patients.
Assessing the accuracy and reproducibility of ChatGPT for responding to patient inquiries about otosclerosis

Utku Mete,Ömer Afşın Özmen

DOI: https://doi.org/10.1007/s00405-024-09039-4

2024-10-26

Abstract:Background: Patients increasingly use chatbots powered by artificial intelligence to seek information. However, there is a lack of reliable studies on the accuracy and reproducibility of the information provided by these models. Therefore, we conducted a study investigating the ChatGPT's responses to questions about otosclerosis. Methods: 96 otosclerosis-related questions were collected from internet searches and websites of professional institutions and societies. Questions are divided into four sub-categories. These questions were directed at the latest version of ChatGPT Plus, and these responses were assessed by two otorhinolaryngology surgeons. Accuracy was graded as correct, incomplete, mixed, and irrelevant. Reproducibility was evaluated by comparing the consistency of the two answers to each specific question. Results: The overall accuracy and reproducibility rates of GPT-4o for correct answers were found to be 64.60% and 89.60%, respectively. The findings showed correct answers for accuracy and reproducibility for basic knowledge were 64.70% and 91.20%; for diagnosis & management, 64.0% and 92.0%; for medical & surgical treatment, 52.95% and 82.35%; and for operative risks & postoperative period, 75.0% and 90.0%, respectively. There were no significant differences found between the answers and groups in terms of accuracy and reproducibility (p = 0.073 and p = 0.752, respectively). Conclusion: GPT-4o achieved satisfactory accuracy results, except in the diagnosis & management and medical & surgical treatment categories. Reproducibility was generally high across all categories. With the audio and visual communication capabilities of GPT-4o, under the supervision of a medical professional, this model can be utilized to provide medical information and support for otosclerosis patients.
ChatGPT performance in prosthodontics: Assessment of accuracy and repeatability in answer generation

Yolanda Freire,Andrea Santamaría Laorden,Jaime Orejas Pérez,Margarita Gómez Sánchez,Víctor Díaz-Flores García,Ana Suárez

DOI: https://doi.org/10.1016/j.prosdent.2024.01.018

IF: 4.148

2024-02-03

Journal of Prosthetic Dentistry

Abstract:Statement of problem The artificial intelligence (AI) software program ChatGPT is based on large language models (LLMs) and is widely accessible. However, in prosthodontics, little is known about its performance in generating answers. Purpose The purpose of this study was to determine the performance of ChatGPT in generating answers about removable dental prostheses (RDPs) and tooth-supported fixed dental prostheses (FDPs). Material and methods Thirty short questions were designed about RDPs and tooth-supported FDP, and 30 answers were generated for each of the questions using ChatGPT-4 in October 2023. The 900 generated answers were independently graded by experts using a 3-point Likert scale. The relative frequency and absolute percentage of answers were described. Accuracy was assessed using the Wald binomial method, while repeatability was evaluated using percentage agreement, Brennan and Prediger coefficient, Conger generalized Cohen kappa, Fleiss kappa, Gwet AC, and Krippendorff alpha methods. Confidence intervals were set at 95%. Statistical analysis was performed using the STATA software program. Results The performance of ChatGPT in generating answers related to RDP and tooth-supported FDP was limited. The answers showed a reliability of 25.6%, with a confidence range between 22.9% and 28.6%. The repeatability ranged from substantial to moderate. Conclusions The results show that currently ChatGPT has limited ability to generate answers related to RDPs and tooth-supported FDPs. Therefore, ChatGPT cannot replace a dentist, and, if professionals were to use it, they should be aware of its limitations.

dentistry, oral surgery & medicine
Artificial intelligence performance in clinical neurology queries: the ChatGPT model

Erman Altunisik,Yasemin Ekmekyapar Firat,Emine Kilicparlar Cengiz,Gulsum Bayana Comruk

DOI: https://doi.org/10.1080/01616412.2024.2334118

2024-03-26

Neurological Research

Abstract:Introduction The use of artificial intelligence technology is progressively expanding and advancing in the health and biomedical literature. Since its launch, ChatGPT has rapidly gained popularity and become one of the fastest-growing artificial intelligence applications in history. This study evaluated the accuracy and comprehensiveness of ChatGPT-generated responses to medical queries in clinical neurology.

neurosciences,clinical neurology
Comparative performance of artificial intelligence models in rheumatology board-level questions: evaluating Google Gemini and ChatGPT-4o

Enes Efe Is,Ahmet Kivanc Menekseoglu

DOI: https://doi.org/10.1007/s10067-024-07154-5

2024-09-29

Clinical Rheumatology

Abstract:This study evaluates the performance of AI models, ChatGPT-4o and Google Gemini, in answering rheumatology board-level questions, comparing their effectiveness, reliability, and applicability in clinical practice.

rheumatology
Evaluating the reliability of the responses of large language models to keratoconus-related questions

Mustafa Kayabaşı,Seher Köksaldı,Ceren Durmaz Engin

DOI: https://doi.org/10.1080/08164622.2024.2419524

2024-10-24

Abstract:Clinical relevance: Artificial intelligence has undergone a rapid evolution and large language models (LLMs) have become promising tools for healthcare, with the ability of providing human-like responses to questions. The capabilities of these tools in addressing questions related to keratoconus (KCN) have not been previously explored. Background: In this study, the responses were evaluated from three LLMs - ChatGPT-4, Copilot, and Gemini - to common patient questions regarding KCN. Methods: Fifty real-life patient inquiries regarding general information, aetiology, symptoms and diagnosis, progression, and treatment of KCN were presented to the LLMs. Evaluations of the answers were conducted by three ophthalmologists with a 5-point Likert scale ranging from 'strongly disagreed' to 'strongly agreed'. The reliability of the responses provided by LLMs was evaluated using the DISCERN and the Ensuring Quality Information for Patients (EQIP) scales. Readability metrics (Flesch Reading Ease Score, Flesch-Kincaid Grade Level, and Coleman-Liau Index) were calculated to evaluate the complexity of responses. Results: ChatGPT-4 consistently scored 3 points or higher for all (100%) its responses, while Copilot had five (10%) and Gemini had two (4%) responses scoring 2 points or below. ChatGPT-4 achieved a 'strongly agree' rate of 74% across all questions, markedly superior to Copilot at 34% and Gemini at 42% (p < 0.001); and recorded the highest 'strongly agree' rates in general information and symptoms & diagnosis categories (90% for both). The median Likert scores differed among LLMs (p < 0.001), with ChatGPT-4 scoring highest and Copilot scoring lowest. Although ChatGPT-4 exhibited more reliability based on the DISCERN scale, it was characterised by lower readability and higher complexity. While all LLMs provided responses categorised as 'extremely difficult to read', the responses provided by Copilot showed higher readability. Conclusions: Despite the responses provided by ChatGPT-4 exhibiting lower readability and greater complexity, it emerged as the most proficient in answering KCN-related questions.
Artificial intelligence in dental education: ChatGPT's performance on the periodontic in‐service examination

Arman Danesh,Hirad Pazouki,Farzad Danesh,Arsalan Danesh,Saynur Vardar‐Sengul

DOI: https://doi.org/10.1002/jper.23-0514

2024-01-11

Journal of Periodontology

Abstract:Background ChatGPT's (Chat Generative Pre‐Trained Transformer) remarkable capacity to generate human‐like output makes it an appealing learning tool for healthcare students worldwide. Nevertheless, the chatbot's responses may be subject to inaccuracies, putting forth an intense risk of misinformation. ChatGPT's capabilities should be examined in every corner of healthcare education, including dentistry and its specialties, to understand the potential of misinformation associated with the chatbot's use as a learning tool. Our investigation aims to explore ChatGPT's foundation of knowledge in the field of periodontology by evaluating the chatbot's performance on questions obtained from an in‐service examination administered by the American Academy of Periodontology (AAP). Methods ChatGPT3.5 and ChatGPT4 were evaluated on 311 multiple‐choice questions obtained from the 2023 in‐service examination administered by the AAP. The dataset of in‐service examination questions was accessed through Nova Southeastern University's Department of Periodontology. Our study excluded questions containing an image as ChatGPT does not accept image inputs. Results ChatGPT3.5 and ChatGPT4 answered 57.9% and 73.6% of in‐service questions correctly on the 2023 Periodontics In‐Service Written Examination, respectively. A two‐tailed t test was incorporated to compare independent sample means, and sample proportions were compared using a two‐tailed χ2 test. A p value below the threshold of 0.05 was deemed statistically significant. Conclusion While ChatGPT4 showed a higher proficiency compared to ChatGPT3.5, both chatbot models leave considerable room for misinformation with their responses relating to periodontology. The findings of the study encourage residents to scrutinize the periodontic information generated by ChatGPT to account for the chatbot's current limitations.

dentistry, oral surgery & medicine
Innovating dental diagnostics: ChatGPT's accuracy on diagnostic challenges

Arman Danesh,Arsalan Danesh,Farzad Danesh

DOI: https://doi.org/10.1111/odi.15082

2024-07-22

Abstract:Introduction: Complex patient diagnoses in dentistry require a multifaceted approach which combines interpretations of clinical observations with an in-depth understanding of patient history and presenting problems. The present study aims to elucidate the implications of ChatGPT (OpenAI) as a comprehensive diagnostic tool in the dental clinic through examining the chatbot's diagnostic performance on challenging patient cases retrieved from the literature. Methods: Our study subjected ChatGPT3.5 and ChatGPT4 to descriptions of patient cases for diagnostic challenges retrieved from the literature. Sample means were compared using a two-tailed t-test, while sample proportions were compared using a two-tailed χ2 test. A p-value below the threshold of 0.05 was deemed statistically significant. Results: When prompted to generate their own differential diagnoses, ChatGPT3.5 and ChatGPT4 achieved a diagnostic accuracy of 40% and 62%, respectively. When basing their diagnostic processes on a differential diagnosis retrieved from the literature, ChatGPT3.5 and ChatGPT4 achieved a diagnostic accuracy of 70% and 80%, respectively. Conclusion: ChatGPT displays an impressive capacity to correctly diagnose complex diagnostic challenges in the field of dentistry. Our study paints a promising potential for the chatbot to 1 day serve as a comprehensive diagnostic tool in the dental clinic.

Can artificial intelligence models serve as patient information consultants in orthodontics?

How reliable is the artificial intelligence product large language model ChatGPT in orthodontics?

Content analysis of AI-generated (ChatGPT) responses concerning orthodontic clear aligners

An evaluation of orthodontic information quality regarding artificial intelligence (AI) chatbot technologies: A comparison of ChatGPT and google BARD

Evaluating ChatGPT-4's performance as a digital health advisor for otosclerosis surgery

A comparative analysis of AI-based chatbots: Assessing data quality in orthognathic surgery related patient information

"Dr. AI Will See You Now": How Do ChatGPT-4 Treatment Recommendations Align With Orthopaedic Clinical Practice Guidelines?

Assessing the Accuracy of AI Models in Orthodontic Knowledge: A Comparative Study Between ChatGPT-4 and Google Bard

Assessing the Accuracy, Completeness, and Reliability of Artificial Intelligence-Generated Responses in Dentistry: A Pilot Study Evaluating the ChatGPT Model

A Comparative Study of ChatGPT-3.5, ChatGPT-4.0, Gemini, Copilot, Chatsonic, and Perplexity

The performance of artificial intelligence models in generating responses to general orthodontic questions: ChatGPT vs Google Bard

Can ChatGPT help patients answer their otolaryngology questions?

Large-Language Models in Orthodontics: Assessing Reliability and Validity of ChatGPT in Pretreatment Patient Education

Assessing the accuracy and reproducibility of ChatGPT for responding to patient inquiries about otosclerosis

ChatGPT performance in prosthodontics: Assessment of accuracy and repeatability in answer generation

Artificial intelligence performance in clinical neurology queries: the ChatGPT model

Comparative performance of artificial intelligence models in rheumatology board-level questions: evaluating Google Gemini and ChatGPT-4o

Evaluating the reliability of the responses of large language models to keratoconus-related questions

Artificial intelligence in dental education: ChatGPT's performance on the periodontic in‐service examination

Innovating dental diagnostics: ChatGPT's accuracy on diagnostic challenges