LOCALIZATION OF RADIOACTIVE ELEMENTS : (A Review).

J. Gross,C. P. Leblond

Canadian Medical Association Journal

Abstract:

What problem does this paper attempt to address?

Can ChatGPT help patients answer their otolaryngology questions?

Habib G. Zalzal,Ariel Abraham,Jenhao Cheng,Rahul K. Shah

DOI: https://doi.org/10.1002/lio2.1193

IF: 2.542

2023-12-09

Laryngoscope Investigative Otolaryngology

Abstract:Abstract Background Over the past year, the world has been captivated by the potential of artificial intelligence (AI). The appetite for AI in science, specifically healthcare is huge. It is imperative to understand the credibility of large language models in assisting the public in medical queries. Objective To evaluate the ability of ChatGPT to provide reasonably accurate answers to public queries within the domain of Otolaryngology. Methods Two board‐certified otolaryngologists (HZ, RS) inputted 30 text‐based patient queries into the ChatGPT‐3.5 model. ChatGPT responses were rated by physicians on a scale (accurate, partially accurate, incorrect), while a similar 3‐point scale involving confidence was given to layperson reviewers. Demographic data involving gender and education level was recorded for the public reviewers. Inter‐rater agreement percentage was based on binomial distribution for calculating the 95% confidence intervals and performing significance tests. Statistical significance was defined as p < .05 for two‐sided tests. Results In testing patient queries, both Otolaryngology physicians found that ChatGPT answered 98.3% of questions correctly, but only 79.8% (range 51.7%–100%) of patients were confident that the AI model was accurate in its responses (corrected agreement = 0.682; p < .001). Among the layperson responses, the corrected coefficient was of moderate agreement (0.571; p < .001). No correlation was noted among age, gender, or education level for the layperson responses. Conclusion ChatGPT is highly accurate in responding to questions posed by the public with regards to Otolaryngology from a physician standpoint. Public reviewers were not fully confident in believing the AI model, with subjective concerns related to less trust in AI answers compared to physician explanation. Larger evaluations with a representative public sample and broader medical questions should immediately be conducted by appropriate organizations, governing bodies, and/or governmental agencies to instill public confidence in AI and ChatGPT as a medical resource. Level of Evidence 4.

otorhinolaryngology
Assessing ChatGPT's Responses to Otolaryngology Patient Questions

Jonathan M. Carnino,William R. Pellegrini,Megan Willis,Michael B. Cohen,Marianella Paz-Lansberg,Elizabeth M. Davis,Gregory A. Grillone,Jessica R. Levi

DOI: https://doi.org/10.1177/00034894241249621

2024-04-28

Annals of Otology Rhinology & Laryngology

Abstract:Annals of Otology, Rhinology &Laryngology, Ahead of Print. Objective:This study aims to evaluate ChatGPT's performance in addressing real-world otolaryngology patient questions, focusing on accuracy, comprehensiveness, and patient safety, to assess its suitability for integration into healthcare.Methods:A cross-sectional study was conducted using patient questions from the public online forum Reddit's r/AskDocs, where medical advice is sought from healthcare professionals. Patient questions were input into ChatGPT (GPT-3.5), and responses were reviewed by 5 board-certified otolaryngologists. The evaluation criteria included difficulty, accuracy, comprehensiveness, and bedside manner/empathy. Statistical analysis explored the relationship between patient question characteristics and ChatGPT response scores. Potentially dangerous responses were also identified.Results:Patient questions averaged 224.93 words, while ChatGPT responses were longer at 414.93 words. The accuracy scores for ChatGPT responses were 3.76/5, comprehensiveness scores were 3.59/5, and bedside manner/empathy scores were 4.28/5. Longer patient questions did not correlate with higher response ratings. However, longer ChatGPT responses scored higher in bedside manner/empathy. Higher question difficulty correlated with lower comprehensiveness. Five responses were flagged as potentially dangerous.Conclusion:While ChatGPT exhibits promise in addressing otolaryngology patient questions, this study demonstrates its limitations, particularly in accuracy and comprehensiveness. The identification of potentially dangerous responses underscores the need for a cautious approach to AI in medical advice. Responsible integration of AI into healthcare necessitates thorough assessments of model performance and ethical considerations for patient safety.

otorhinolaryngology
Is ChatGPT 3.5 smarter than Otolaryngology trainees? A comparison study of board style exam questions

Jaimin Patel,Peyton Robinson,Elisa Illing,Benjamin Anthony

DOI: https://doi.org/10.1371/journal.pone.0306233

IF: 3.7

2024-09-26

PLoS ONE

Abstract:Objectives: This study compares the performance of the artificial intelligence (AI) platform Chat Generative Pre-Trained Transformer (ChatGPT) to Otolaryngology trainees on board-style exam questions. Methods: We administered a set of 30 Otolaryngology board-style questions to medical students (MS) and Otolaryngology residents (OR). 31 MSs and 17 ORs completed the questionnaire. The same test was administered to ChatGPT version 3.5, five times. Comparisons of performance were achieved using a one-way ANOVA with Tukey Post Hoc test, along with a regression analysis to explore the relationship between education level and performance. Results: The average scores increased each year from MS1 to PGY5. A one-way ANOVA revealed that ChatGPT outperformed trainee years MS1, MS2, and MS3 (p = <0.001, 0.003, and 0.019, respectively). PGY4 and PGY5 otolaryngology residents outperformed ChatGPT (p = 0.033 and 0.002, respectively). For years MS4, PGY1, PGY2, and PGY3 there was no statistical difference between trainee scores and ChatGPT (p = .104, .996, and 1.000, respectively). Conclusion: ChatGPT can outperform lower-level medical trainees on Otolaryngology board-style exam but still lacks the ability to outperform higher-level trainees. These questions primarily test rote memorization of medical facts; in contrast, the art of practicing medicine is predicated on the synthesis of complex presentations of disease and multilayered application of knowledge of the healing process. Given that upper-level trainees outperform ChatGPT, it is unlikely that ChatGPT, in its current form will provide significant clinical utility over an Otolaryngologist.
Is ChatGPT smarter than Otolaryngology trainees? A comparison study of board style exam questions

Jaimin Patel,Peyton Robinson,Elisa Illing,Benjamin Anthony

DOI: https://doi.org/10.1101/2024.06.16.24308998

2024-06-18

Abstract:Abstract Objectives: This study compares the performance of the artificial intelligence (AI) platform Chat Generative Pre-Trained Transformer (ChatGPT) to Otolaryngology trainees on board style exam questions. Methods: We administered a set of 30 Otolaryngology board style questions to medical students (MS) and Otolaryngology residents (OR). 31 MSs and 17 ORs completed the questionnaire. The same test was administered to ChatGPT version 3.5, five times. Comparisons of performance were achieved using a one-way ANOVA with Tukey Post Hoc test, along with a regression analysis to explore the relationship between education level and performance. Results: The average scores increased each year from MS1 to PGY5. A one-way ANOVA revealed that ChatGPT outperformed trainee years MS1, MS2, and MS3 (p = <0.001, 0.003, and 0.019, respectively). PGY4 and PGY5 otolaryngology residents outperformed ChatGPT (p = 0.033 and 0.002, respectively). For years MS4, PGY1, PGY2, and PGY3 there was no statistical difference between trainee scores and ChatGPT (p = .104, .996, and 1.000, respectively). Conclusion: ChatGPT can outperform lower-level medical trainees on Otolaryngology board-style exam but still lacks the ability to outperform higher-level trainees. These questions primarily test rote memorization of medical facts; in contrast, the art of practicing medicine is predicated on the synthesis of complex presentations of disease and multilayered application of knowledge of the healing process. Given that upper-level trainees outperform ChatGPT, it is unlikely that ChatGPT, in its current form will provide significant clinical utility over an Otolaryngologist.

Otolaryngology
Does ChatGPT Answer Otolaryngology Questions Accurately?

Matthew Maksimoski,Anisha Rhea Noble,David F. Smith

DOI: https://doi.org/10.1002/lary.31410

IF: 2.97

2024-03-30

The Laryngoscope

Abstract:Artificial intelligence software has become incredibly popular, but this study showed that it answered otolaryngology questions correctly 45% of the time, and only 37% of the time for questions based on clinical practice guidelines. Practitioners and patients should exercise caution when asking medical information from software such as ChatGPT, especially since sources are not cited for the information it presents. Objective Investigate the accuracy of ChatGPT in the manner of medical questions related to otolaryngology. Methods A ChatGPT session was opened within which 93 questions were asked related to otolaryngology topics. Questions were drawn from all major domains within otolaryngology and based upon key action statements (KAS) from clinical practice guidelines (CPGs). Twenty‐one "patient‐level" questions were also asked of the program. Answers were graded as either "correct," "partially correct," "incorrect," or "non‐answer." Results Correct answers were given at a rate of 45.5% (71.4% correct in patient‐level, 37.3% CPG); partially correct answers at 31.8% (28.6% patient‐level, 32.8% CPG); incorrect at 21.6% (0% patient‐level, 28.4% CPG); and 1.1% non‐answers (% patient‐level, 1.5% CPG). There was no difference in the rate of correct answers between CPGs published before or after the period of data collection cited by ChatGPT. CPG‐based questions were less likely to be correct than patient‐level questions (p = 0.003). Conclusion Publicly available artificial intelligence software has become increasingly popular with consumers for everything from story‐telling to data collection. In this study, we examined the accuracy of ChatGPT responses to questions related to otolaryngology over 7 domains and 21 published CPGs. Physicians and patients should understand the limitations of this software as it applies to otolaryngology, and programmers in future iterations should consider giving greater weight to information published by well‐established journals and written by national content experts. Level of Evidence N/A Laryngoscope, 2024

medicine, research & experimental,otorhinolaryngology
Effects of mycorrhizae on growth and demography of tallgrass prairie forbs.

G. Wilson,D. Hartnett,Melinda D. Smith,Kerri Kobbeman

DOI: https://doi.org/10.2307/3558453

2001-08-01

Abstract:The effects of arbuscular mycorrhizal (AM) symbiosis on ramet and genet densities, vegetative growth rates, and flowering of three forb species were studied in native tallgrass prairie in northeastern Kansas. Mycorrhizal activity was experimentally suppressed for six growing seasons on replicate plots in an annually burned and an infrequently burned watershed with the fungicide benomyl. Benomyl reduced mycorrhizal root colonization to an average of 4.2%, approximately a two-thirds reduction relative to controls (13.7% colonization). Mycorrhizae influenced the population structure of these forbs. Although mycorrhizal suppression had no long-term effect on genet densities and no effect on ramet survivorship throughout the growing season, the number of ramets per individual was significantly increased such that ramet densities of all three species were approximately doubled in response to long-term mycorrhizal suppression. Effects of mycorrhizae on ramet growth and reproduction varied among species. Ramet growth rates, biomass, and flowering of Salvia azurea were greater in plots with active mycorrhizal symbiosis, whereas mycorrhizae reduced ramet growth rates and biomass of Artemesia ludoviciana. Aster sericeus ramet growth rates and biomass were unaffected by the fungicide applications, but its flowering was reduced.The pattern of responses of these three species to mycorrhizae differed considerably between the two sites of contrasting fire regime, indicating that the interaction of fire-induced shifts in resource availability and mycorrhizal symbiosis together modulates plant responses and the intensity and patterns of interspecific competition between and among tallgrass prairie grass and forb species. Further, the results indicate that effects of mycorrhizae on community structure are a result of interspecific differences in the balance between direct positive effects of the symbiosis on host plant performance and indirect negative effects mediated through altered competitive interactions.
ChatGPT's adherence to otolaryngology clinical practice guidelines

Idit Tessler,Amit Wolfovitz,Eran E Alon,Nir A Gecel,Nir Livneh,Eyal Zimlichman,Eyal Klang

DOI: https://doi.org/10.1007/s00405-024-08634-9

Abstract:Objectives: Large language models, including ChatGPT, has the potential to transform the way we approach medical knowledge, yet accuracy in clinical topics is critical. Here we assessed ChatGPT's performance in adhering to the American Academy of Otolaryngology-Head and Neck Surgery guidelines. Methods: We presented ChatGPT with 24 clinical otolaryngology questions based on the guidelines of the American Academy of Otolaryngology. This was done three times (N = 72) to test the model's consistency. Two otolaryngologists evaluated the responses for accuracy and relevance to the guidelines. Cohen's Kappa was used to measure evaluator agreement, and Cronbach's alpha assessed the consistency of ChatGPT's responses. Results: The study revealed mixed results; 59.7% (43/72) of ChatGPT's responses were highly accurate, while only 2.8% (2/72) directly contradicted the guidelines. The model showed 100% accuracy in Head and Neck, but lower accuracy in Rhinology and Otology/Neurotology (66%), Laryngology (50%), and Pediatrics (8%). The model's responses were consistent in 17/24 (70.8%), with a Cronbach's alpha value of 0.87, indicating a reasonable consistency across tests. Conclusions: Using a guideline-based set of structured questions, ChatGPT demonstrates consistency but variable accuracy in otolaryngology. Its lower performance in some areas, especially Pediatrics, suggests that further rigorous evaluation is needed before considering real-world clinical use.
Assessing ChatGPT's potential as a clinical resource for medical oncologists: An evaluation with board-style questions and real-world patient cases.

Hyun Lee,Zahra Hamedi,David Oliver,Hui Lin,Uzma Athar,Hiba Zainab Vohra,Peter Daniels,Sejal Kothadia

DOI: https://doi.org/10.1200/jco.2024.42.16_suppl.e13628

IF: 45.3

2024-05-31

Journal of Clinical Oncology

Abstract:e13628 Background: ChatGPT is a conversational artificial intelligence (AI) model that learns from massive text-based datasets and then responds to user input, which often involves completing tasks or answering questions. Recent studies showed ChatGPT’s success in passing multiple specialty medical licensing and board examinations, showcasing its promising capabilities in the medical domain. Here, we investigated ChatGPT's potential as a swift and reliable information source for medical oncologists using board examination style questions and real patient cases. Methods: We randomly selected 121 board-style questions from the American Society of Clinical Oncology Self-Evaluation Program (ASCO SEP). The questions were entered into ChatGPT in both multiple-choice (MC) and open-ended (OE) prompts. ChatGPT’s answers and explanations were evaluated for accuracy and concordance. Non-inferiority analysis was performed with power of 80% at α = 0.05 and non-inferiority margin set at 70% correct answers given the historical board exam pass rate of about 65% correct answers. For subgroup analysis, the questions were categorized by tested competency and primary tumor pathology. ChatGPT was also given 10 questions derived from real patient cases. We compared its responses to the answers provided by experienced oncologists to determine accuracy and practical applicability. Results: ChatGPT answered 75 (62.0%) MC queries correctly. Among the correctly answered queries, 2 responses contained faulty explanations. Such inaccurate or discordant explanations were found in 26 of the 46 incorrectly answered queries. In OE prompts, ChatGPT answered 53 (43.8%) questions correctly with correct explanations for all. Of the 68 incorrect responses, 32 of them contained inaccurate or discordant explanations. Subgroup analysis suggested varying performance across the categories. The best performance was seen with malignant hematology (81.8% of MC and 72.8% of OE prompts answered correctly) while the weakest performance was seen with genitourinary malignancies (60% of MC and 20% of OE prompts answered correctly). As for the real-world patient case questions, responses from ChatGPT and the clinicians were concordant in 5 questions. None of the discordant responses contained inaccurate information while 80% of the concordant responses contained sufficient details to assist with patient management decisions. Conclusions: ChatGPT's performance fell short of the non-inferiority margin, highlighting the challenges with incorporating AI in the rapidly evolving field of medical oncology. Despite the limitations, ChatGPT’s partial success, in both board-style and real-world patient care questions, affirms its potential for clinical utility in future.

oncology
The Accuracy of Artificial Intelligence ChatGPT in Oncology Examination Questions

Ronald Chow,Shaakir Hasan,Ajay Zheng,Chenxi Gao,Gilmer Valdes,Francis Yu,Arpit Chhabra,Srinivas Raman,J. Isabelle Choi,Haibo Lin,Charles B. Simone

DOI: https://doi.org/10.1016/j.jacr.2024.07.011

IF: 6.24

2024-08-28

Journal of the American College of Radiology

Abstract:The aim of this study is to assess the accuracy of Chat Generative Pretrained Transformer (ChatGPT) in response to oncology examination questions in the setting of one-shot learning. Consecutive national radiation oncology in-service multiple-choice examinations were collected and inputted into ChatGPT 4o and ChatGPT 3.5 to determine ChatGPT's answers. ChatGPT's answers were then compared with the answer keys to determine whether ChatGPT correctly or incorrectly answered each question and to determine if improvements in responses were seen with the newer ChatGPT version. A total of 600 consecutive questions were inputted into ChatGPT. ChatGPT 4o answered 72.2% questions correctly, whereas 3.5 answered 53.8% questions correctly. There was a significant difference in performance by question category ( P < .01). ChatGPT performed poorer with respect to knowledge of landmark studies and treatment recommendations and planning. ChatGPT is a promising technology, with the latest version showing marked improvement. Although it still has limitations, with further evolution, it may be considered a reliable resource for medical training and decision making in the oncology space.

radiology, nuclear medicine & medical imaging
Assessment of Artificial Intelligence Performance on the Otolaryngology Residency In-Service Exam

Arushi P Mahajan,Christina L Shabet,Joshua Smith,Shannon F Rudy,Robbi A Kupfer,Lauren A Bohm

DOI: https://doi.org/10.1002/oto2.98

2023-11-29

OTO Open

Abstract:Objectives: This study seeks to determine the potential use and reliability of a large language learning model for answering questions in a sub-specialized area of medicine, specifically practice exam questions in otolaryngology-head and neck surgery and assess its current efficacy for surgical trainees and learners. Study design and setting: All available questions from a public, paid-access question bank were manually input through ChatGPT. Methods: Outputs from ChatGPT were compared against the benchmark of the answers and explanations from the question bank. Questions were assessed in 2 domains: accuracy and comprehensiveness of explanations. Results: Overall, our study demonstrates a ChatGPT correct answer rate of 53% and a correct explanation rate of 54%. We find that with increasing difficulty of questions there is a decreasing rate of answer and explanation accuracy. Conclusion: Currently, artificial intelligence-driven learning platforms are not robust enough to be reliable medical education resources to assist learners in sub-specialty specific patient decision making scenarios.
A cross-sectional comparative study: ChatGPT 3.5 versus diverse levels of medical experts in the diagnosis of ENT diseases

Mikhael Makhoul,Antoine E. Melkane,Patrick El Khoury,Christopher El Hadi,Nayla Matar

DOI: https://doi.org/10.1007/s00405-024-08509-z

2024-02-16

European Archives of Oto-Rhino-Laryngology

Abstract:PurposeWith recent advances in artificial intelligence (AI), it has become crucial to thoroughly evaluate its applicability in healthcare. This study aimed to assess the accuracy of ChatGPT in diagnosing ear, nose, and throat (ENT) pathology, and comparing its performance to that of medical experts.MethodsWe conducted a cross-sectional comparative study where 32 ENT cases were presented to ChatGPT 3.5, ENT physicians, ENT residents, family medicine (FM) specialists, second-year medical students (Med2), and third-year medical students (Med3). Each participant provided three differential diagnoses. The study analyzed diagnostic accuracy rates and inter-rater agreement within and between participant groups and ChatGPT.ResultsThe accuracy rate of ChatGPT was 70.8%, being not significantly different from ENT physicians or ENT residents. However, a significant difference in correctness rate existed between ChatGPT and FM specialists (49.8%, p < 0.001), and between ChatGPT and medical students (Med2 47.5%, p < 0.001; Med3 47%, p < 0.001). Inter-rater agreement for the differential diagnosis between ChatGPT and each participant group was either poor or fair. In 68.75% of cases, ChatGPT failed to mention the most critical diagnosis.ConclusionsChatGPT demonstrated accuracy comparable to that of ENT physicians and ENT residents in diagnosing ENT pathology, outperforming FM specialists, Med2 and Med3. However, it showed limitations in identifying the most critical diagnosis.

otorhinolaryngology
Evaluating ChatGPT Performance on the Orthopaedic In-Training Examination

Justin E Kung,Christopher Marshall,Chase Gauthier,Tyler A Gonzalez,J Benjamin Jackson 3rd

DOI: https://doi.org/10.2106/JBJS.OA.23.00056

2023-09-08

JBJS Open Access

Abstract:Background: Artificial intelligence (AI) holds potential in improving medical education and healthcare delivery. ChatGPT is a state-of-the-art natural language processing AI model which has shown impressive capabilities, scoring in the top percentiles on numerous standardized examinations, including the Uniform Bar Exam and Scholastic Aptitude Test. The goal of this study was to evaluate ChatGPT performance on the Orthopaedic In-Training Examination (OITE), an assessment of medical knowledge for orthopedic residents. Methods: OITE 2020, 2021, and 2022 questions without images were inputted into ChatGPT version 3.5 and version 4 (GPT-4) with zero prompting. The performance of ChatGPT was evaluated as a percentage of correct responses and compared with the national average of orthopedic surgery residents at each postgraduate year (PGY) level. ChatGPT was asked to provide a source for its answer, which was categorized as being a journal article, book, or website, and if the source could be verified. Impact factor for the journal cited was also recorded. Results: ChatGPT answered 196 of 360 answers correctly (54.3%), corresponding to a PGY-1 level. ChatGPT cited a verifiable source in 47.2% of questions, with an average median journal impact factor of 5.4. GPT-4 answered 265 of 360 questions correctly (73.6%), corresponding to the average performance of a PGY-5 and exceeding the corresponding passing score for the American Board of Orthopaedic Surgery Part I Examination of 67%. GPT-4 cited a verifiable source in 87.9% of questions, with an average median journal impact factor of 5.2. Conclusions: ChatGPT performed above the average PGY-1 level and GPT-4 performed better than the average PGY-5 level, showing major improvement. Further investigation is needed to determine how successive versions of ChatGPT would perform and how to optimize this technology to improve medical education. Clinical relevance: AI has the potential to aid in medical education and healthcare delivery.
Comparative Performance of ChatGPT 3.5 and GPT4 on Rhinology Standardized Board Examination Questions

Evan A Patel,Lindsay Fleischer,Peter Filip,Michael Eggerstedt,Michael Hutz,Elias Michaelides,Pete S Batra,Bobby A Tajudeen

DOI: https://doi.org/10.1002/oto2.164

2024-06-27

OTO Open

Abstract:Objective: Advances in deep learning and artificial intelligence (AI) have led to the emergence of large language models (LLM) like ChatGPT from OpenAI. The study aimed to evaluate the performance of ChatGPT 3.5 and GPT4 on Otolaryngology (Rhinology) Standardized Board Examination questions in comparison to Otolaryngology residents. Methods: This study selected all 127 rhinology standardized questions from www.boardvitals.com, a commonly used study tool by otolaryngology residents preparing for board exams. Ninety-three text-based questions were administered to ChatGPT 3.5 and GPT4, and their answers were compared with the average results of the question bank (used primarily by otolaryngology residents). Thirty-four image-based questions were provided to GPT4 and underwent the same analysis. Based on the findings of an earlier study, a pass-fail cutoff was set at the 10th percentile. Results: On text-based questions, ChatGPT 3.5 answered correctly 45.2% of the time (8th percentile) (P = .0001), while GPT4 achieved 86.0% (66th percentile) (P = .001). GPT4 answered image-based questions correctly 64.7% of the time. Projections suggest that ChatGPT 3.5 might not pass the American Board of Otolaryngology Written Question Exam (ABOto WQE), whereas GPT4 stands a strong chance of passing. Discussion: The older LLM, ChatGPT 3.5, is unlikely to pass the ABOto WQE. However, the advanced GPT4 model exhibits a much higher likelihood of success. This rapid progression in AI indicates its potential future role in otolaryngology education. Implications for practice: As AI technology rapidly advances, it may be that AI-assisted medical education, diagnosis, and treatment planning become commonplace in the medical and surgical landscape. Level of evidence: Level 5.
Leprosy work--the Bethany experience.

S. Paul

1981-03-01

Abstract:
Total-energy and pressure calculations for random substitutional alloys.

Duane D. Johnson,D. Nicholson,F. J. Pinski,B. L. Gyorffy,G. M. Stocks

DOI: https://doi.org/10.1103/PHYSREVB.41.9701

1990-05-15

Abstract:We present the details and the derivation of density-functional-based expressions for the total energy and pressure for random substitutional alloys (RSA) using the Korringa-Kohn-Rostoker Green's-function approach in combination with the coherent-potential approximation (CPA) to treat the configurational averaging. This includes algebraic cancellation of various electronic core contributions to the total energy and pressure, as in ordered-solid muffin-tin-potential calculations. Thus, within the CPA, total-energy and pressure calculations for RSA have the same foundation and have been found to have the same accuracy as those obtained in similar calculations for ordered solids. Results of our calculations for the impurity formation energy, and for the bulk moduli, the lattice parameters, and the energy of mixing as a function of concentration in fcc Cu{sub {ital c}}Zn{sub 1{minus}{ital c}} alloys show that this generalized density-functional theory will be useful in studying alloy phase stability.
Inadequate Performance of ChatGPT on Orthopedic Board-Style Written Exams

Chandler A Sparks,Matthew J Kraeutler,Grace A Chester,Edward V Contrada,Eric Zhu,Sydney M Fasulo,Anthony J Scillia

DOI: https://doi.org/10.7759/cureus.62643

2024-06-18

Cureus

Abstract:Background: Chat Generative Pre-Trained Transformer (ChatGPT) is an artificial intelligence (AI) chatbot capable of delivering human-like responses to a seemingly infinite number of inquiries. For the technology to perform certain healthcare-related tasks or act as a study aid, the technology should have up-to-date knowledge and the ability to reason through medical information. The purpose of this study was to assess the orthopedic knowledge and reasoning ability of ChatGPT by querying it with orthopedic board-style questions. Methodology: We queried ChatGPT (GPT-3.5) with a total of 472 questions from the Orthobullets dataset (n = 239), the 2022 Orthopaedic In-Training Examination (OITE) (n = 124), and the 2021 OITE (n = 109). The importance, difficulty, and category were recorded for questions from the Orthobullets question bank. Responses were assessed for answer choice correctness if the explanation given matched that of the dataset, answer integrity, and reason for incorrectness. Results: ChatGPT correctly answered 55.9% (264/472) of questions and, of those answered correctly, gave an explanation that matched that of the dataset for 92.8% (245/264) of the questions. The chatbot used information internal to the question in all responses (100%) and used information external to the question (98.3%) as well as logical reasoning (96.4%) in most responses. There was no significant difference in the proportion of questions answered correctly between the datasets (P = 0.62). There was no significant difference in the proportion of questions answered correctly by question category (P = 0.67), importance (P = 0.95), or difficulty (P = 0.87) within the Orthobullets dataset questions. ChatGPT mostly got questions incorrect due to information error (i.e., failure to identify the information required to answer the question) (81.7% of incorrect responses). Conclusions: ChatGPT performs below a threshold likely to pass the American Board of Orthopedic Surgery (ABOS) Part I written exam. The chatbot's performance on the 2022 and 2021 OITEs was between the average performance of an intern and to second-year resident. A major limitation of the current model is the failure to identify the information required to correctly answer the questions.
Integrating ChatGPT in Orthopedic Education for Medical Undergraduates: Randomized Controlled Trial

Wenyi Gan,Jianfeng Ouyang,Hua Li,Zhaowen Xue,Yiming Zhang,Qiu Dong,Jiadong Huang,Xiaofei Zheng,Yiyi Zhang

DOI: https://doi.org/10.2196/57037

2024-08-20

Abstract:Background: ChatGPT is a natural language processing model developed by OpenAI, which can be iteratively updated and optimized to accommodate the changing and complex requirements of human verbal communication. Objective: The study aimed to evaluate ChatGPT's accuracy in answering orthopedics-related multiple-choice questions (MCQs) and assess its short-term effects as a learning aid through a randomized controlled trial. In addition, long-term effects on student performance in other subjects were measured using final examination results. Methods: We first evaluated ChatGPT's accuracy in answering MCQs pertaining to orthopedics across various question formats. Then, 129 undergraduate medical students participated in a randomized controlled study in which the ChatGPT group used ChatGPT as a learning tool, while the control group was prohibited from using artificial intelligence software to support learning. Following a 2-week intervention, the 2 groups' understanding of orthopedics was assessed by an orthopedics test, and variations in the 2 groups' performance in other disciplines were noted through a follow-up at the end of the semester. Results: ChatGPT-4.0 answered 1051 orthopedics-related MCQs with a 70.60% (742/1051) accuracy rate, including 71.8% (237/330) accuracy for A1 MCQs, 73.7% (330/448) accuracy for A2 MCQs, 70.2% (92/131) accuracy for A3/4 MCQs, and 58.5% (83/142) accuracy for case analysis MCQs. As of April 7, 2023, a total of 129 individuals participated in the experiment. However, 19 individuals withdrew from the experiment at various phases; thus, as of July 1, 2023, a total of 110 individuals accomplished the trial and completed all follow-up work. After we intervened in the learning style of the students in the short term, the ChatGPT group answered more questions correctly than the control group (ChatGPT group: mean 141.20, SD 26.68; control group: mean 130.80, SD 25.56; P=.04) in the orthopedics test, particularly on A1 (ChatGPT group: mean 46.57, SD 8.52; control group: mean 42.18, SD 9.43; P=.01), A2 (ChatGPT group: mean 60.59, SD 10.58; control group: mean 56.66, SD 9.91; P=.047), and A3/4 MCQs (ChatGPT group: mean 19.57, SD 5.48; control group: mean 16.46, SD 4.58; P=.002). At the end of the semester, we found that the ChatGPT group performed better on final examinations in surgery (ChatGPT group: mean 76.54, SD 9.79; control group: mean 72.54, SD 8.11; P=.02) and obstetrics and gynecology (ChatGPT group: mean 75.98, SD 8.94; control group: mean 72.54, SD 8.66; P=.04) than the control group. Conclusions: ChatGPT answers orthopedics-related MCQs accurately, and students using it excel in both short-term and long-term assessments. Our findings strongly support ChatGPT's integration into medical education, enhancing contemporary instructional methods. Trial registration: Chinese Clinical Trial Registry Chictr2300071774; https://www.chictr.org.cn/hvshowproject.html ?id=225740&v=1.0.
ChatGPT Generated Otorhinolaryngology Multiple-Choice Questions: Quality, Psychometric Properties, and Suitability for Assessments

Cecilia Lotto,Sean C Sheppard,Wilma Anschuetz,Daniel Stricker,Giulia Molinari,Sören Huwendiek,Lukas Anschuetz

DOI: https://doi.org/10.1002/oto2.70018

2024-09-26

OTO Open

Abstract:Objective: To explore Chat Generative Pretrained Transformer's (ChatGPT's) capability to create multiple-choice questions about otorhinolaryngology (ORL). Study design: Experimental question generation and exam simulation. Setting: Tertiary academic center. Methods: ChatGPT 3.5 was prompted: "Can you please create a challenging 20-question multiple-choice questionnaire about clinical cases in otolaryngology, offering five answer options?." The generated questionnaire was sent to medical students, residents, and consultants. Questions were investigated regarding quality criteria. Answers were anonymized and the resulting data was analyzed in terms of difficulty and internal consistency. Results: ChatGPT 3.5 generated 20 exam questions of which 1 question was considered off-topic, 3 questions had a false answer, and 3 questions had multiple correct answers. Subspecialty theme repartition was as follows: 5 questions were on otology, 5 about rhinology, and 10 questions addressed head and neck. The qualities of focus and relevance were good while the vignette and distractor qualities were low. The level of difficulty was suitable for undergraduate medical students (n = 24), but too easy for residents (n = 30) or consultants (n = 10) in ORL. Cronbach's α was highest (.69) with 15 selected questions using students' results. Conclusion: ChatGPT 3.5 is able to generate grammatically correct simple ORL multiple choice questions for a medical student level. However, the overall quality of the questions was average, needing thorough review and revision by a medical expert to ensure suitability in future exams.
Evaluation of ChatGPT as a diagnostic tool for medical learners and clinicians

Ali Hadi,Edward Tran,Branavan Nagarajan,Amrit Kirpalani

DOI: https://doi.org/10.1371/journal.pone.0307383

IF: 3.7

2024-08-01

PLoS ONE

Abstract:ChatGPT is a large language model (LLM) trained on over 400 billion words from books, articles, and websites. Its extensive training draws from a large database of information, making it valuable as a diagnostic aid. Moreover, its capacity to comprehend and generate human language allows medical trainees to interact with it, enhancing its appeal as an educational resource. This study aims to investigate ChatGPT's diagnostic accuracy and utility in medical education. 150 Medscape case challenges (September 2021 to January 2023) were inputted into ChatGPT. The primary outcome was the number (%) of cases for which the answer given was correct. Secondary outcomes included diagnostic accuracy, cognitive load, and quality of medical information. A qualitative content analysis was also conducted to assess its responses. ChatGPT answered 49% (74/150) cases correctly. It had an overall accuracy of 74%, a precision of 48.67%, sensitivity of 48.67%, specificity of 82.89%, and an AUC of 0.66. Most answers were considered low cognitive load 51% (77/150) and most answers were complete and relevant 52% (78/150). ChatGPT in its current form is not accurate as a diagnostic tool. ChatGPT does not necessarily give factual correctness, despite the vast amount of information it was trained on. Based on our qualitative analysis, ChatGPT struggles with the interpretation of laboratory values, imaging results, and may overlook key information relevant to the diagnosis. However, it still offers utility as an educational tool. ChatGPT was generally correct in ruling out a specific differential diagnosis and providing reasonable next diagnostic steps. Additionally, answers were easy to understand, showcasing a potential benefit in simplifying complex concepts for medical learners. Our results should guide future research into harnessing ChatGPT's potential educational benefits, such as simplifying medical concepts and offering guidance on differential diagnoses and next steps.

multidisciplinary sciences
Evaluation of ChatGPT's performance in Medical Education: A Comparative Analysis with Students in a Pulmonology Examination

Cherif,H.,Moussa,C.,Ben Rjab,S.,Mokaddem,S.,Dhahri,B.

DOI: https://doi.org/10.1183/13993003.congress-2024.pa4378

IF: 24.3

2024-11-01

European Respiratory Journal

Abstract:The rapid evolution of ChatGPT has raised concerns in the field of medical education. Objective: To evaluate the performance of ChatGPT in a pneumology examination. Methodology: We conducted a cross-sectional comparative study involving two distinct groups. The 1st group comprised 244 third-year medical students who had taken the pneumology examination in 2020. The 2nd group included two variants of ChatGPT-3.5: ChatGPT-V1 (lacking contextualization) and ChatGPT-V2 (enhanced with contextual information). The examination consisted of a total of 9 multiple-choice questions (MCQs), 13 short open-ended questions (SOEQs), and 7 clinical cases. ChatGPT's responses to each question were compared to those of the students and examined based on the difficulty index (DI). A DI below 0.4 qualifies a question as difficult, while a DI above 0.6 qualifies a question as easy. Results: ChatGPT-V1 demonstrated remarkable proficiency in radiology, microbiology, and thoracic surgery, surpassing the majority of medical students in these areas. However, it encountered challenges in anatomopathology, pharmacology, and clinical pneumology. Conversely, ChatGPT-V2 consistently provided more accurate responses across different question categories. ChatGPT exhibited suboptimal performance in MCQs compared to students. ChatGPT-V2 excelled in responding to SOEQs. Both versions, particularly ChatGPT-V2, outperformed students in handling questions with low and moderate DI. Additionally, students demonstrated increased proficiency when faced with highly DI questions. ChatGPT-V1 did not pass the examination; however, ChatGPT-V2 successfully passed, surpassing 62.1% of human candidates. Conclusion:Despite ChatGPT's access to online data, its performance closely mirrors that of an average-level medical student.

respiratory system

LOCALIZATION OF RADIOACTIVE ELEMENTS : (A Review).

Can ChatGPT help patients answer their otolaryngology questions?

Assessing ChatGPT's Responses to Otolaryngology Patient Questions

Is ChatGPT 3.5 smarter than Otolaryngology trainees? A comparison study of board style exam questions

Is ChatGPT smarter than Otolaryngology trainees? A comparison study of board style exam questions

Does ChatGPT Answer Otolaryngology Questions Accurately?

Effects of mycorrhizae on growth and demography of tallgrass prairie forbs.

ChatGPT's adherence to otolaryngology clinical practice guidelines

Assessing ChatGPT's potential as a clinical resource for medical oncologists: An evaluation with board-style questions and real-world patient cases.

The Accuracy of Artificial Intelligence ChatGPT in Oncology Examination Questions

Assessment of Artificial Intelligence Performance on the Otolaryngology Residency In-Service Exam

A cross-sectional comparative study: ChatGPT 3.5 versus diverse levels of medical experts in the diagnosis of ENT diseases

Evaluating ChatGPT Performance on the Orthopaedic In-Training Examination

Comparative Performance of ChatGPT 3.5 and GPT4 on Rhinology Standardized Board Examination Questions

Leprosy work--the Bethany experience.

Total-energy and pressure calculations for random substitutional alloys.

Inadequate Performance of ChatGPT on Orthopedic Board-Style Written Exams

Integrating ChatGPT in Orthopedic Education for Medical Undergraduates: Randomized Controlled Trial

ChatGPT Generated Otorhinolaryngology Multiple-Choice Questions: Quality, Psychometric Properties, and Suitability for Assessments

Evaluation of ChatGPT as a diagnostic tool for medical learners and clinicians

Evaluation of ChatGPT's performance in Medical Education: A Comparative Analysis with Students in a Pulmonology Examination