Abstract:Background: Adherence to evidence-based practice is indispensable in health care. Recently, the utility of generative artificial intelligence (AI) models in health care has been evaluated extensively. However, the lack of consensus guidelines on the design and reporting of findings of these studies poses a challenge for the interpretation and synthesis of evidence. Objective: This study aimed to develop a preliminary checklist to standardize the reporting of generative AI-based studies in health care education and practice. Methods: A literature review was conducted in Scopus, PubMed, and Google Scholar. Published records with "ChatGPT," "Bing," or "Bard" in the title were retrieved. Careful examination of the methodologies employed in the included records was conducted to identify the common pertinent themes and the possible gaps in reporting. A panel discussion was held to establish a unified and thorough checklist for the reporting of AI studies in health care. The finalized checklist was used to evaluate the included records by 2 independent raters. Cohen κ was used as the method to evaluate the interrater reliability. Results: The final data set that formed the basis for pertinent theme identification and analysis comprised a total of 34 records. The finalized checklist included 9 pertinent themes collectively referred to as METRICS (Model, Evaluation, Timing, Range/Randomization, Individual factors, Count, and Specificity of prompts and language). Their details are as follows: (1) Model used and its exact settings; (2) Evaluation approach for the generated content; (3) Timing of testing the model; (4) Transparency of the data source; (5) Range of tested topics; (6) Randomization of selecting the queries; (7) Individual factors in selecting the queries and interrater reliability; (8) Count of queries executed to test the model; and (9) Specificity of the prompts and language used. The overall mean METRICS score was 3.0 (SD 0.58). The tested METRICS score was acceptable, with the range of Cohen κ of 0.558 to 0.962 ( P <.001 for the 9 tested items). With classification per item, the highest average METRICS score was recorded for the "Model" item, followed by the "Specificity" item, while the lowest scores were recorded for the "Randomization" item (classified as suboptimal) and "Individual factors" item (classified as satisfactory). Conclusions: The METRICS checklist can facilitate the design of studies guiding researchers toward best practices in reporting results. The findings highlight the need for standardized reporting algorithms for generative AI-based studies in health care, considering the variability observed in methodologies and reporting. The proposed METRICS checklist could be a preliminary helpful base to establish a universally accepted approach to standardize the design and reporting of generative AI-based studies in health care, which is a swiftly evolving research topic.

Measuring the Masses: Mass-Gathering Medical Case Reporting, Conceptual Modeling – The DREAM Model (Paper 5)

An Emergency Medical Information System for Mass Gatherings: Design and Implementation

Use of an Agent-Based Simulation Model to Evaluate a Mobile-Based System for Supporting Emergency Evacuation Decision Making

Design and Development of A Mobile-based System for Supporting Emergency Triage Decision Making

Development of the science of mass casualty incident management: reflection on the medical response to the Wenchuan earthquake and Hangzhou bus fire

Quantitative Metrics in Mass-Gathering Studies: A Comprehensive Systematic Review

Mass-Gathering Medical Care Provided by a Collegiate-Based First Response Service at an Annual College Music Festival and Campus-Wide Celebration

Standards of Medical Planning and Response for Emergency Medical Teams During Mass Gatherings: A Systematic Review

Portal of medical data models: information infrastructure for medical research and healthcare

Mass gathering event medical preparedness and response: A review of Canadian legislation and guidelines

Assessing the Mental Model State of Emergency Responders in the Context of Radiological Dispersal Device (RDD) Incidents: A Multi-state Study

Utilizing AEMS (Assess-EOP-Map-Simulate) Algorithm to Assess and Spatially Link Prehospital Emergency Medical Services Resources in Road-Traffic Mass Casualty Incidents in Kumasi, Ghana

Medical encounters (including injury and illness) at mass community-based endurance sports events: an international consensus statement on definitions and methods of data recording and reporting

Medical Coordination Rescue Members' and Ambulance Nurses' Perspectives on a New Model for Mass Casualty and Disaster Management and a Novel Terror Attack Mitigation Approach in the Netherlands: A Qualitative Study

Understanding the drivers on medical workloads: an analysis of spectators at the Australian Football League

Rethinking Prehospital Response to Mass Casualty Events: Move, Treat, and Transport

Ambulance Transports from NCAA Division 1 Football Games

A toolbox of different approaches to analyze and present PRO-CTCAE data in oncology studies

Optimizing Concussion Care Seeking: Using Machine Learning to Predict Delayed Concussion Reporting

A Preliminary Checklist (METRICS) to Standardize the Design and Reporting of Studies on Generative Artificial Intelligence–Based Models in Health Care Education and Practice: Development Study Involving a Literature Review

Social mass gathering events influence emergency medical services call volume