Evaluation of a preoperative multimedia information program in surgical oncology.

S. Evrard,S. Mathoulin-Pélissier,C. Larrue,P. Lapouge,E. Bussiéres,C. Tunon de Lara

DOI: https://doi.org/10.1016/J.EJSO.2004.09.019

IF: 4.037

2005-02-01

European Journal of Surgical Oncology

Abstract:

What problem does this paper attempt to address?

Resident dashboards: helping your clinical competency committee visualize trainees’ key performance indicators

Karen A. Friedman,John Raimo,Kelly Spielmann,Saima Chaudhry

DOI: https://doi.org/10.3402/meo.v21.29838

IF: 6

2016-01-01

Medical Education Online

Abstract:INTRODUCTION: Under the Next Accreditation System, programs need to find ways to collect and assess meaningful reportable information on its residents to assist the program director regarding resident milestone progression. This paper discusses the process that one large Internal Medicine Residency Program used to provide both quantitative and qualitative data to its clinical competency committee (CCC) through the creation of a resident dashboard.METHODS: Program leadership at a large university-based program developed four new end of rotation evaluations based on the American Board of Internal Medicine (ABIM) and Accreditation Council of Graduated Medical Education's (ACGME) 22 reportable milestones. A resident dashboard was then created to pull together both milestone- and non-milestone-based quantitative data and qualitative data compiled from faculty, nurses, peers, staff, and patients.RESULTS: Dashboards were distributed to the members of the CCC in preparation for the semiannual CCC meeting. CCC members adjudicated quantitative and qualitative data to present their cohort of residents at the CCC meeting. Based on the committee's response, evaluation scores remained the same or were adjusted. Final milestone scores were then entered into the accreditation data system (ADS) on the ACGME website.CONCLUSIONS: The process of resident assessment is complex and should comprise both quantitative and qualitative data. The dashboard is a valuable tool for program leadership to use both when evaluating house staff on a semiannual basis at the CCC and to the resident in person.

education & educational research
Developing a dashboard to meet Competence Committee needs: a design-based research project

Brent Thoma,Venkat Bandi,Robert Carey,Debajyoti Mondal,Rob Woods,Lynsey Martin,Teresa Chan,Robert Woods

DOI: https://doi.org/10.36834/cmej.68903

2020-01-07

Canadian Medical Education Journal

Abstract:Background: Competency-based programs are being adopted in medical education around the world. Competence Committees must visualize learner assessment data effectively to support their decision-making. Dashboards play an integral role in decision support systems in other fields. Design-based research allows the simultaneous development and study of educational environments. Methods: We utilized a design-based research process within the emergency medicine residency program at the University of Saskatchewan to identify the data, analytics, and visualizations needed by its Competence Committee, and developed a dashboard incorporating these elements. Narrative data were collected from two focus groups, five interviews, and the observation of two Competence Committee meetings. Data were qualitatively analyzed to develop a thematic framework outlining the needs of the Competence Committee and to inform the development of the dashboard. Results: The qualitative analysis identified four Competence Committee needs (Explore Workplace-Based Assessment Data, Explore Other Assessment Data, Understand the Data in Context, and Ensure the Security of the Data). These needs were described with narratives and represented through visualizations of the dashboard elements. Conclusions: This work addresses the practical challenges of supporting data-driven decision making by Competence Committees and will inform the development of dashboards for programs, institutions, and learner management systems.
Design, Usability, and Acceptability of a Needs-Based, Automated Dashboard to Provide Individualized Patient-Care Data to Pediatric Residents

Julia K W Yarahuan,Huay-Ying Lo,Lanessa Bass,Jeff Wright,Lauren M Hess

DOI: https://doi.org/10.1055/s-0042-1744388

Abstract:Background and objectives: Pediatric residency programs are required by the Accreditation Council for Graduate Medical Education to provide residents with patient-care and quality metrics to facilitate self-identification of knowledge gaps to prioritize improvement efforts. Trainees are interested in receiving this data, but this is a largely unmet need. Our objectives were to (1) design and implement an automated dashboard providing individualized data to residents, and (2) examine the usability and acceptability of the dashboard among pediatric residents. Methods: We developed a dashboard containing individualized patient-care data for pediatric residents with emphasis on needs identified by residents and residency leadership. To build the dashboard, we created a connection from a clinical data warehouse to data visualization software. We allocated patients to residents based on note authorship and created individualized reports with masked identities that preserved anonymity. After development, we conducted usability and acceptability testing with 11 resident users utilizing a mixed-methods approach. We conducted interviews and anonymous surveys which evaluated technical features of the application, ease of use, as well as users' attitudes toward using the dashboard. Categories and subcategories from usability interviews were identified using a content analysis approach. Results: Our dashboard provides individualized metrics including diagnosis exposure counts, procedure counts, efficiency metrics, and quality metrics. In content analysis of the usability testing interviews, the most frequently mentioned use of the dashboard was to aid a resident's self-directed learning. Residents had few concerns about the dashboard overall. Surveyed residents found the dashboard easy to use and expressed intention to use the dashboard in the future. Conclusion: Automated dashboards may be a solution to the current challenge of providing trainees with individualized patient-care data. Our usability testing revealed that residents found our dashboard to be useful and that they intended to use this tool to facilitate development of self-directed learning plans.
Legitimizing Continuous Quality Improvement (CQI): Navigating Rationality in Undergraduate Medical Education

Terry D. Stratton

DOI: https://doi.org/10.1007/s11606-019-04875-1

IF: 5.7

2019-02-20

Journal of General Internal Medicine

Abstract:In July of 2015, the Liaison Committee on Medical Education (LCME)—the primary accrediting body for North American allopathic medical schools—formally advanced a model of “formative accreditation” by requiring that medical schools engage in “ongoing planning and continuous quality improvement processes that establish short and long-term programmatic goals, result in the achievement of measurable outcomes that are used to improve programmatic quality, and ensure effective monitoring of the medical education program’s compliance with accreditation standards.”As these and parallel forces redefine undergraduate medical education (UME) in increasingly rationalistic (i.e., operational, measureable, controllable) terms, efforts to implement meaningful continuous quality improvement (CQI) processes may be challenged to overcome perceptions of questionable purpose, worth, and impact often associated with administration mandates. This commentary discusses potential factors underlying the growing rationalism in UME and offers practical strategies to shield CQI from being passively dismissed, excessively routinized, or redirected toward other institutional ends—remaining, instead, purposefully focused on the task at hand: Enhancing teaching and learning in undergraduate medical curricula.

medicine, general & internal,health care sciences & services
A Framework for Evaluating Dashboards in Healthcare

Mengdie Zhuang,David Concannon,Ed Manley

DOI: https://doi.org/10.1109/tvcg.2022.3147154

IF: 5.2

2022-04-01

IEEE Transactions on Visualization and Computer Graphics

Abstract:In the era of 'information overload', effective information provision is essential for enabling rapid response and critical decision making. In making sense of diverse information sources, dashboards have become an indispensable tool, providing fast, effective, adaptable, and personalized access to information for professionals and the general public alike. However, these objectives place heavy requirements on dashboards as information systems in usability and effective design. Understanding these issues is challenging given the absence of consistent and comprehensive approaches to dashboard evaluation. In this article we systematically review literature on dashboard implementation in healthcare, where dashboards have been employed widely, and where there is widespread interest for improving the current state of the art, and subsequently analyse approaches taken towards evaluation. We draw upon consolidated dashboard literature and our own observations to introduce a general definition of dashboards which is more relevant to current trends, together with seven evaluation scenarios - task performance, behaviour change, interaction workflow, perceived engagement, potential utility, algorithm performance and system implementation. These scenarios distinguish different evaluation purposes which we illustrate through measurements, example studies, and common challenges in evaluation study design. We provide a breakdown of each evaluation scenario, and highlight some of the more subtle questions. We demonstrate the use of the proposed framework by a design study guided by this framework. We conclude by comparing this framework with existing literature, outlining a number of active discussion points and a set of dashboard evaluation best practices for the academic, clinical and software development communities alike.

computer science, software engineering
Improving the Efficiency and Ease of Healthcare Analysis Through Use of Data Visualization Dashboards

Jennifer G. Stadler,Kipp Donlon,Jordan D. Siewert,Tessa Franken,Nathaniel E. Lewis

DOI: https://doi.org/10.1089/big.2015.0059

IF: 4.426

2016-06-01

Big Data

Abstract:The digitization of a patient's health record has profoundly impacted medicine and healthcare. The compilation and accessibility of medical history has provided clinicians an unprecedented, holistic account of a patient's conditions, procedures, medications, family history, and social situation. In addition to the bedside benefits, this level of information has opened the door for population-level monitoring and research, the results of which can be used to guide initiatives that are aimed at improving quality of care. Cerner Corporation partners with health systems to help guide population management and quality improvement projects. With such an enormous and diverse client base-varying in geography, size, organizational structure, and analytic needs-discerning meaning in the data and how they fit with that particular hospital's goals is a slow, difficult task that requires clinical, statistical, and technical literacy. This article describes the development of dashboards for efficient data visualization at the healthcare facility level. Focusing on two areas with broad clinical importance, sepsis patient outcomes and 30-day hospital readmissions, dashboards were developed with the goal of aggregating data and providing meaningful summary statistics, highlighting critical performance metrics, and providing easily digestible visuals that can be understood by a wide range of personnel with varying levels of skill and areas of expertise. These internal-use dashboards have allowed associates in multiple roles to perform a quick and thorough assessment on a hospital of interest by providing the data to answer necessary questions and to identify important trends or opportunities. This automation of a previously manual process has greatly increased efficiency, saving hours of work time per hospital analyzed. Additionally, the dashboards have standardized the analysis process, ensuring use of the same metrics and processes so that overall themes can be compared across hospitals and health systems.

computer science, theory & methods, interdisciplinary applications
Application of continuous quality improvement to medical education

Brian M. Wong,Linda A. Headrick

DOI: https://doi.org/10.1111/medu.14351

2020-09-27

Medical Education

Abstract:<section class="article-section__content"><h3 class="article-section__sub-title section1"> Context</h3><p>The explicit, intentional and systematic application of continuous quality improvement (QI) in medical education practice and research can improve medical education and help it achieve its goals. Quality improvement and medical education share a foundation centered on <i>learning</i> – experiencing, reflecting, thinking and acting in continuous cycles that spiral to sustained advancement. This suggests that a QI mindset can be brought to bear on various aspects of medical education research and practice. </p></section><section class="article-section__content"><h3 class="article-section__sub-title section1"> Discussion</h3><p>To explore this possibility, we turn to W. Edwards Deming's System of Profound Knowledge, widely regarded as one of the foundational frameworks in quality improvement, where he argues strongly that there are four highly interrelated elements that are required for improvement: Appreciation of a System, Theory of Knowledge, Knowledge about Variation, Knowledge of Psychology. In this article, we define and explore each of the four domains and their application in medical education, highlighting both opportunities and challenges.</p></section><section class="article-section__content"><h3 class="article-section__sub-title section1"> Conclusion</h3><p>Medical educators who utilize QI in their educational practices can help create learning environments that imprint positively on learners and contribute to better outcomes in their clinical learning environments. We provide recommendations for how educators' informed use of QI can improve medical education and help it achieve its ultimate goal of improved health and health care.</p></section>

education, scientific disciplines,health care sciences & services
The Impact of Visualization Dashboards on Quality of Care and Clinician Satisfaction: Integrative Literature Review

Saif Sherif Khairat,Aniesha Dukkipati,Heather Alico Lauria,Thomas Bice,Debbie Travers,Shannon S Carson

DOI: https://doi.org/10.2196/humanfactors.9328

2018-05-31

JMIR Human Factors

Abstract:BACKGROUND: Intensive Care Units (ICUs) in the United States admit more than 5.7 million people each year. The ICU level of care helps people with life-threatening illness or injuries and involves close, constant attention by a team of specially-trained health care providers. Delay between condition onset and implementation of necessary interventions can dramatically impact the prognosis of patients with life-threatening diagnoses. Evidence supports a connection between information overload and medical errors. A tool that improves display and retrieval of key clinical information has great potential to benefit patient outcomes. The purpose of this review is to synthesize research on the use of visualization dashboards in health care.OBJECTIVE: The purpose of conducting this literature review is to synthesize previous research on the use of dashboards visualizing electronic health record information for health care providers. A review of the existing literature on this subject can be used to identify gaps in prior research and to inform further research efforts on this topic. Ultimately, this evidence can be used to guide the development, testing, and implementation of a new solution to optimize the visualization of clinical information, reduce clinician cognitive overload, and improve patient outcomes.METHODS: Articles were included if they addressed the development, testing, implementation, or use of a visualization dashboard solution in a health care setting. An initial search was conducted of literature on dashboards only in the intensive care unit setting, but there were not many articles found that met the inclusion criteria. A secondary follow-up search was conducted to broaden the results to any health care setting. The initial and follow-up searches returned a total of 17 articles that were analyzed for this literature review.RESULTS: Visualization dashboard solutions decrease time spent on data gathering, difficulty of data gathering process, cognitive load, time to task completion, errors, and improve situation awareness, compliance with evidence-based safety guidelines, usability, and navigation.CONCLUSIONS: Researchers can build on the findings, strengths, and limitations of the work identified in this literature review to bolster development, testing, and implementation of novel visualization dashboard solutions. Due to the relatively few studies conducted in this area, there is plenty of room for researchers to test their solutions and add significantly to the field of knowledge on this subject.
Precipitation of fluorouracil in elastomeric infusers with a polyisoprene reservoir and in polypropylene syringes with an elastomeric joint.

V. Corbrion,S. Crauste-Manciet,P. Allain,D. Brossard

DOI: https://doi.org/10.1093/AJHP/54.16.1845

1997-08-15

American Journal of Health-System Pharmacy

Abstract:
Evaluation of continuous quality improvement in accreditation for medical education

Nesibe Akdemir,Linda N. Peterson,Craig M. Campbell,Fedde Scheele

DOI: https://doi.org/10.1186/s12909-020-02124-2

IF: 3.263

2020-09-01

BMC Medical Education

Abstract:Abstract Background Accreditation systems are based on a number of principles and purposes that vary across jurisdictions. Decision making about accreditation governance suffers from a paucity of evidence. This paper evaluates the pros and cons of continuous quality improvement (CQI) within educational institutions that have traditionally been accredited based on episodic evaluation by external reviewers. Methods A naturalistic utility-focused evaluation was performed. Seven criteria, each relevant to government oversight, were used to evaluate the pros and cons of the use of CQI in three medical school accreditation systems across the continuum of medical education. The authors, all involved in the governance of accreditation, iteratively discussed CQI in their medical education contexts in light of the seven criteria until consensus was reached about general patterns. Results Because institutional CQI makes use of early warning systems, it may enhance the reflective function of accreditation. In the three medical accreditation systems examined, external accreditors lacked the ability to respond quickly to local events or societal developments. There is a potential role for CQI in safeguarding the public interest. Moreover, the central governance structure of accreditation may benefit from decentralized CQI. However, CQI has weaknesses with respect to impartiality, independence, and public accountability, as well as with the ability to balance expectations with capacity. Conclusion CQI, as evaluated with the seven criteria of oversight, has pros and cons. Its use still depends on the balance between the expected positive effects—especially increased reflection and faster response to important issues—versus the potential impediments. A toxic culture that affects impartiality and independence, as well as the need to invest in bureaucratic systems may make in impractical for some institutions to undertake CQI.

education & educational research,education, scientific disciplines
Dashboarding to Monitor Machine-Learning-Based Clinical Decision Support Interventions

Daniel J Hekman,Hanna J Barton,Apoorva P Maru,Graham Wills,Amy L Cochran,Corey Fritsch,Douglas A Wiegmann,Frank Liao,Brian W Patterson

DOI: https://doi.org/10.1055/a-2219-5175

Abstract:Background: Existing monitoring of machine-learning-based clinical decision support (ML-CDS) is focused predominantly on the ML outputs and accuracy thereof. Improving patient care requires not only accurate algorithms but also systems of care that enable the output of these algorithms to drive specific actions by care teams, necessitating expanding their monitoring. Objectives: In this case report, we describe the creation of a dashboard that allows the intervention development team and operational stakeholders to govern and identify potential issues that may require corrective action by bridging the monitoring gap between model outputs and patient outcomes. Methods: We used an iterative development process to build a dashboard to monitor the performance of our intervention in the broader context of the care system. Results: Our investigation of best practices elsewhere, iterative design, and expert consultation led us to anchor our dashboard on alluvial charts and control charts. Both the development process and the dashboard itself illuminated areas to improve the broader intervention. Conclusion: We propose that monitoring ML-CDS algorithms with regular dashboards that allow both a context-level view of the system and a drilled down view of specific components is a critical part of implementing these algorithms to ensure that these tools function appropriately within the broader care system.
Use of Visual Dashboards to Enhance Pharmacy Teaching

Andrew Bartlett,Carl R. Schneider,Jonathan Penm,Ardalan Mirzaei

DOI: https://doi.org/10.3390/pharmacy9020093

2021-04-23

Pharmacy

Abstract:Teaching large cohorts of pharmacy students with a team of multiple tutors in a feedback intensive course poses challenges in relation the amount of data generated, data integrity, interpretation of the data and importantly application of the insights gained from the data. The dispensing and counselling course in the third year BPharm at the University of Sydney has implemented the USyd Pharmacy Dashboard, developed to address these challenges following the Technological Pedagogical Content Knowledge Framework (TPACK) to integrate technology into teaching. The dashboard was designed to improve the student experience through more consistent feedback, gain insights to improve teaching delivery and provide efficiencies in maintaining data integrity. The tool has been developed using an action-based research approach whereby ideas are put into practice as the means to further develop the idea and improve practice. Refinement of the USyd Pharmacy Dashboard over three years has shown improvements in teaching delivery as teachers can respond to emerging trends. Student performance and satisfaction scores have increased, mainly due to improved consistency between tutors and improved delivery of feedback. Time involved with administrative tasks such as data maintenance is reduced. Opportunities for further refinements such as real time benchmarking and developing an open learner model have become apparent.
Fostering Student-Faculty Partnerships for Continuous Curricular Improvement in Undergraduate Medical Education

Kirstin W Scott,Dana G Callahan,Jie Jane Chen,Marissa H Lynn,David J Cote,Anna Morenz,Josephine Fisher,Varnel L Antoine,Elizabeth R Lemoine,Shaunak K Bakshi,Jessie Stuart,Edward M Hundert,Bernard S Chang,Holly Gooding

DOI: https://doi.org/10.1097/ACM.0000000000002726

Abstract:Problem: A number of medical schools have used curricular reform as an opportunity to formalize student involvement in medical education, but there are few published assessments of these programs. Formal evaluation of a program's acceptability and use is essential for determining its potential for sustainability and generalizability. Approach: Harvard Medical School's Education Representatives (Ed Reps) program was created in 2015 to launch alongside a new curriculum. The program aimed to foster partnerships between faculty and students for continuous and real-time curricular improvement. Ed Reps, course directors, and core faculty met regularly to convey bidirectional feedback to optimize the learning environment in real time. Outcomes: A survey to assess the program's impact was sent to students and faculty. The majority of students (202/222; 91.0%) reported Ed Reps had a positive impact on the curriculum. Among faculty, 35/37 (94.6%) reported making changes to their courses as a result of Ed Reps feedback, and 34/37 (91.9%) agreed the program had a positive impact on the learning environment. Qualitative feedback from students and faculty demonstrated a change in school culture, reflecting the primary goals of partnership and continuous quality improvement (CQI). Next steps: This student-faculty partnership demonstrated high rates of awareness, use, and satisfaction among faculty and students, suggesting its potential for local sustainability and implementation at other schools seeking to formalize student engagement in CQI. Next steps include ensuring the feedback provided is representative of the student body and identifying new areas for student CQI input as the curriculum becomes more established.
Electronic Health Record User Dashboard for Optimization of Surgical Resident Procedural Reporting

Parker T Evans,Scott Nelson,Adam Wright,Chetan Aher

DOI: https://doi.org/10.1055/a-2444-0342

2024-10-17

Abstract:Background While necessary and educationally beneficial, administrative tasks such as case and patient tracking may carry additional burden for surgical trainees. Automated systems targeting these tasks are scarce, leading to manual and inefficient workflows. Methods We created an electronic health record (EHR)-based, user-specific dashboard for surgical residents to compile resident-specific data: bedside procedures performed, operative cases performed or participated in, and notes written by the resident as a surrogate for patients cared for. Usability testing was performed with resident volunteers, and residents were also surveyed post-implementation to assess for efficacy and satisfaction. Access log data from the EHR was used to assess dashboard usage over time. Descriptive statistics were calculated. Results The dashboard was implemented on a population of approximately 175 surgical residents in 5 different departments (General Surgery, Obstetrics and Gynecology, Neurosurgery, Orthopedics, and Otolaryngology) at a single academic medical center. 6 resident volunteers participating in usability testing completed an average of 96% of preset tasks independently. Average responses to five questions extracted from the System Usability Scale (SUS) questions ranged from 4.0 to 4.67 out of 5. Post-implementation surveys indicated high resident satisfaction (4.39 out of 5) and moderate rates of use, with 46.4% of residents using the dashboard at least monthly. Daily use of the dashboard has increased over time, especially after making the dashboard a default for surgical residents. Conclusion An EHR-based dashboard compiling resident-specific data can improve the efficiency of administrative tasks and supplement longitudinal education.
Re-thinking How We Teach Quality Improvement: Adding Meaning to an ACGME Requirement

Katlyn Baxter,Chelsey Petz,Jennifer L. Middleton,Miriam Chan

DOI: https://doi.org/10.1007/s11606-019-05059-7

IF: 5.7

2019-06-13

Journal of General Internal Medicine

Abstract:BackgroundMany residency programs have struggled to meaningfully meet the ACGME quality improvement (QI) requirements. Similarly, our residents were receiving limited QI education, and their longitudinal projects were ineffective.AimCreate an integrated didactic and experiential learning environment that equips residents to become leaders of QI.Setting and ParticipantsInternal medicine (IM) residency program of 45 residents in a large community hospital.Program DescriptionThis curriculum included eight content areas. Games, real-life application, and project celebrations cultivated engagement. Sessions occurred during residents' 2-week outpatient rotations. Project development was standardized.Program EvaluationThe QI Knowledge Application Tool-Revised (QIKAT-R) and separate surveys were used before and after the curriculum's implementation to evaluate resident QI knowledge and confidence, respectively. We also tracked QI scholarship and faculty engagement. Mean QIKAT-R scores improved significantly from 7.0 (SD 2.9) at baseline to 16.6 (SD 4.7) post-curriculum (n = 37 pairs, p = 0.043). Residents' adverse event reporting increased from 44% (19/43) at baseline to 90% (28/31) post-curriculum. Seven presentations were accepted for local, regional, and national conferences, compared with one presentation the preceding year.DiscussionA QI curriculum can be successfully integrated in a "4 + 2" program.

medicine, general & internal,health care sciences & services
Drug metabolizing capacity in vitro and in vivo--II. Correlations between hepatic microsomal monooxygenase markers in phenobarbital-induced rats.

D. Matthew,J. Houston

DOI: https://doi.org/10.1016/0006-2952(90)90311-8

IF: 6.1

1990-08-15

Biochemical Pharmacology

Abstract:
Design and Evaluation of a Performance Dashboard for the Faculty of Allied Medical Sciences, AJA University of Medical Sciences: A Protocol for a Mixed Methods Study

Sohrab Almasi,Nahid Mehrabi,Mahdi Ghorbani

DOI: https://doi.org/10.5812/semj-137592

2023-09-30

Shiraz E-Medical Journal

Abstract:Background: Faculties, as educational systems, comprise various educational groups, faculty members, researchers, students, and administrative staff. The management of data records related to the performance and activities of the faculty and its members leads to better monitoring, identification of weaknesses and strengths, and, ultimately, promotion of the faculty's performance. Dashboards are data management tools that can be used for monitoring and evaluating a faculty's performance. Objectives: This study aims to develop a protocol for the design of a faculty performance dashboard with a sequential mixed methods approach. Methods: This cross-sectional study will be conducted in the Faculty of Allied Medical Sciences, AJA University of Medical Sciences, in 2023. A mixed methods study with a sequential mixed (qualitative and then quantitative) design will be conducted in four phases. First, all the resources related to the functional dashboard are reviewed to identify its operational requirements. Second, the requirements and necessities of the software are determined by qualitative (interviews) and then quantitative (Delphi) methods. In this phase, 8 people will be interviewed during the qualitative phase, and thematic analysis will be used to analyze the data. For the quantitative step, the 2-round Delphi technique will be conducted by the purposive selection of 21 individuals. Data analysis for the quantitative step will be conducted in SPSS v. 22 by using descriptive statistics, including mode, median, mean, and percentage of agreement. Third, software coding will be performed in C# programming language in Visual Studio. Finally, 15 people among faculty members and managers will be selected by using purposive sampling to evaluate the software. In this phase, the qualitative method and then the quantitative method are used for software evaluation. In the qualitative method, the think-aloud protocol will be used to evaluate usability, and in the quantitative method, the users' satisfaction with the dashboard software will be assessed using a questionnaire. The validity and reliability of the questionnaire have been confirmed previously (Cronbach's alpha: 0.94). The data will be analyzed using descriptive statistics in SPSS v. 21. Results: The paper identifies four steps that should be followed when designing and adopting performance dashboards to support student agency and empowerment. Conclusions: The final product of this study is a dashboard for monitoring, evaluating performance, and managing resources at the faculty level.
Preliminary Development of a Program ABSITE Dashboard (PAD) to Guide Curriculum Innovation

Brian Carter,Jason Sidrak,Brandie Wagner,Claire Travis,Mark Nehler,Nicole Christian

DOI: https://doi.org/10.1016/j.jsurg.2023.10.014

IF: 3.524

2024-01-10

Journal of Surgical Education

Abstract:Purpose Medical Knowledge for general surgery residents' is assessed by the American Board of Surgery In- Training Examination (ABSITE). ASBITE score reports contain many metrics residency directors can utilize to assess resident progress and perform program evaluation. The purpose of this study was to develop a framework to evaluate program effectiveness in teaching specific subtest and subtopic areas of the ABSITE, using ABSITE score reports as an indicator. The aim is to demonstrate the identification of topic areas of weakness in program-wide performance on the ABSITE to guide proposed modification of the general surgery residency program curriculum, and to initiate development of a data visualizing dashboard to communicate these metrics. Methods A single institution retrospective study was performed utilizing ABSITE score reports from general surgery residents at a large academic training program from 2017 to 2020. ABSITE performance metrics from 320 unique records were entered into a database; statistical analysis for linear trends and variance were conducted for standard scores, subtest standard scores, and incorrect subtest topics. Deviation from national average scores were calculated by subtracting the national average score from each subtest score for each trainee. Data were displayed as medians or proportions and are displayed to optimize visualization as a proof-of-concept for the development of a program dashboard. Results Trends and variance in general surgery program and cohort performance on various elements of the ABSITE were visualized using figures and tables that represent a prototype for a program dashboard. Figure A1 demonstrates one example, in which a heatmap displays the median deviation from national average scores for each subtest by program year. Boxplots show the distribution of the deviation from national average, range for national average scores, and the recorded scores for each subtest by program year. Trends in median deviation from the national average scores are displayed for each program year paneled by subtest or for each exam year paneled by cohort. Median change in overall test scores from one program year to another in a cohort is visualized as a table. Bar graphs show the most often missed topics across all program years and heatmaps were generated showing the proportion of times each topic was missed for each subtest and exam year. Conclusions We demonstrate use of ABSITE reports to identify specific thematic areas of opportunities for curriculum modification and innovation as an element of program evaluation. In this study we demonstrate, through data analysis and visualization, feasibility for the creation of a Program ABSITE Dashboard (PAD) that enhances the use of ABSITE reports for formative program evaluation and can guide modifications to surgery program curriculum and educational practices.

surgery,education, scientific disciplines
A Quality Education: A Comprehensive Review of a Combined Longitudinal and Specialty Track Quality Improvement and Patient Safety Medical School Curriculum

Tyler J Murphy,Brittany N Saldivar,Carolyn K Holland,Michele N Lossius

DOI: https://doi.org/10.1097/01.JMQ.0000735524.78789.4a

Abstract:Structured quality improvement and patient safety (QI/PS) education has increased at every level of medical education; however, great variability exists in the content taught. Here, the authors present a longitudinal model for medical student QI/PS education that is currently implemented at the University of Florida College of Medicine. The curriculum is taught with a variety of teaching methods incorporated into each year with increasing levels of clinical implementation. This curriculum is multimodal and introduces students to QI/PS concepts, presents mock scenarios, and eventually encourages clinical application to situations students experience during their own clinical practice. Additionally, a specialized track for students to have further immersion into this field of medicine is described, which involves specialized training, expanded educational opportunities, and a capstone project. Both the curriculum and specialized track contain explicit clinical integration to ensure students are prepared to enter the medical profession to engage in QI/PS endeavors.
Using Natural Language Processing to Visualize Narrative Feedback in a Medical Student Performance Dashboard

Christina Maimone,Brigid M Dolan,Marianne M Green,Sandra M Sanguino,Celia Laird O'Brien

DOI: https://doi.org/10.1097/ACM.0000000000005800

2024-07-03

Abstract:Problem: Clinical competency committees rely on narrative feedback for important insight into learner performance, but reviewing comments can be time-consuming. Techniques such as natural language processing (NLP) could create efficiencies in narrative feedback review. In this study, the authors explored whether using NLP to create a visual dashboard of narrative feedback to preclerkship medical students would improve the competency review efficiency. Approach: Preclerkship competency review data collected at the Northwestern University Feinberg School of Medicine from 2014 to 2021 were used to identify relevant features of narrative data associated with review outcome (ready or not ready) and draft visual summary reports of the findings. A user needs analysis was held with experienced reviewers to better understand work processes in December 2019. Dashboards were designed based on this input to help reviewers efficiently navigate large amounts of narrative data. The dashboards displayed the model's prediction of the review outcome along with visualizations of how narratives in a student's portfolio compared with previous students' narratives. Excerpts of the most relevant comments were also provided. Six faculty reviewers who comprised the competency committee in spring 2023 were surveyed on the dashboard's utility. Outcomes: Reviewers found the predictive component of the dashboard most useful. Only 1 of 6 reviewers (17%) agreed that the dashboard improved process efficiency. However, 3 (50%) thought the visuals made them more confident in decisions about competence, and 3 (50%) thought they would use the visual summaries for future reviews. The outcomes highlight limitations of visualizing and summarizing narrative feedback in a comprehensive assessment system. Next steps: Future work will explore how to optimize the dashboards to meet reviewer needs. Ongoing advancements in large language models may facilitate these efforts. Opportunities to collaborate with other institutions to apply the model to an external context will also be sought.

Evaluation of a preoperative multimedia information program in surgical oncology.

Resident dashboards: helping your clinical competency committee visualize trainees’ key performance indicators

Developing a dashboard to meet Competence Committee needs: a design-based research project

Design, Usability, and Acceptability of a Needs-Based, Automated Dashboard to Provide Individualized Patient-Care Data to Pediatric Residents

Legitimizing Continuous Quality Improvement (CQI): Navigating Rationality in Undergraduate Medical Education

A Framework for Evaluating Dashboards in Healthcare

Improving the Efficiency and Ease of Healthcare Analysis Through Use of Data Visualization Dashboards

Application of continuous quality improvement to medical education

The Impact of Visualization Dashboards on Quality of Care and Clinician Satisfaction: Integrative Literature Review

Precipitation of fluorouracil in elastomeric infusers with a polyisoprene reservoir and in polypropylene syringes with an elastomeric joint.

Evaluation of continuous quality improvement in accreditation for medical education

Dashboarding to Monitor Machine-Learning-Based Clinical Decision Support Interventions

Use of Visual Dashboards to Enhance Pharmacy Teaching

Fostering Student-Faculty Partnerships for Continuous Curricular Improvement in Undergraduate Medical Education

Electronic Health Record User Dashboard for Optimization of Surgical Resident Procedural Reporting

Re-thinking How We Teach Quality Improvement: Adding Meaning to an ACGME Requirement

Drug metabolizing capacity in vitro and in vivo--II. Correlations between hepatic microsomal monooxygenase markers in phenobarbital-induced rats.

Design and Evaluation of a Performance Dashboard for the Faculty of Allied Medical Sciences, AJA University of Medical Sciences: A Protocol for a Mixed Methods Study

Preliminary Development of a Program ABSITE Dashboard (PAD) to Guide Curriculum Innovation

A Quality Education: A Comprehensive Review of a Combined Longitudinal and Specialty Track Quality Improvement and Patient Safety Medical School Curriculum

Using Natural Language Processing to Visualize Narrative Feedback in a Medical Student Performance Dashboard