Abstract:Recent developments in networking and computing have enabled collaborative biomedical engineering research by geographically separated participants. One of the most promising goals is to use these technologies to extend human intellectual capabilities in medical decision making. These emerging technologies are poised to drastically reduce healthcare cost by providing service at remote locations. This also increases diagnosis capacity since information is made available to experts at any location. In this paper, we propose a novel application of a recently developed interactive and distributed system in medical consultation and education. Our approach builds on the notion that interactive and distributive capabilities of the system are crucial for medical consultation and education. The presented application uses a multiuser, collaborative environment with multimodal human/machine communication in the dimensions of sight, sound, and touch. The experimental setup, consisting of two user stations, and the multimodal interfaces, including sight (eye-tracking), sound (automatic speech), and touch (microbeam pen), were tested and evaluated. The system uses a collaborative workspace as a common visualization space. Users communicate with the application through a fusion agent by eye-tracking, speech, and microbeam pen. The audio/video teleconferencing is also included to help the radiologists to communicate with each other simultaneously while they are working on the mammograms. The system used in this study has three software agents: a fusion agent, a conversational agent, and an analytic agent. The fusion agent interprets multimodal commands by integrating the multimodal inputs. The conversational agent answers the user's questions and detects human-related or semantic errors and notifies the user about the results of the image analysis. The analytic agent enhances the digitized images using the wavelet denoising algorithm if requested by the user. To show how well the system performs in practice, we used the system for medical consultation on mammograms. Results also show that the relevant information about the region of interest (ROI) of the mammograms chosen by the users is extracted automatically and used to enhance the mammograms.

MCRDR Knowledge-Based 3D Dialogue Simulation in Clinical Training and Assessment

End-to-End Knowledge-Routed Relational Dialogue System for Automatic Diagnosis

Dialogue-Based Simulation For Cultural Awareness Training

Immersive training of clinical decision making with AI driven virtual patients - a new VR platform called medical tr.AI.ning

Dynamic Curriculum Learning with Co-training for Medical Dialogue Generation

Synthetic Patients: Simulating Difficult Conversations with Multimodal Generative AI for Medical Education

Communication Skills Training Using Remote Augmented Reality Medical Simulation: a Feasibility and Acceptability Qualitative Study

MedKP: Medical Dialogue with Knowledge Enhancement and Clinical Pathway Encoding

Blurring the boundaries: scenario-based simulation in a clinical setting.

Collaborative System Design of Mixed Reality Communication for Medical Training

Clinical Camel: An Open-Source Expert-Level Medical Language Model with Dialogue-Based Knowledge Encoding

Leveraging Large Language Model as Simulated Patients for Clinical Education

Synthetic Patient-Physician Dialogue Generation from Clinical Notes Using LLM

MedDiT: A Knowledge-Controlled Diffusion Transformer Framework for Dynamic Medical Image Generation in Virtual Simulated Patient

DialSim: A Real-Time Simulator for Evaluating Long-Term Multi-Party Dialogue Understanding of Conversational Agents

A system for medical consultation and education using multimodal human/machine communication

Designing Virtual Reality-Based Conversational Agents to Train Clinicians in Verbal De-escalation Skills: Exploratory Usability Study

Efficient Evidence-Based Dialogue System for Medical Diagnosis.

The Virtual Operative Assistant: An explainable artificial intelligence tool for simulation-based training in surgery and medicine

A Knowledge-Enhanced Hierarchical Reinforcement Learning-Based Dialogue System for Automatic Disease Diagnosis

Natural Language-Driven Dialogue Systems for Support in Physical Medicine and Rehabilitation