Abstract:Experts have a remarkable capability of locating, perceptually organizing, identifying, and categorizing objects in images specific to their domains of expertise. In this article, we present a hierarchical probabilistic framework to discover the stereotypical and idiosyncratic viewing behaviors exhibited with expertise-specific groups. Through these patterned eye movement behaviors we are able to elicit the domain-specific knowledge and perceptual skills from the subjects whose eye movements are recorded during diagnostic reasoning processes on medical images. Analyzing experts' eye movement patterns provides us insight into cognitive strategies exploited to solve complex perceptual reasoning tasks. An experiment was conducted to collect both eye movement and verbal narrative data from three groups of subjects with different levels or no medical training (eleven board-certified dermatologists, four dermatologists in training and thirteen undergraduates) while they were examining and describing 50 photographic dermatological images. We use a hidden Markov model to describe each subject's eye movement sequence combined with hierarchical stochastic processes to capture and differentiate the discovered eye movement patterns shared by multiple subjects within and among the three groups. Independent experts' annotations of diagnostic conceptual units of thought in the transcribed verbal narratives are time-aligned with discovered eye movement patterns to help interpret the patterns' meanings. By mapping eye movement patterns to thought units, we uncover the relationships between visual and linguistic elements of their reasoning and perceptual processes, and show the manner in which these subjects varied their behaviors while parsing the images. We also show that inferred eye movement patterns characterize groups of similar temporal and spatial properties, and specify a subset of distinctive eye movement patterns which are commonly exhibited across multiple images. Based on the combinations of the occurrences of these eye movement patterns, we are able to categorize the images from the perspective of experts' viewing strategies in a novel way. In each category, images share similar lesion distributions and configurations. Our results show that modeling with multi-modal data, representative of physicians' diagnostic viewing behaviors and thought processes, is feasible and informative to gain insights into physicians' cognitive strategies, as well as medical image understanding.

Idiosyncratic biases in the perception of medical images

Controllable Medical Image Generation via Generative Adversarial Networks

On the notion of Hallucinations from the lens of Bias and Validity in Synthetic CXR Images

Sequentially Dependent Errors Generalize Across Naturalistic Mammogram Stimuli

Diagnostic error and bias in the department of radiology: a pictorial essay

Neurocognitive Mechanism of Radiologists Perceptual Errors: Results of Preliminary Studies

Bias in artificial intelligence for medical imaging: fundamentals, detection, avoidance, mitigation, challenges, ethics, and prospects

Gender and Ethnicity Bias of Text-to-Image Generative Artificial Intelligence in Medical Imaging, Part 2: Analysis of DALL-E 3

The Limits of Fair Medical Imaging AI In The Wild

Towards objective and systematic evaluation of bias in artificial intelligence for medical imaging

Visual perceptual artefacts in cardiothoracic imaging: Implications for diagnostic accuracy and strategies for mitigation

Discrimination of Radiologists Utilizing Eye-Tracking Technology and Machine Learning: A Case Study

Modeling eye movement patterns to characterize perceptual skill in image-based diagnostic reasoning processes

Optimizing Analysis, Visualization, and Navigation of Large Image Data Sets: One 5000-Section CT Scan Can Ruin Your Whole Day

Gender and Ethnicity Bias of Text-to-Image Generative Artificial Intelligence in Medical Imaging, Part 1: Preliminary Evaluation

Informatics in Radiology: What Can You See in a Single Glance and How Might This Guide Visual Search in Medical Images?

Algorithmic transparency and interpretability measures improve radiologists' performance in BI-RADS 4 classification

Demographic Bias of Expert-Level Vision-Language Foundation Models in Medical Imaging

Assessing GPT-4 multimodal performance in radiological image analysis

Controllable Medical Image Generation via GAN