Interactive cross and multimodal biomedical image retrieval based on automatic region-of-interest (ROI) identification and classification
Md Mahmudur Rahman,Daekeun You,Matthew S. Simpson,Sameer K. Antani,Dina Demner-Fushman,George R. Thoma
DOI: https://doi.org/10.1007/s13735-014-0057-9
2014-05-13
International Journal of Multimedia Information Retrieval
Abstract:In biomedical articles, authors often use annotation markers such as arrows, letters, or symbols overlaid on figures and illustrations to highlight ROIs. These annotations are then referenced and correlated with concepts in the caption text or figure citations in the article text. This association creates a bridge between the visual characteristics of important regions within an image and their semantic interpretation. Identifying these assists in extracting ROIs that are likely to be highly relevant to the discussion in the article text. The aim of this work is to perform semantic search without knowing the concept keyword or the specific name of the visual pattern or appearance. We consider the problem of cross and multimodal retrieval of images from articles which contains components of text and images. Our proposed method localizes and recognizes the annotations by utilizing a combination of rule-based and statistical image processing techniques. The image regions are then annotated for classification using biomedical concepts obtained from a glossary of imaging terms. Similar automatic ROI extraction can be applied to query images, or to interactively mark an ROI. As a result, visual characteristics of the ROIs can be mapped to text concepts and then used to search image captions. In addition, the system can toggle the search process from purely perceptual to a conceptual one (crossmodal) based on utilizing user feedback or integrate both perceptual and conceptual search in a multimodal search process. The hypothesis, that such approaches would improve biomedical image retrieval, was validated through experiments on a biomedical article dataset of thoracic CT scans.
computer science, artificial intelligence, software engineering