A new biomedical image search and visual literature navigation system
David Gelernter,Martin Schultz,Michael Krauthammer,Songhua Xu
2010-01-01
Abstract:As online resources continue to multiply, it becomes increasingly more difficult to keep up with the latest research results and make effective use of the literature and the relevant images on the Internet. This trend is especially true for the biomedical domain: PubMed has indexed more than 19 million biomedical articles. Such an explosion of biomedical information demands novel software support for effective information retrieval and management, in order to help biomedical professionals have access to the vast amount of knowledge available. In biomedical publications, images often concisely summarize a paper's key ideas and findings. Recent studies have therefore explored the use of images for tasks such as document classification or retrieval (IR) in biomedicine, working with low-level image features or image captions. However, none of the previous work has taken advantage of crucial information carried by image text, i.e., the text within images, such as image labels or annotations. We have shown in our own studies that up to 70% of image text is not contained in the captions of biomedical images. This can easily be seen in heat map images, where most of the labels (such as gene names) are not discussed in the caption. This discovery reveals new opportunities to improve access to the biomedical literature by indexing image text in research publications. Based on this idea, I will introduce a new biomedical image search and navigation system, called the "Yale Image Finder" (YIF), that makes novel use of image text for biomedical image retrieval and image guided visual literature navigation. YIF is a publicly accessible search engine featuring a new way of retrieving biomedical images and the associated papers by allowing users to search over text materials contained in images, which are extracted using optical character recognition (OCR). Image queries can be issued against image text and image caption, as well as the abstract and title of the paper that an image is associated with. Currently, YIF has indexed over 500,000 open access biomedical images from PubMed Central and it has been visited by more than 17,000 users worldwide, with over 76,000 pageviews. YIF introduces several novel features that provide a unique image search and image guided visual literature navigation experience. First, it allows users to search over text contained within the images themselves, such as image labels and annotations. This feature results in the retrieval of considerably more images than when searching over image caption alone. In our study, we find YIF is capable of retrieving 30-175% more biomedical images than comparable engines that search over image caption alone. Second, YIF allows users to explore related publications by assessing image similarity and displaying and linking sets of related images. By providing this feature of related images, YIF allows people to intuitively navigate through the biomedical literature in an image-oriented way. A typical search scenario of YIF starts with a thumbnail view of the initial search result, followed by a high resolution view of a user selected image, as well as YIF's recommendation on the most related images. In the first part of this thesis, I will discuss the image and text processing techniques developed to improve YIF's image text extraction and image retrieval performance, including an iterative and pivoting text region detection algorithm for locating and separating text regions from graphical regions in biomedical images; the use of the image text region detection and separation procedure as a preprocessing technique for improving the recall of image text extraction; a post-processing technique using context-based OCR error correction for improving the precision of image text extraction; and an algorithm for detecting associated image elements in biomedical images for more precise high-level image querying. In the second part of this thesis, I will discuss YIF's functionalities, system architecture, implementation, and its user characteristics.