Improve web search using image snippets

Xiao-Bing Xue

2006-01-01

Abstract:We propose two new online methods for estimating the size of a backtracking search tree. The first method is based on a weighted sample of the branches visited by chronological backtracking. The second is a recursive method based on assuming that the ...

What problem does this paper attempt to address?

Exploiting image contents in web search

Zhi-Hua Zhou,Hong-Bin Dai

2007-01-01

Abstract:Web search is a challenging task. Previous research mainly exploits texts on the Web pages or link information between the pages, while multimedia information is largely ignored. This paper proposes a new framework for Web search, which exploits image contents to help improve the search performance. In this framework, candidate images are retrieved at first by considering their associated text information. Then, images related to the query are identified by analyzing the density of the visual feature space. After that, an image-based rank of the Web pages is generated, which is combined with the traditional keyword-based search result to produce the final search result. Experiments demonstrate the promise of the proposed framework.
Effective browsing of web image search results.

Hao Liu,Xing Xie,Xiaoou Tang,Zhiwei Li,Wei-Ying Ma

DOI: https://doi.org/10.1145/1026711.1026726

2004-01-01

Abstract:ABSTRACTThe rapid development of web image search engines has enabled users to search hundred million of images available on the Web. However, due to the unsatisfactory performance of current search technologies, people still need to spend much time in navigating through the large number of result pages to find images of their interest. In this paper, we analyze the user information needs for web image search results browsing and propose to employ a similarity-based organization to present the search results. A user study is carried out to compare our approach with a ranking-based list interface and a cluster-based interface. Experimental results show that similarity-based presentation can help users to explore image search results more naturally and efficiently
Improving Search Snippets in Context-Aware Web Search Scenarios.

Jia Chen,Jiaxin Mao,Yiqun Liu,Min Zhang,Shaoping Ma

DOI: https://doi.org/10.1007/978-3-030-56725-5_1

2020-01-01

Abstract:As an essential part in web search, search snippets usually provide result previews for users to either gather useful information or make click-through decisions. In complex search scenarios, users may need to submit multiple queries to search systems until their information needs are satisfied. As user intents tend to be ambiguous, incorporating contextual information for user modeling has been proved effective in many session-level tasks. Therefore, the generation of search snippets may also benefit from the integration of context information. However, to our best knowledge, most existing snippet generation methods ignore user interaction and focus merely on the query content. Whether it is useful of exploiting session contexts to improve search snippets still remains inscrutable. To this end, we propose a snippet generation model which considers session contexts. The proposed method utilizes the query sequence as well as users’ interaction behaviors within a session to model users’ session-level information needs. We also adopt practical log-based search data to evaluate the performance of the proposed method. Experiment results based on both expert annotation and user preference test show the effectiveness of considering contextual information in search snippet generation.
Intuitive and effective interfaces for WWW image search engines.

Zhiwei Li,Xing Xie,Hao Liu,Xiaoou Tang,Mingjing Li,Wei-Ying Ma

DOI: https://doi.org/10.1145/1027527.1027697

2004-01-01

Abstract:ABSTRACTWeb image search engine has become an important tool to organize digital images on the Web. However, most commercial search engines still use a list presentation while little effort has been placed on improving their usability. How to present the image search results in a more intuitive and effective way is still an open question to be carefully studied. In this demo, we present iFind, a scalable Web image search engine, in which we integrated two kinds of search result browsing interfaces. User study results have proved that our interfaces are superior to traditional interfaces.
PictureBook: A Text-and-Image Summary System for Web Search Result

Mei Wang,Hongtao Xu,Guoyu Hao,Xiangdong Zhou,Wei Wang,Qi Zhang,Baile Shi

DOI: https://doi.org/10.1109/icde.2008.4497634

2008-01-01

Abstract:Search engine technology plays an important role in Web information retrieval. However, with Internet information explosion, traditional searching techniques cannot provide satisfactory result due to problems such as huge number of result Web pages, unintuitive ranking, etc. Therefore, the reorganization and post-processing of Web search results have been extensively studied to help user effectively obtain useful information. Previous studies mainly focused on Web page clustering, document summary, visualization of search results, etc, which are applied separately to either text or image search. In this paper, we propose a demo to illustrate a new Web search result summary system - PictureBook, which combines text and image retrieval using techniques of multiple document summarization and image semantics analysis. Particularly, audience can interactively investigate the effect of the combined text and image summary in Web information searching and knowledge acquisition. We also introduce our new image semantic analysis method based on generalized discriminant analysis(GDA).
Web-Based Image Retrieval: A Hybrid Approach

YT Zhuang,Q Li,RWH Lau

DOI: https://doi.org/10.1109/cgi.2001.934659

2001-01-01

Abstract:In recent years, image retrieval has received tremendous attention and some progress has been made. However, most existing work on image retrieval focuses on specific issues and techniques local to image computing or access. Little work has been done to combine various aspects into a single framework. We describe our approach of developing a general-purpose image retrieval system over the Web. A main feature of our system is its hybrid approach by integrating both semantic and visual feature retrieval methods. By combining keyword-based query selection with content-based retrieval techniques, the system is able to provide effective image searching and retrieval. The results are further improved through a relevance feedback process. A research prototype system has been constructed on the Web environment, and experimental results demonstrate the effectiveness of the new approach.
Improve Ranking by Using Image Information

Qing Yu,Shuming Shi,Zhiwei Li,Ji-Rong Wen,Wei-Ying Ma

DOI: https://doi.org/10.1007/978-3-540-71496-5_62

2007-01-01

Abstract:This paper explores the feasibility of including image information embedded in Web pages in relevance computation to improve search performance. In determining the ranking of Web pages against a given query, most (if not all) modern Web search engines consider two kinds of factors: text information (including title, URL, body text, anchor text, etc) and static ranking (e.g. PageRank [1]). Although images have been widely used to help represent Web pages and carry valuable information, little work has been done to take advantage of them in computing the relevance score of a Web page given a query. We propose, in this paper, a framework to contain image information in ranking functions. Preliminary experimental results show that, when image information is used properly, ranking results can be improved.
Visual query suggestion: Towards capturing user intent in internet image search

Zheng-Jun Zha,Linjun Yang,Tao Mei,Meng Wang,Zengfu Wang,Tat-Seng Chua,Xian-Sheng Hua

DOI: https://doi.org/10.1145/1823746.1823747

2010-01-01

Abstract:Query suggestion is an effective approach to bridge the Intention Gap between the users' search intents and queries. Most existing search engines are able to automatically suggest a list of textual query terms based on users' current query input, which can be called Textual Query Suggestion. This article proposes a new query suggestion scheme named Visual Query Suggestion (VQS) which is dedicated to image search. VQS provides a more effective query interface to help users to precisely express their search intents by joint text and image suggestions. When a user submits a textual query, VQS first provides a list of suggestions, each containing a keyword and a collection of representative images in a dropdown menu. Once the user selects one of the suggestions, the corresponding keyword will be added to complement the initial query as the new textual query, while the image collection will be used as the visual query to further represent the search intent. VQS then performs image search based on the new textual query using text search techniques, as well as content-based visual retrieval to refine the search results by using the corresponding images as query examples. We compare VQS against three popular image search engines, and show that VQS outperforms these engines in terms of both the quality of query suggestion and the search performance.
Improving XML Search by Generating and Utilizing Informative Result Snippets

Ziyang Liu,Yu Huang,Yi Chen

DOI: https://doi.org/10.1145/1806907.1806911

IF: 1.6289

2010-01-01

ACM Transactions on Database Systems

Abstract:Snippets are used by almost every text search engine to complement the ranking scheme in order to effectively handle user searches, which are inherently ambiguous and whose relevance semantics are difficult to assess. Despite the fact that XML is a standard representation format of Web data, research on generating result snippets for XML search remains limited. To tackle this important yet open problem, in this article, we present a system eXtract which generates snippets for XML search results. We identify that a good XML result snippet should be a meaningful information unit of a small size that effectively summarizes this query result and differentiates it from others, according to which users can quickly assess the relevance of the query result. We have designed and implemented a novel algorithm to satisfy these requirements. Furthermore, we propose to cluster the query results based on their snippets. Since XML result clustering can only be done at query time, snippet-based clustering significantly improves the efficiency while compromising little clustering accuracy. We verified the efficiency and effectiveness of our approach through experiments.
Using Implicit Relevane Feedback to Advance Web Image Search

En Cheng,Feng Jing,Mingjing Li,Weiying Ma,Hai Jin

DOI: https://doi.org/10.1109/icme.2006.262895

2006-01-01

Abstract:Although relevance feedback has been extensively studied in content-based image retrieval in the academic area, no commercial Web image search engine has employed the idea. There are several obstacles for Web image search engines in applying relevance feedback. To overcome these obstacles, we proposed an efficient implicit relevance feedback mechanism. The proposed mechanism shows advantage over traditional relevance feedback methods in the following three aspects. Firstly, instead of enforcing the users to make explicit judgment on the results, our method regards user's click-through data as implicit relevance feedback which release burden from users. Secondly, a hierarchical image search results clustering algorithm is proposed to semantically organize the search results. Using the clustering results as features, our relevance feedback scheme could catch and reflect users' search intention precisely. Lastly, unlike traditional relevance feedback user interface which hardily substitutes subsequent results for previous ones, our method employed friendly recommendation rather than substitution to let the user narrow down on the refined images. To evaluate the implicit relevance feedback mechanism, comprehensive user studies were performed
The Contemporary Art of Image Search: Iterative User Intent Expansion via Vision-Language Model

Yilin Ye,Qian Zhu,Shishi Xiao,Kang Zhang,Wei Zeng

2023-12-05

Abstract:Image search is an essential and user-friendly method to explore vast galleries of digital images. However, existing image search methods heavily rely on proximity measurements like tag matching or image similarity, requiring precise user inputs for satisfactory results. To meet the growing demand for a contemporary image search engine that enables accurate comprehension of users' search intentions, we introduce an innovative user intent expansion framework. Our framework leverages visual-language models to parse and compose multi-modal user inputs to provide more accurate and satisfying results. It comprises two-stage processes: 1) a parsing stage that incorporates a language parsing module with large language models to enhance the comprehension of textual inputs, along with a visual parsing module that integrates an interactive segmentation module to swiftly identify detailed visual elements within images; and 2) a logic composition stage that combines multiple user search intents into a unified logic expression for more sophisticated operations in complex searching scenarios. Moreover, the intent expansion framework enables users to perform flexible contextualized interactions with the search results to further specify or adjust their detailed search intents iteratively. We implemented the framework into an image search system for NFT (non-fungible token) search and conducted a user study to evaluate its usability and novel properties. The results indicate that the proposed framework significantly improves users' image search experience. Particularly the parsing and contextualized interactions prove useful in allowing users to express their search intents more accurately and engage in a more enjoyable iterative search experience.

Information Retrieval,Artificial Intelligence,Computer Vision and Pattern Recognition,Human-Computer Interaction
Dialog-based Interactive Image Retrieval

Xiaoxiao Guo,Hui Wu,Yu Cheng,Steven Rennie,Gerald Tesauro,Rogerio Schmidt Feris

DOI: https://doi.org/10.48550/arXiv.1805.00145

2018-12-21

Abstract:Existing methods for interactive image retrieval have demonstrated the merit of integrating user feedback, improving retrieval results. However, most current systems rely on restricted forms of user feedback, such as binary relevance responses, or feedback based on a fixed set of relative attributes, which limits their impact. In this paper, we introduce a new approach to interactive image search that enables users to provide feedback via natural language, allowing for more natural and effective interaction. We formulate the task of dialog-based interactive image retrieval as a reinforcement learning problem, and reward the dialog system for improving the rank of the target image during each dialog turn. To mitigate the cumbersome and costly process of collecting human-machine conversations as the dialog system learns, we train our system with a user simulator, which is itself trained to describe the differences between target and candidate images. The efficacy of our approach is demonstrated in a footwear retrieval application. Experiments on both simulated and real-world data show that 1) our proposed learning framework achieves better accuracy than other supervised and reinforcement learning baselines and 2) user feedback based on natural language rather than pre-specified attributes leads to more effective retrieval results, and a more natural and expressive communication interface.

Computer Vision and Pattern Recognition,Artificial Intelligence
Social-Sensed Image Search.

Peng Cui,Shao-Wei Liu,Wen-Wu Zhu,Huan-Bo Luan,Tat-Seng Chua,Shi-Qiang Yang

DOI: https://doi.org/10.1145/2590974

2014-01-01

Abstract:Although Web search techniques have greatly facilitate users’ information seeking, there are still quite a lot of search sessions that cannot provide satisfactory results, which are more serious in Web image search scenarios. How to understand user intent from observed data is a fundamental issue and of paramount significance in improving image search performance. Previous research efforts mostly focus on discovering user intent either from clickthrough behavior in user search logs (e.g., Google), or from social data to facilitate vertical image search in a few limited social media platforms (e.g., Flickr). This article aims to combine the virtues of these two information sources to complement each other, that is, sensing and understanding users’ interests from social media platforms and transferring this knowledge to rerank the image search results in general image search engines. Toward this goal, we first propose a novel social-sensed image search framework, where both social media and search engine are jointly considered. To effectively and efficiently leverage these two kinds of platforms, we propose an example-based user interest representation and modeling method, where we construct a hybrid graph from social media and propose a hybrid random-walk algorithm to derive the user-image interest graph. Moreover, we propose a social-sensed image reranking method to integrate the user-image interest graph from social media and search results from general image search engines to rerank the images by fusing their social relevance and visual relevance. We conducted extensive experiments on real-world data from Flickr and Google image search, and the results demonstrated that the proposed methods can significantly improve the social relevance of image search results while maintaining visual relevance well.
Optimization of interactive visual-similarity-based search

Giang Phuong Nguyen,Marcel Worring

DOI: https://doi.org/10.1145/1324287.1324294

2008-01-01

Abstract:At one end of the spectrum, research in interactive content-based retrieval concentrates on machine learning methods for effective use of relevance feedback. On the other end, the information visualization community focuses on effective methods for conveying information to the user. What is lacking is research considering the information visualization and interactive retrieval as truly integrated parts of one content-based search system. In such an integrated system, there are many degrees of freedom like the similarity function, the number of images to display, the image size, different visualization modes, and possible feedback modes. To base the optimal values for all of those on user studies is unfeasible. We therefore develop search scenarios in which tasks and user actions are simulated. From there, the proposed scheme is optimized based on objective constraints and evaluation criteria. In such a manner, the degrees of freedom are reduced and the remaining degrees can be evaluated in user studies. In this article, we present a system that integrates advanced similarity based visualization with active learning. We have performed extensive experimentation on interactive category search with different image collections. The results using the proposed simulation scheme show that indeed the use of advanced visualization and active learning pays off in all of these datasets.
iWebImage: Enabling real-time interactive access to web images

Wenchang Xu,Yuanchun Shi,Xin Yang

DOI: https://doi.org/10.1109/ICPCA.2010.5704110

2010-01-01

Abstract:Images are widely used in web pages. However, most web images can only be viewed passively. Currently, it remains inconvenient for users to collect and save web images for further editing. The positioning of a certain image in web pages with a large number of images, such as the image search results, is also troublesome and time-wasting, especially for mobile devices with small screens. In this paper, we propose to enable real-time interactive access to web images and design three modes of browsing web images including normal mode, starred mode and advanced mode. We implement a plug-in for Microsoft Internet Explorer, called iWeblmage, which incorporates efficient computer graphics algorithms and provides a customized user interface supporting real-time interactive access to web images. Experimental results show the usage scenarios of the three browsing modes and prove that iWeblmage is well accepted by users.
Search Web Images Using Objects, Backgrounds and Conditions

Jiemi Zhang,Chenxia Wu,Deng Cai

DOI: https://doi.org/10.1145/2393347.2396350

2012-01-01

Abstract:As the volumes of web images have grown rapidly in the last decade, Content-Based Image Retrieval (CBIR) has attracted substantial interests as an effective tool to manage the images. Most existing CBIR systems focus on the object in the image, while ignoring the conditions (day/night, sunny/rain, etc ) and the backgrounds, both of which are very helpful to meet the user's information need. To overcome this shortcoming, in this paper, we present a novel CBIR system depending on a novel query formulation considering three aspects: Object, Background and Condition. Specifically, we design a user-friendly interface to help the user formulate a query. The interface can allow the user to give the percentage, relative position and size of each object in the background. Moreover, a corresponding effective ranking method is proposed to return the desirable search results. Experimental results demonstrate that our proposed system improves the searching performance and the user experience compared with the existing searching systems.
Photo-to-search: using multimodal queries to search the web from mobile devices.

Xin Fan,Xing Xie,Zhiwei Li,Mingjing Li,Wei-Ying Ma

DOI: https://doi.org/10.1145/1101826.1101851

2005-01-01

Abstract:ABSTRACTNowadays, mobile phones with the digital camera are getting more and more popular. With necessary technologies, they are possible to become a powerful tool to search the Web on the go. Most Web search engines only support text queries. Therefore, users have to convert their information needs into words. However, it is sometimes difficult to describe the needs in text and the text input is inconvenient on small devices. To solve the problem, we propose a system named Photo-to-Search which allows users to input multimodal queries. Particularly, we study queries with captured images and optional text messages in this paper. For example, the user can simply take a photo of the flower and input a few terms like "flower". Textually relevant Web images are retrieved according to the query terms. Afterwards, the snapped picture is compared with these images by the CBIR (Content Based Image Retrieval) method. According to the context of the visually similar images, related key phrases are extracted. Finally, the search results are returned in multiple forms. Our system can also search for very similar images on the Web, such as movie posters or photos of film stars, to find related information. Experimental results on the large scale data showed our system achieved satisfactory efficiency and performance.
Visual Query Suggestion

Zheng-Jun Zha,Linjun Yang,Tao Mei,Meng Wang,Zengfu Wang

DOI: https://doi.org/10.1145/1631272.1631278

2009-01-01

Abstract:Query suggestion is an effective approach to improve the usability of image search. Most existing search engines are able to automatically suggest a list of textual query terms based on users' current query input, which can be called Textual Query Suggestion. This paper proposes a new query suggestion scheme named Visual Query Suggestion (VQS) which is dedicated to image search. It provides a more effective query interface to formulate an intent-specific query by joint text and image suggestions. We show that VQS is able to more precisely and more quickly help users specify and deliver their search intents. When a user submits a text query, VQS first provides a list of suggestions, each containing a keyword and a collection of representative images in a dropdown menu. If the user selects one of the suggestions, the corresponding keyword will be added to complement the initial text query as the new text query, while the image collection will be formulated as the visual query. VQS then performs image search based on the new text query using text search techniques, as well as content-based visual retrieval to refine the search results by using the corresponding images as query examples. We compare VQS with three popular image search engines, and show that VQS outperforms these engines in terms of both the quality of query suggestion and search performance.
Why People Search for Images using Web Search Engines

Xiaohui Xie,Yiqun Liu,Maarten de Rijke,Jiyin He,Min Zhang,Shaoping Ma

DOI: https://doi.org/10.1145/3159652.3159686

2018-02-02

Abstract:What are the intents or goals behind human interactions with image search engines? Knowing why people search for images is of major concern to Web image search engines because user satisfaction may vary as intent varies. Previous analyses of image search behavior have mostly been query-based, focusing on what images people search for, rather than intent-based, that is, why people search for images. To date, there is no thorough investigation of how different image search intents affect users» search behavior. In this paper, we address the following questions: (1)Why do people search for images in text-based Web image search systems? (2)How does image search behavior change with user intent? (3)Can we predict user intent effectively from interactions during the early stages of a search session? To this end, we conduct both a lab-based user study and a commercial search log analysis. We show that user intents in image search can be grouped into three classes: Explore/Learn, Entertain, and Locate/Acquire. Our lab-based user study reveals different user behavior patterns under these three intents, such as first click time, query reformulation, dwell time and mouse movement on the result page. Based on user interaction features during the early stages of an image search session, that is, before mouse scroll, we develop an intent classifier that is able to achieve promising results for classifying intents into our three intent classes. Given that all features can be obtained online and unobtrusively, the predicted intents can provide guidance for choosing ranking methods immediately after scrolling.
New Query Refinement and Semantics Integrated Image Retrieval System with Semiautomatic Annotation Scheme.

XQ Zhu,HL Zhang,WY Liu,CH Hu,LD Wu

DOI: https://doi.org/10.1117/1.1406504

IF: 0.829

2001-01-01

Journal of Electronic Imaging

Abstract:Relevance feedback is a powerful and widely used technique in content-based image retrieval systems. However, most relevance feedback approaches use only weighted feature sum of the feedback images to optimize the query for refining image similarity assessment. Such approaches do not work very well in most cases, especially when the user wants to express an ‘‘OR’’ relationship among the queries. In this paper, we propose three methods, weighted distance sum, minimal distance (MD), and minimal distance rank (MDR), to measure the similarity between images in database and the feedback images in query refinement. After experimental comparisons we propose a relevance feedback scheme using the MDR method and the MD method to describe the user’s multiple intentions. Based on this scheme, an image retrieval and semiautomatic annotation system, iFind [Y. Lu, C. H. Hu, X. Q. Zhu, H. J. Zhang, and Q. Yang, ‘‘A unified framework for semantics and feature based relevance feedback in image retrieval systems,’’ in Proc. 8th ACM International Conference on Multimedia, pp. 31–37 (2000) and X. Q. Zhu, W. Y. Liu, H. J. Zhang, and L. D. Wu, ‘‘An image retrieval and semi-automatic annotation scheme for large image databases on the web,’’ in Proc. SPIE 4311, 168–177 (2001)], which integrates query refinement and semantic information, is presented. Experiments show that the proposed methods can result in substantial improvement in retrieval accuracy and can be especially useful for retrieval or annotating large image databases.

Improve web search using image snippets

Exploiting image contents in web search

Effective browsing of web image search results.

Improving Search Snippets in Context-Aware Web Search Scenarios.

Intuitive and effective interfaces for WWW image search engines.

PictureBook: A Text-and-Image Summary System for Web Search Result

Web-Based Image Retrieval: A Hybrid Approach

Improve Ranking by Using Image Information

Visual query suggestion: Towards capturing user intent in internet image search

Improving XML Search by Generating and Utilizing Informative Result Snippets

Using Implicit Relevane Feedback to Advance Web Image Search

The Contemporary Art of Image Search: Iterative User Intent Expansion via Vision-Language Model

Dialog-based Interactive Image Retrieval

Social-Sensed Image Search.

Optimization of interactive visual-similarity-based search

iWebImage: Enabling real-time interactive access to web images

Search Web Images Using Objects, Backgrounds and Conditions

Photo-to-search: using multimodal queries to search the web from mobile devices.

Visual Query Suggestion

Why People Search for Images using Web Search Engines

New Query Refinement and Semantics Integrated Image Retrieval System with Semiautomatic Annotation Scheme.