Abstract:Image search stands as a pivotal task in multimedia and computer vision, finding applications across diverse domains, ranging from internet search to medical diagnostics. Conventional image search systems operate by accepting textual or visual queries, retrieving the top-relevant candidate results from the database. However, prevalent methods often rely on single-turn procedures, introducing potential inaccuracies and limited recall. These methods also face the challenges, such as vocabulary mismatch and the semantic gap, constraining their overall effectiveness. To address these issues, we propose an interactive image retrieval system capable of refining queries based on user relevance feedback in a multi-turn setting. This system incorporates a vision language model (VLM) based image captioner to enhance the quality of text-based queries, resulting in more informative queries with each iteration. Moreover, we introduce a large language model (LLM) based denoiser to refine text-based query expansions, mitigating inaccuracies in image descriptions generated by captioning models. To evaluate our system, we curate a new dataset by adapting the MSR-VTT video retrieval dataset to the image retrieval task, offering multiple relevant ground truth images for each query. Through comprehensive experiments, we validate the effectiveness of our proposed system against baseline methods, achieving state-of-the-art performance with a notable 10\% improvement in terms of recall. Our contributions encompass the development of an innovative interactive image retrieval system, the integration of an LLM-based denoiser, the curation of a meticulously designed evaluation dataset, and thorough experimental validation.

Evolving optimal feature set by interactive reinforcement learning for image retrieval

Image retrieval based on incremental subspace learning

Incremental semi-supervised subspace learning for image retrieval.

Learning A Semantic Space from User'S Relevance Feedback for Image Retrieval

Optimizing Learning in Image Retrieval

Optimal Adaptive Learning For Image Retrieval

Dialog-based Interactive Image Retrieval

Towards Optimal Query Design For Relevance Feedback In Image Retrieval

A Neural Network Based Self-Learning Algorithm of Image Retrieval

A unified optimization based learning method for image retrieval

Relevance Feedback Learning with Feature Selection in Region-Based Image Retrieval

Exploration of Image Search Results Quality Assessment

Improving image retrieval performance by integrating long-term learning with short-term learning

A Rich Get Richer Strategy For Content-Based Image Retrieval

Adaptive tree similarity learning for image retrieval

Relevance Feedback For Keyword And Visual Feature-Based Image Retrieval

Adaptive relevance feedback based on Bayesian inference for image retrieval

Enhancing Relevance Feedback in Image Retrieval Using Unlabeled Data

Image Retrieval Framework Driven by Association Feedback with Feature Elements Evaluation Built In

Enhancing Interactive Image Retrieval With Query Rewriting Using Large Language Models and Vision Language Models

An Exploratory Study on a Reinforcement Learning Prototype for Multimodal Image Retrieval Using a Conversational Search Interface