Abstract:The advent of Large Language Models (LLMs) has paved the way for AI search engines, e.g., SearchGPT, showcasing a new paradigm in human-internet interaction. However, most current AI search engines are limited to text-only settings, neglecting the multimodal user queries and the text-image interleaved nature of website information. Recently, Large Multimodal Models (LMMs) have made impressive strides. Yet, whether they can function as AI search engines remains under-explored, leaving the potential of LMMs in multimodal search an open question. To this end, we first design a delicate pipeline, MMSearch-Engine, to empower any LMMs with multimodal search capabilities. On top of this, we introduce MMSearch, a comprehensive evaluation benchmark to assess the multimodal search performance of LMMs. The curated dataset contains 300 manually collected instances spanning 14 subfields, which involves no overlap with the current LMMs' training data, ensuring the correct answer can only be obtained within searching. By using MMSearch-Engine, the LMMs are evaluated by performing three individual tasks (requery, rerank, and summarization), and one challenging end-to-end task with a complete searching process. We conduct extensive experiments on closed-source and open-source LMMs. Among all tested models, GPT-4o with MMSearch-Engine achieves the best results, which surpasses the commercial product, Perplexity Pro, in the end-to-end task, demonstrating the effectiveness of our proposed pipeline. We further present error analysis to unveil current LMMs still struggle to fully grasp the multimodal search tasks, and conduct ablation study to indicate the potential of scaling test-time computation for AI search engine. We hope MMSearch may provide unique insights to guide the future development of multimodal AI search engine. Project Page: <a class="link-external link-https" href="https://mmsearch.github.io" rel="external noopener nofollow">this https URL</a>

Region-action LSTM for Mouse Interaction Sequence Based Search Satisfaction Evaluation

Search Action Sequence Modeling With Long Short-Term Memory For Search Task Success Evaluation

Search Task Success Evaluation by Exploiting Multi-View Active Semi-Supervised Learning.

A Multi-View Semi-Supervised Approach for Task-Level Web Search Success Evaluation.

User Satisfaction Prediction with Mouse Movement Information in Heterogeneous Search Environment

Different Users, Different Opinions: Predicting Search Satisfaction With Mouse Movement Information

Explorations of Skeleton Features for LSTM-based Action Recognition

Learning Efficient Representations of Mouse Movements to Predict User Attention

Behavior Recognition in Mouse Videos Using Contextual Features Encoded by Spatial-temporal Stacked Fisher Vectors.

Still Haven't Found What You're Looking For -- Detecting the Intent of Web Search Missions from User Interaction Features

Fusing Geometric Features for Skeleton-Based Action Recognition Using Multilayer LSTM Networks

Social Behavior Recognition in Mouse Video Using Agent Embedding and LSTM Modelling.

Satisfaction Prediction of Web Search Users

MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Engines

SERank: Optimize Sequencewise Learning to Rank Using Squeeze-and-Excitation Network

Sequential Search with Off-Policy Reinforcement Learning

Deep Learning of Human Information Foraging Behavior with a Search Engine

Long- and Short-term Preference Learning with Enhanced Spatial Transformer for Next POI Recommendation

A Deep Reinforcement Learning Approach for Interactive Search with Sentence-level Feedback

SRSA: A Cost-Efficient Strategy-Router Search Agent for Real-world Human-Machine Interactions

RES-StS: Referring Expression Speaker via Self-training with Scorer for Goal-Oriented Vision-Language Navigation