ChatGPT vs. Google: A Comparative Study of Search Performance and User Experience

Ruiyun Xu,Yue Feng,Hailiang Chen

2023-07-04

Abstract:The advent of ChatGPT, a large language model-powered chatbot, has prompted questions about its potential implications for traditional search engines. In this study, we investigate the differences in user behavior when employing search engines and chatbot tools for information-seeking tasks. We carry out a randomized online experiment, dividing participants into two groups: one using a ChatGPT-like tool and the other using a Google Search-like tool. Our findings reveal that the ChatGPT group consistently spends less time on all tasks, with no significant difference in overall task performance between the groups. Notably, ChatGPT levels user search performance across different education levels and excels in answering straightforward questions and providing general solutions but falls short in fact-checking tasks. Users perceive ChatGPT's responses as having higher information quality compared to Google Search, despite displaying a similar level of trust in both tools. Furthermore, participants using ChatGPT report significantly better user experiences in terms of usefulness, enjoyment, and satisfaction, while perceived ease of use remains comparable between the two tools. However, ChatGPT may also lead to overreliance and generate or replicate misinformation, yielding inconsistent results. Our study offers valuable insights for search engine management and highlights opportunities for integrating chatbot technologies into search engine designs.

Artificial Intelligence,Human-Computer Interaction,Information Retrieval

What problem does this paper attempt to address?

The paper attempts to address the issue of exploring the differences in user behavior, search performance, and user experience between ChatGPT and traditional search engines (such as Google) in information retrieval tasks. Specifically, the researchers conducted a randomized online experiment to compare the behavior and performance of users when using a chat-based tool powered by large language models (similar to ChatGPT) and traditional search engine tools (similar to Google Search), aiming to answer the following research questions:

1. **Differences in User Behavior**: How do users' behaviors differ when using ChatGPT versus Google Search for information retrieval tasks? Are these differences consistent across different types of tasks?
2. **Impact of Education Level**: Can ChatGPT balance the search performance of users with different education levels?
3. **Information Quality and User Experience**: How do users evaluate the quality, trustworthiness, and overall user experience of the information provided by ChatGPT compared to traditional search engines (such as Google Search)?

Through these questions, the researchers hope to reveal the advantages and limitations of chatbot technologies like ChatGPT in the field of information retrieval, providing valuable references for the design and development of future search tools.