Abstract:Recent advances in the development of large language models are rapidly changing how online applications function. LLM-based search tools, for instance, offer a natural language interface that can accommodate complex queries and provide detailed, direct responses. At the same time, there have been concerns about the veracity of the information provided by LLM-based tools due to potential mistakes or fabrications that can arise in algorithmically generated text. In a set of online experiments we investigate how LLM-based search changes people's behavior relative to traditional search, and what can be done to mitigate overreliance on LLM-based output. Participants in our experiments were asked to solve a series of decision tasks that involved researching and comparing different products, and were randomly assigned to do so with either an LLM-based search tool or a traditional search engine. In our first experiment, we find that participants using the LLM-based tool were able to complete their tasks more quickly, using fewer but more complex queries than those who used traditional search. Moreover, these participants reported a more satisfying experience with the LLM-based search tool. When the information presented by the LLM was reliable, participants using the tool made decisions with a comparable level of accuracy to those using traditional search, however we observed overreliance on incorrect information when the LLM erred. Our second experiment further investigated this issue by randomly assigning some users to see a simple color-coded highlighting scheme to alert them to potentially incorrect or misleading information in the LLM responses. Overall we find that this confidence-based highlighting substantially increases the rate at which users spot incorrect information, improving the accuracy of their overall decisions while leaving most other measures unaffected.

Choosing Between an LLM versus Search for Learning: A HigherEd Student Perspective

"Which LLM should I use?": Evaluating LLMs for tasks performed by Undergraduate Computer Science Students

Large Language Models vs. Search Engines: Evaluating User Preferences Across Varied Information Retrieval Scenarios

Understanding Help-Seeking Behavior of Students Using LLMs vs. Web Search for Writing SQL Queries

Student Perspectives on Using a Large Language Model (LLM) for an Assignment on Professional Ethics

Enhancing Exploratory Learning through Exploratory Search with the Emergence of Large Language Models

Search Engines, LLMs or Both? Evaluating Information Seeking Strategies for Answering Health Questions

"With Great Power Comes Great Responsibility!": Student and Instructor Perspectives on the influence of LLMs on Undergraduate Engineering Education

Comparing Traditional and LLM-based Search for Consumer Choice: A Randomized Experiment

An Empirical Study on Usage and Perceptions of LLMs in a Software Engineering Project

When Search Engine Services meet Large Language Models: Visions and Challenges

Comparing the Utility, Preference, and Performance of Course Material Search Functionality and Retrieval-Augmented Generation Large Language Model (RAG-LLM) AI Chatbots in Information-Seeking Tasks

Analyzing LLM Usage in an Advanced Computing Class in India

Evaluating the Impact of Advanced LLM Techniques on AI-Lecture Tutors for a Robotics Course

CS1-LLM: Integrating LLMs into CS1 Instruction

An Exploratory Study on Upper-Level Computing Students' Use of Large Language Models as Tools in a Semester-Long Project

Comparing Traditional and LLM-based Search for Image Geolocation

Leveraging LLMs to Enable Natural Language Search on Go-to-market Platforms

Evaluating the Effectiveness of LLMs in Introductory Computer Science Education: A Semester-Long Field Study

LLMs as Research Tools: A Large Scale Survey of Researchers' Usage and Perceptions

LLM With Tools: A Survey