Stefanie H. Klein,Dominik Papies,Sonja Utz
Abstract:As more and more people regularly interact with chatbots in their everyday lives, it is crucial to understand how users perceive and evaluate them. This study examined whether free text (vs. button) interaction with and social (vs. neutral) error responses of chatbots effectively improve user responses. Using an online experiment ( N = 416) in which participants interacted with a customer service chatbot, we investigated how interaction mechanism and error response influence perceived warmth and competence, satisfaction, usage intention, and emotional connection with the company. Contrary to our hypotheses, we found no evidence that free text interaction increases acceptance. Interacting via buttons was preferred, potentially because participants perceived the free text chatbot as less warm and competent. Interestingly, social error responses only increased satisfaction. We discuss the results considering recent advances in Large Language Models and highlight how the findings can guide further research and help practitioners develop efficient, user-friendly chatbots.
What problem does this paper attempt to address?
### The Problem the Paper Attempts to Solve
This paper aims to explore how different interaction mechanisms (free text input vs. button selection) and error response styles (social vs. neutral) affect user reactions when interacting with customer service chatbots. Specifically, the study focuses on the following five aspects:
1. **Perceived Warmth**: The friendliness and approachability of the chatbot as perceived by the user.
2. **Perceived Competence**: The user's evaluation of the chatbot's intelligence and problem-solving ability.
3. **Satisfaction**: The user's satisfaction with the chatbot experience.
4. **Usage Intention**: The user's willingness to continue using the chatbot in the future.
5. **Emotional Connection**: The user's emotional connection to the company.
### Research Background
With the development of artificial intelligence and automation technologies, the application of chatbots is becoming increasingly widespread, especially in customer service. Implementing chatbot technology can help companies save on labor and training costs but requires initial investment and technical support. Therefore, systematically studying how users perceive and evaluate chatbots becomes particularly important.
### Research Question
The main research question of this paper is:
- How do free text interaction and social error responses affect users' perceived warmth, perceived competence, satisfaction, usage intention, and emotional connection with the chatbot?
### Research Hypotheses
1. **H1**: When users primarily interact with the chatbot through free text (rather than buttons), (a) perceived warmth, (b) perceived competence, (c) satisfaction, (d) usage intention, and (e) emotional connection with the company will be higher.
2. **H2**: When the chatbot's error response is social (rather than neutral), (a) perceived warmth, (b) perceived competence, (c) satisfaction, (d) usage intention, and (e) emotional connection with the company will be higher.
3. **H3**: The type of interaction mechanism moderates the effect of error response on outcomes, i.e., when the interaction mechanism is free text (rather than buttons), the effect of social (rather than neutral) error responses is stronger.
4. **H4.1**: Perceived warmth mediates the positive effect of free text interaction on (a) satisfaction, (b) usage intention, and (c) emotional connection with the company.
5. **H4.2**: Perceived competence mediates the positive effect of free text interaction on (a) satisfaction, (b) usage intention, and (c) emotional connection with the company.
6. **H5.1**: Perceived warmth mediates the positive effect of social error responses on (a) satisfaction, (b) usage intention, and (c) emotional connection with the company.
7. **H5.2**: Perceived competence mediates the positive effect of social error responses on (a) satisfaction, (b) usage intention, and (c) emotional connection with the company.
### Method
The study employed a 2 (interaction mechanism: button vs. free text) × 2 (error response: neutral vs. social) between-subjects online experimental design. Participants were randomly assigned to one of four conditions and interacted with a real customer service chatbot. Each condition included 93 to 110 participants. The study received ethical committee approval and was pre-registered.
### Participants
A total of 450 German participants were recruited through the online sampling platform Prolific, with the final sample including 416 participants. The average age of participants was 29.90 years, with 50.00% being female, and 61.78% having used a chatbot at least once in the past three months.
### Experimental Procedure
Participants were asked to imagine themselves in a typical customer service scenario where they needed to complete a series of tasks, such as checking a gift card balance or inquiring about package delivery status. All participants received the same task list and fictitious customer credentials. After reading the instructions, participants clicked a button to open the chat window embedded in the online survey and interacted with the chatbot. To make the conversation more realistic, participants encountered at least one instance where the chatbot did not understand their request.
### Conclusion
Through this study, the authors hope to provide guidance for companies and practitioners to design efficient and user-friendly chatbots, thereby improving user experience and bringing economic benefits to the company.