CHOPS: CHat with custOmer Profile Systems for Customer Service with LLMs

Jingzhe Shi,Jialuo Li,Qinwei Ma,Zaiwen Yang,Huan Ma,Lei Li

2024-07-17

Abstract:Businesses and software platforms are increasingly turning to Large Language Models (LLMs) such as GPT-3.5, GPT-4, GLM-3, and LLaMa-2 for chat assistance with file access or as reasoning agents for customer service. However, current LLM-based customer service models have limited integration with customer profiles and lack the operational capabilities necessary for effective service. Moreover, existing API integrations emphasize diversity over the precision and error avoidance essential in real-world customer service scenarios. To address these issues, we propose an LLM agent named CHOPS (CHat with custOmer Profile in existing System), designed to: (1) efficiently utilize existing databases or systems for accessing user information or interacting with these systems following existing guidelines; (2) provide accurate and reasonable responses or carry out required operations in the system while avoiding harmful operations; and (3) leverage a combination of small and large LLMs to achieve satisfying performance at a reasonable inference cost. We introduce a practical dataset, the CPHOS-dataset, which includes a database, guiding files, and QA pairs collected from CPHOS, an online platform that facilitates the organization of simulated Physics Olympiads for high school teachers and students. We have conducted extensive experiments to validate the performance of our proposed CHOPS architecture using the CPHOS-dataset, with the aim of demonstrating how LLMs can enhance or serve as alternatives to human customer service. Code for our proposed architecture and dataset can be found at {<a class="link-external link-https" href="https://github.com/JingzheShi/CHOPS" rel="external noopener nofollow">this https URL</a>}.

Computation and Language,Artificial Intelligence

What problem does this paper attempt to address?

The main problem this paper attempts to address is the inadequacy of existing customer service systems based on large language models (LLMs) in terms of integration and operational capabilities with customer profiles. Specifically:

1. **Limited Customer Profile Integration**: Current LLM customer service models perform poorly in integrating customer profiles, failing to fully utilize existing databases or systems to access user information.
2. **Lack of Operational Capability**: These models have limitations in executing system operations, especially in avoiding harmful operations.
3. **API Integration Issues**: Existing API integration methods focus more on diversity rather than precision and error avoidance, which is crucial in real customer service scenarios.

To tackle these issues, the authors propose an LLM agent architecture named CHOPS (CHat with custOmer Profile in existing System), which has the following features:

1. **Efficient Use of Existing Databases or Systems**: Capable of accessing user information or interacting with systems based on existing guidelines.
2. **Providing Accurate and Reasonable Responses**: Avoiding harmful operations when performing required actions in the system.
3. **Combining Small and Large LLMs**: By combining small and large LLMs, it provides satisfactory performance while maintaining low inference costs.

Additionally, the authors introduce a practical dataset called the CPHOS-dataset, which includes databases, guideline documents, and Q&A pairs collected from CPHOS* to validate the performance of the CHOPS architecture. Experimental results show that the CHOPS architecture improves accuracy while effectively controlling costs.