A Financial Service Chatbot based on Deep Bidirectional Transformers

Shi Yu,Yuxin Chen,Hussain Zaidi

DOI: https://doi.org/10.48550/arXiv.2003.04987

2020-02-18

Abstract:We develop a chatbot using Deep Bidirectional Transformer models (BERT) to handle client questions in financial investment customer service. The bot can recognize 381 intents, and decides when to say "I don't know" and escalates irrelevant/uncertain questions to human operators. Our main novel contribution is the discussion about uncertainty measure for BERT, where three different approaches are systematically compared on real problems. We investigated two uncertainty metrics, information entropy and variance of dropout sampling in BERT, followed by mixed-integer programming to optimize decision thresholds. Another novel contribution is the usage of BERT as a language model in automatic spelling correction. Inputs with accidental spelling errors can significantly decrease intent classification performance. The proposed approach combines probabilities from masked language model and word edit distances to find the best corrections for misspelled words. The chatbot and the entire conversational AI system are developed using open-source tools, and deployed within our company's intranet. The proposed approach can be useful for industries seeking similar in-house solutions in their specific business domains. We share all our code and a sample chatbot built on a public dataset on Github.

Computation and Language,Machine Learning

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is to develop a financial investment customer - service chatbot based on the deep bidirectional Transformer model (such as BERT) to handle customer inquiries. Specifically, this paper mainly focuses on the following aspects:

1. **Intent Recognition and Classification**:
- The chatbot needs to be able to recognize 381 different intents and accurately classify customers' queries. This involves the intent classification task in natural language processing (NLP).

2. **Uncertainty Measurement**:
- The paper proposes and compares three different methods to measure the uncertainty of BERT model predictions: information entropy, Dropout sampling variance, and introducing a dummy class. These methods aim to help the robot determine when it should escalate uncertain or irrelevant questions to human customer service representatives for handling.

3. **Automatic Spelling Correction**:
- Spelling mistakes can significantly reduce the performance of intent classification. Therefore, the paper explores the method of using BERT as a language model to perform automatic spelling correction, combining the probabilities of the masked - language model and the word - edit distance to find the best spelling - correction solution.

4. **Optimizing Decision Thresholds**:
- Optimize decision thresholds through mixed - integer programming (MIP) to ensure that questions can be effectively escalated to human experts for handling when high uncertainty is detected. This process involves understanding how to make optimal decisions based on the predicted uncertainty.

5. **In - house Deployment in Enterprises**:
- To protect sensitive information, the chatbot system is developed and deployed in the company's internal network, using open - source tools such as Rasa, and is fully customized to meet specific business needs.

### Formula Display

- **Information Entropy Formula**:
\[
H = -\sum_{k = 1}^{K}p_{ik}\log p_{ik}
\]
where \(p_{ik}\) is the predicted probability that the \(i\) - th sample belongs to the \(k\) - th class.

- **Dropout Variance Formula**:
\[
\text{Var}(p_{ik})=\frac{1}{T}\sum_{t = 1}^{T}(p_{ik}^{(t)}-\bar{p}_{ik})^2
\]
where \(p_{ik}^{(t)}\) is the predicted probability of the \(t\) - th Monte Carlo sampling, and \(\bar{p}_{ik}\) is the average predicted probability.

- **Mixed - Integer Programming Optimization Problem**:
\[
\min_{x,b}\sum_{i,k}(x_{ik}-l_{ik})^2
\]
Subject to the constraints:
\[
x_{ik} = 0\quad\text{if}\quad E_i\geq b,\quad\text{for}\quad k\in1,\ldots,K
\]
\[
x_{ik} = 1\quad\text{if}\quad E_i\geq b,\quad\text{for}\quad k = K + 1
\]
\[
x_{ik}\in\{0,1\},\quad\sum_{k = 1}^{K + 1}x_{ik}=1,\quad\forall i\in1,\ldots,N
\]
\[
b\geq0
\]

Through the above methods, the paper aims to improve the accuracy and reliability of the chatbot, especially when handling complex and uncertain customer queries.