Abstract:With the development of Natural Language Processing, Automatic question-answering system such as Waston, Siri, Alexa, has become one of the most important NLP applications. Nowadays, enterprises try to build automatic custom service chatbots to save human resources and provide a 24-hour customer service. Evaluation of chatbots currently relied greatly on human annotation which cost a plenty of time. Thus, has initiated a new Short Text Conversation subtask called Dialogue Quality (DQ) and Nugget Detection (ND) which aim to automatically evaluate dialogues generated by chatbots. In this paper, we solve the DQ and ND subtasks by deep neural network. We proposed two models for both DQ and ND subtasks which is constructed by hierarchical structure: embedding layer, utterance layer, context layer and memory layer, to hierarchical learn dialogue representation from word level, sentence level, context level to long range context level. Furthermore, we apply gating and attention mechanism at utterance layer and context layer to improve the performance. We also tried BERT to replace embedding layer and utterance layer as sentence representation. The result shows that BERT produced a better utterance representation than multi-stack CNN for both DQ and ND subtasks and outperform other models proposed by other researches. The evaluation measures are proposed by , that is, NMD, RSNOD for DQ and JSD, RNSS for ND, which is not traditional evaluation measures such as accuracy, precision, recall and f1-score. Thus, we have done a series of experiments by using traditional evaluation measures and analyze the performance and error.

CNN-BLSTM Based Question Detection from Dialogs Considering Phase and Context Information.

Question Detection from Acoustic Features Using Recurrent Neural Network with Gated Recurrent Unit

Analysis on Gated Recurrent Unit based Question Detection Approach

Detecting Context Dependent Messages in a Conversational Environment.

Acoustic features prominence based Chinese question detection

An Interpretable and Generalizable Speech Detector Based on a CNN-LSTM Framework

Detection of Hidden Feature Requests from Massive Chat Messages Via Deep Siamese Network

Short Text Conversation Based on Deep Neural Network and Analysis on Evaluation Measures

Dialog context language modeling with recurrent neural networks

Multimodal Dialog System: Relational Graph-based Context-aware Question Understanding

Annotation and Detection of Emotion in Text-based Dialogue Systems with CNN

Joint Intent Detection Model for Task-oriented Human-Computer Dialogue System Using Asynchronous Training

An attention mechanism and multi-granularity-based Bi-LSTM model for Chinese Q&A system

Neural Multi-Task Learning for Teacher Question Detection in Online Classrooms

Topic Segmentation of Web Documents with Automatic Cue Phrase Identification and BLSTM-CNN

Combining CNN and BLSTM to Extract Textual and Acoustic Features for Recognizing Stances in Mandarin Ideological Debate Competition

Context-aware Natural Language Generation for Spoken Dialogue Systems.

Multi-level Gated Recurrent Neural Network for Dialog Act Classification

Chinese Dialogue Analysis Using Multi-Task Learning Framework

Channel-Aware Decoupling Network for Multiturn Dialog Comprehension.

BERT-CoQAC: BERT-based Conversational Question Answering in Context