Graph Neural Networks for Contextual ASR with the Tree-Constrained Pointer Generator

Guangzhi Sun,Chao Zhang,Phil Woodland

2023-05-30

Abstract:The incorporation of biasing words obtained through contextual knowledge is of paramount importance in automatic speech recognition (ASR) applications. This paper proposes an innovative method for achieving end-to-end contextual ASR using graph neural network (GNN) encodings based on the tree-constrained pointer generator method. GNN node encodings facilitate lookahead for future word pieces in the process of ASR decoding at each tree node by incorporating information about all word pieces on the tree branches rooted from it. This results in a more precise prediction of the generation probability of the biasing words. The study explores three GNN encoding techniques, namely tree recursive neural networks, graph convolutional network (GCN), and GraphSAGE, along with different combinations of the complementary GCN and GraphSAGE structures. The performance of the systems was evaluated using the Librispeech and AMI corpus, following the visual-grounded contextual ASR pipeline. The findings indicate that using GNN encodings achieved consistent and significant reductions in word error rate (WER), particularly for words that are rare or have not been seen during the training process. Notably, the most effective combination of GNN encodings obtained more than 60% WER reduction for rare and unseen words compared to standard end-to-end systems.

Computation and Language,Sound,Audio and Speech Processing

What problem does this paper attempt to address?

The paper aims to address the accuracy issues faced by automatic speech recognition (ASR) systems when dealing with rare "long-tail" vocabulary that does not appear in the training data. Specifically, the paper proposes an innovative approach that utilizes a graph neural network (GNN) encoding based on a tree-constrained pointer generator method to achieve end-to-end context-aware ASR. By applying GNN node encoding to each tree node during the ASR decoding process, future word segment information can be obtained in advance, thereby improving the prediction accuracy of biased vocabulary generation probabilities. The study explores three GNN encoding techniques: Tree Recursive Neural Network, Graph Convolutional Network (GCN) and its variant GCNII, and GraphSAGE, and tests their different combinations. Experimental results show that using GNN encoding significantly reduces the word error rate (WER), especially for rare or unseen vocabulary during training. The most effective GNN encoding combination achieved over a 60% reduction in WER for rare and unseen words compared to the standard end-to-end system.

In summary, the main goal of the paper is to enhance the existing tree-constrained pointer generator component by introducing graph neural networks to improve the recognition capability of end-to-end ASR systems for rare vocabulary.