Structured Tree Alignment for Evaluation of (Speech) Constituency Parsing

Freda Shi,Kevin Gimpel,Karen Livescu

2024-06-20

Abstract:We present the structured average intersection-over-union ratio (STRUCT-IOU), a similarity metric between constituency parse trees motivated by the problem of evaluating speech parsers. STRUCT-IOU enables comparison between a constituency parse tree (over automatically recognized spoken word boundaries) with the ground-truth parse (over written words). To compute the metric, we project the ground-truth parse tree to the speech domain by forced alignment, align the projected ground-truth constituents with the predicted ones under certain structured constraints, and calculate the average IOU score across all aligned constituent pairs. STRUCT-IOU takes word boundaries into account and overcomes the challenge that the predicted words and ground truth may not have perfect one-to-one correspondence. Extending to the evaluation of text constituency parsing, we demonstrate that STRUCT-IOU can address token-mismatch issues, and shows higher tolerance to syntactically plausible parses than PARSEVAL (Black et al., 1991).

Computation and Language,Data Structures and Algorithms

What problem does this paper attempt to address?

The paper primarily focuses on addressing the following issues:

1. **Developing New Evaluation Metrics**: The paper introduces a new evaluation metric called STRUCT-IOU, which is designed to assess the performance of syntax structure analysis based on speech (specifically, sentence constituent analysis). This metric aims to overcome the limitations of traditional evaluation methods (such as PARSEVAL) when dealing with speech data.

2. **Adapting to the Characteristics of Speech Parsing**: Given the potential inaccuracy of word boundaries in automatic speech recognition, STRUCT-IOU takes into account the uncertainty of word boundaries and is capable of evaluating the similarity between parse trees, even when there is no one-to-one correspondence between the predicted words and the actual words.

3. **Addressing Token Mismatch Issues in Text Parsing**: Although designed for speech parsing, STRUCT-IOU is also applicable to the evaluation of text parsing. It can better handle issues caused by token inconsistencies and shows a higher tolerance for grammatically reasonable parses compared to the traditional PARSEVAL metric.

In summary, the goal of this research is to improve existing parsing evaluation methods by proposing STRUCT-IOU, making them more suitable for evaluating the results of syntactic structure parsing in speech and text, especially in situations where word boundaries are uncertain or there are token mismatches.