Abstract:While Machine Comprehension (MC) has attracted extensive research interests in recent years, existing approaches mainly belong to the category of Machine Reading Comprehension task which mines textual inputs (paragraphs and questions) to predict the answers (choices or text spans). However, there are a lot of MC tasks that accept audio input in addition to the textual input, e.g. English listening comprehension test. In this paper, we target the problem of Audio-Oriented Multimodal Machine Comprehension, and its goal is to answer questions based on the given audio and textual information. To solve this problem, we propose a Dynamic Inter- and Intra-modality Attention (DIIA) model to effectively fuse the two modalities (audio and textual). DIIA can work as an independent component and thus be easily integrated into existing MC models. Moreover, we further develop a Multimodal Knowledge Distillation (MKD) module to enable our multimodal MC model to accurately predict the answers based only on either the text or the audio. As a result, the proposed approach can handle various tasks including: Audio-Oriented Multimodal Machine Comprehension, Machine Reading Comprehension and Machine Listening Comprehension, in a single model, making fair comparisons possible between our model and the existing unimodal MC models. Experimental results and analysis prove the effectiveness of the proposed approaches. First, the proposed DIIA boosts the baseline models by up to 21.08% in terms of accuracy; Second, under the unimodal scenarios, the MKD module allows our multimodal MC model to significantly outperform the unimodal models by up to 18.87%, which are trained and tested with only audio or textual data.

Machine Comprehension of Spoken Content: TOEFL Listening Test and Spoken SQuAD

Towards Machine Comprehension of Spoken Content: Initial TOEFL Listening Comprehension Test by Machine

Spoken SQuAD: A Study of Mitigating the Impact of Speech Recognition Errors on Listening Comprehension

Language Cognition and Language Computation—human and Ma-Chine Language Understanding

Hierarchical Attention Model for Improved Machine Comprehension of Spoken Content

Machine Comprehension Using Match-LSTM and Answer Pointer

Smarnet: Teaching Machines to Read and Comprehend Like Human.

Exploring Speech Recognition, Translation, and Understanding with Discrete Speech Units: A Comparative Study

A Computer-Assisted Tool for Automatically Measuring Non-Native Japanese Oral Proficiency

Sogou Machine Reading Comprehension Toolkit

Teaching Machines to Extract Main Content for Machine Reading Comprehension

Audio-Oriented Multimodal Machine Comprehension: Task, Dataset and Model

A Survey on Neural Machine Reading Comprehension

Improving Textless Spoken Language Understanding with Discrete Units as Intermediate Target

A Spoken Language Understanding Approach Using Successive Learners

ODSQA: Open-domain Spoken Question Answering Dataset

CATSLU: the 1st Chinese Audio-Textual Spoken Language Understanding Challenge

Machine scoring of reading aloud item of large-scale college English oral tests

Utterance-focusing Multiway-Matching Network for Dialogue-Based Multiple-Choice Machine Reading Comprehension

VlogQA: Task, Dataset, and Baseline Models for Vietnamese Spoken-Based Machine Reading Comprehension

Bridging The Gap: Entailment Fused-T5 for Open-retrieval Conversational Machine Reading Comprehension