Abstract:Lecture slide presentations, a sequence of pages that contain text and figures accompanied by speech, are constructed and presented carefully in order to optimally transfer knowledge to students. Previous studies in multimedia and psychology attribute the effectiveness of lecture presentations to their multimodal nature. As a step toward developing AI to aid in student learning as intelligent teacher assistants, we introduce the Multimodal Lecture Presentations dataset as a large-scale benchmark testing the capabilities of machine learning models in multimodal understanding of educational content. Our dataset contains aligned slides and spoken language, for 180+ hours of video and 9000+ slides, with 10 lecturers from various subjects (e.g., computer science, dentistry, biology). We introduce two research tasks which are designed as stepping stones towards AI agents that can explain (automatically captioning a lecture presentation) and illustrate (synthesizing visual figures to accompany spoken explanations) educational content. We provide manual annotations to help implement these two research tasks and evaluate state-of-the-art models on them. Comparing baselines and human student performances, we find that current models struggle in (1) weak crossmodal alignment between slides and spoken text, (2) learning novel visual mediums, (3) technical language, and (4) long-range sequences. Towards addressing this issue, we also introduce PolyViLT, a multimodal transformer trained with a multi-instance learning loss that is more effective than current approaches. We conclude by shedding light on the challenges and opportunities in multimodal understanding of educational presentations.

Alignment Of Spoken Utterances With Slide Content For Easier Learning With Recorded Lectures Using Structured Support Vector Machine (Svm)

AudioVSR: Enhancing Video Speech Recognition with Audio Data

MaViLS, a Benchmark Dataset for Video-to-Slide Alignment, Assessing Baseline Accuracy with a Multimodal Alignment Algorithm Leveraging Speech, OCR, and Visual Features

Spoken English Intelligibility Remediation with PocketSphinx Alignment and Feature Extraction Improves Substantially over the State of the Art

Multimodal Lecture Presentations Dataset: Understanding Multimodality in Educational Slides

Automatic Error Detection For Unit Selection Speech Synthesis Using Log Likelihood Ratio Based Svm Classifier

Unified Locally Linear Classifiers with Diversity-Promoting Anchor Points

Improved spoken term detection using support vector machines with acoustic and context features from pseudo-relevance feedback

AlignVSR: Audio-Visual Cross-Modal Alignment for Visual Speech Recognition

Aligner²: Enhancing Joint Multiple Intent Detection and Slot Filling Via Adjustive and Forced Cross-Task Alignment

Unsupervised Cross-Modal Alignment of Speech and Text Embedding Spaces

Transferable speech-to-text large language model alignment module

SLIDE: A Framework Integrating Small and Large Language Models for Open-Domain Dialogues Evaluation

Regularized Minimum Class Variance Extreme Learning Machine for Language Recognition

Supervised Spoken Document Summarization Jointly Considering Utterance Importance and Redundancy by Structured Support Vector Machine.

Automated Curriculum Learning for Turn-level Spoken Language Understanding with Weak Supervision

Multimodal Audio-textual Architecture for Robust Spoken Language Understanding

SlideSpeech: A Large-Scale Slide-Enriched Audio-Visual Corpus

ML-LMCL: Mutual Learning and Large-Margin Contrastive Learning for Improving ASR Robustness in Spoken Language Understanding

A Speech Recognition System Based on a Hybrid HMM/SVM Architecture

A Spoken English Teaching System Based on Speech Recognition and Machine Learning