Abstract:Lecture slide presentations, a sequence of pages that contain text and figures accompanied by speech, are constructed and presented carefully in order to optimally transfer knowledge to students. Previous studies in multimedia and psychology attribute the effectiveness of lecture presentations to their multimodal nature. As a step toward developing AI to aid in student learning as intelligent teacher assistants, we introduce the Multimodal Lecture Presentations dataset as a large-scale benchmark testing the capabilities of machine learning models in multimodal understanding of educational content. Our dataset contains aligned slides and spoken language, for 180+ hours of video and 9000+ slides, with 10 lecturers from various subjects (e.g., computer science, dentistry, biology). We introduce two research tasks which are designed as stepping stones towards AI agents that can explain (automatically captioning a lecture presentation) and illustrate (synthesizing visual figures to accompany spoken explanations) educational content. We provide manual annotations to help implement these two research tasks and evaluate state-of-the-art models on them. Comparing baselines and human student performances, we find that current models struggle in (1) weak crossmodal alignment between slides and spoken text, (2) learning novel visual mediums, (3) technical language, and (4) long-range sequences. Towards addressing this issue, we also introduce PolyViLT, a multimodal transformer trained with a multi-instance learning loss that is more effective than current approaches. We conclude by shedding light on the challenges and opportunities in multimodal understanding of educational presentations.

Manim Slides: A Python package for presenting Manim content anywhere

ManimML: Communicating Machine Learning Architectures with Animation

Demonstrating the Potential of Visualization in Education with the Manim Python Library: Examples from Algorithms and Data Structures

Making Slide Shows with Zoomquilts.

OutlineSpark: Igniting AI-powered Presentation Slides Creation from Computational Notebooks through Outlines

Automatically Generating Engaging Presentation Slide Decks

Teaching Physical Based Animation via OpenGL Slides

Narvis: Authoring Narrative Slideshows for Introducing Data Visualization Designs

You Only Write Thrice: Creating Documents, Computational Notebooks and Presentations From a Single Source

ASAP: animation system for agent-based presentations

Deimos: A Grammar of Dynamic Embodied Immersive Visualisation Morphs and Transitions

PPSGen: Learning to Generate Presentation Slides for Academic Papers

Guiding and engaging the audience: Visual metadiscourse in PowerPoint slides of Three Minute Thesis presentations

PPSGen: Learning-Based Presentation Slides Generation for Academic Papers

SlideSpawn: An Automatic Slides Generation System for Research Publications

Slideware2.0: A Prototype of Presentation System by Integrating Web2.0 and Second Screen to Promote Education Communication

Multimodal Lecture Presentations Dataset: Understanding Multimodality in Educational Slides

RIMES: Embedding Interactive Multimedia Exercises in Lecture Videos

Educator's blueprint: A how-to guide for creating high-quality slides

Academic Presenter: a New Storytelling Presentation Software for Academic Purposes

Knowledge-Decks: Automatically Generating Presentation Slide Decks of Visual Analytics Knowledge Discovery Applications