Abstract:We study the problem of procedure planning in instructional videos, which aims to make a goal-oriented sequence of action steps given partial visual state observations. The motivation of this problem is to learn a structured and plannable state and action space. Recent works succeeded in sequence modeling of steps with only sequence-level annotations accessible during training, which overlooked the roles of states in the procedures. In this work, we point out that State CHangEs MAtter (SCHEMA) for procedure planning in instructional videos. We aim to establish a more structured state space by investigating the causal relations between steps and states in procedures. Specifically, we explicitly represent each step as state changes and track the state changes in procedures. For step representation, we leveraged the commonsense knowledge in large language models (LLMs) to describe the state changes of steps via our designed chain-of-thought prompting. For state change tracking, we align visual state observations with language state descriptions via cross-modal contrastive learning, and explicitly model the intermediate states of the procedure using LLM-generated state descriptions. Experiments on CrossTask, COIN, and NIV benchmark datasets demonstrate that our proposed SCHEMA model achieves state-of-the-art performance and obtains explainable visualizations.

The Theory of Problem Schema Acquisition and Its Application in Instruction

Schema induction in problem solving: a multidimensional analysis.

Effects of Problem Schema Knowledge on Problem Processing

Effects and Mechanism of Schema Induction in Analogical Problem Solving

Survey on Research on Algebra Word Problem Schema

Application of Schema Theory to Advanced English Teaching

The Instructional Design of English Major's Reading Class Based on Schema-theoretic Approach

The Construction of Schema in Problem-solving: Study on Children's Categorizing for Word Problems

Using Schema-Based Diagrams to Represent and Solve Word Problems: Relationship between Elementary Preservice Teachers’ Knowledge and Demonstration

Study on Application of Schema Theory in College English Writing Experimental Teaching

On Schema Reading Theory and English Teaching

The Application of Schema Theory in Senior High School English Reading Comprehension Teaching

The Use of Schema Theory in the English Vocabulary Teaching

Theoretical Issues in Analogical Problem Solution:A Survey

Research on the schema theory in the teaching of university English reading

SCHEMA: State CHangEs MAtter for Procedure Planning in Instructional Videos

Schema induction and analogical transfer

A Study of the Employment of Schematic Knowledge in Translation

A Brief Analysis of the Schema Theory on the Teaching of English Reading

Application of Schema Theory in English Reading Comprehension Teaching in Senior High School

The Application of Schema Activation and Creation to the Pre-reading Activities of English Reading Teaching at the Middle School Level