TutoAI: A Cross-domain Framework for AI-assisted Mixed-media Tutorial Creation on Physical Tasks
Yuexi Chen,Vlad I. Morariu,Anh Truong,Zhicheng Liu
2024-03-13
Abstract:Mixed-media tutorials, which integrate videos, images, text, and diagrams to
teach procedural skills, offer more browsable alternatives than timeline-based
videos. However, manually creating such tutorials is tedious, and existing
automated solutions are often restricted to a particular domain. While AI
models hold promise, it is unclear how to effectively harness their powers,
given the multi-modal data involved and the vast landscape of models. We
present TutoAI, a cross-domain framework for AI-assisted mixed-media tutorial
creation on physical tasks. First, we distill common tutorial components by
surveying existing work; then, we present an approach to identify, assemble,
and evaluate AI models for component extraction; finally, we propose guidelines
for designing user interfaces (UI) that support tutorial creation based on
AI-generated components. We show that TutoAI has achieved higher or similar
quality compared to a baseline model in preliminary user studies.
Artificial Intelligence,Human-Computer Interaction,Machine Learning