Abstract:Recent progress in the interdisciplinary studies of computer vision (CV) and natural language processing (NLP) has enabled the development of intelligent systems that can describe what they see and answer questions accordingly. However, despite showing usefulness in performing these vision-language tasks, existing methods still struggle in understanding real-life problems (i.e., how to do something) and suggesting step-by-step guidance to solve them. With an overarching goal of developing intelligent systems to assist humans in various daily activities, we propose VisualHow, a free-form and open-ended research that focuses on understanding a real-life problem and deriving its solution by incorporating key components across multiple modalities. We develop a new dataset with 20,028 real-life problems and 102,933 steps that constitute their solutions, where each step consists of both a visual illustration and a textual description that guide the problem solving. To establish better understanding of problems and solutions, we also provide annotations of multimodal attention that localizes important components across modalities and solution graphs that encapsulate different steps in structured representations. These data and annotations enable a family of new vision-language tasks that solve real-life problems. Through extensive experiments with representative models, we demonstrate their effectiveness on training and testing models for the new tasks, and there is significant scope for improvement by learning effective attention mechanisms. Our dataset and models are available at https://github.com/formidify/VisualHow.

Learning from the Tangram to Solve Mini Visual Tasks.

Learning from the Tangram to Solve Mini Visual Tasks

Learning reconfigurable scene representation by tangram model

Masked Visual-Tactile Pre-training for Robot Manipulation

A Reconfigurable Tangram Model for Scene Representation and Categorization

Jigsaw-ViT: Learning Jigsaw Puzzles in Vision Transformer

Adept: Annotation-denoising Auxiliary Tasks with Discrete Cosine Transform Map and Keypoint for Human-Centric Pretraining

TangramTheatre: presenting children's creation on multimodal tabletops.

CogCoM: Train Large Vision-Language Models Diving into Details through Chain of Manipulations

VisualHow: Multimodal Problem Solving

PuzzleTuning: Explicitly Bridge Pathological and Natural Image with Puzzles

Multimodal Self-Instruct: Synthetic Abstract Image and Visual Reasoning Instruction Using Language Model

Integrating Text and Image Pre-training for Multi-modal Algorithmic Reasoning

Solving Masked Jigsaw Puzzles with Diffusion Vision Transformers

Diffusion-based Visual Anagram as Multi-task Learning

SGEITL: Scene Graph Enhanced Image-Text Learning for Visual Commonsense Reasoning

PuzzleVQA: Diagnosing Multimodal Reasoning Challenges of Language Models with Abstract Visual Patterns

Exploring Visual Pre-training for Robot Manipulation: Datasets, Models and Methods

IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning

Learning Visual Symbols for Parsing Human Poses in Images

ViTCN: Vision Transformer Contrastive Network For Reasoning