Abstract:As natural scenes contain a huge number of motion patterns generated by various stochastic processes, how to represent and model these diverse visual patterns, and how to learn and compute the visual patterns efficiently are fundamental problems in computer vision. In this thesis, we present a unified theory with a generative statistical learning and inference framework to analyze complex visual patterns in both motion and scale-space. The goal of my research is to compute the semantic content of image sequences and representing them with symbolic graphs using generic visual vocabulary learned from natural images. Based on the generic graph representation, four important modeling issues are addressed, which are photometric, geometric, dynamic and topological issues. These four aspects span sub-dimensions in image space. The unified framework aims to answer questions like "What do we perceive when we look at a video sequence?" It tries to identify the perceptually meaningful basic elements and transitions in image sequences, model the elements' attributes and interactions in time-space and scale-space, and infer the hidden dynamic graph structures. We augment the model complexity progressively to learn various complicated visual patterns. We use three challenging examples to illustrate the learning and inference on these four aspects. Example (1): Textured motion modeling. Example (2): Topological changes in complex motion modeling. Example (3): Perceptual transitions in scale-space. The integrated model is generic. It can be broadly applied to many vision applications, such as tracking, video annotation, motion segmentation, document image analysis, surveillance, and automatic cartoon animation generation. The model learning and inference is achieved by employing Markov chain Monte Carlo (MCMC) sampling method.

From Agents to Continuous Change via Aesthetics: Learning Mechanics with Visual Agentbased Computational Modeling

Integrated STEM in Elementary Grades Using Distributed Agent-based Computation

Integrating Agent-based Programming with Elementary Science: The Role of Sociomathematical Norms

Perspectival Computational Thinking for Learning Physics: A Case Study of Collaborative Agent-based Modeling

VISTA: Visual Integrated System for Tailored Automation in Math Problem Generation Using LLM

ChatLogo: A Large Language Model-Driven Hybrid Natural-Programming Language Interface for Agent-based Modeling and Programming

Continual Developmental Neurosimulation Using Embodied Computational Agents

Modeling Complex Motion: Photometric, Geometric, Dynamic, and Topological Aspects

A Modular Vision Language Navigation and Manipulation Framework for Long Horizon Compositional Tasks in Indoor Environment

Thinking beyond chatbots' threat to education: Visualizations to elucidate the writing and coding process

Mobility VLA: Multimodal Instruction Navigation with Long-Context VLMs and Topological Graphs

Visualization and Optimization of Continuum Robots: Integration of Lie Group Kinematics and Evolutionary Algorithm

A First Step in Using Machine Learning Methods to Enhance Interaction Analysis for Embodied Learning Environments

Versatile Motion Langauge Models for Multi-Turn Interactive Agents

Modular Continual Learning in a Unified Visual Environment

Behavioral Analysis of Vision-and-Language Navigation Agents

AVA: Towards Autonomous Visualization Agents through Visual Perception‐Driven Decision‐Making

The Future of Embodied Design for Mathematics Teaching and Learning

Learning Agent-based Modeling with LLM Companions: Experiences of Novices and Experts Using ChatGPT & NetLogo Chat

Playing Modeling Games in the Science Classroom: The Case for Disciplinary Integration