Abstract:In this dissertation we present a hierarchical and contextual model for representing image patterns (manmade objects and aerial images) that are highly variant from instance to instance. These types of patterns are difficult to model because objects within the same class may have very different photometric and geometric properties and/or compositions of parts, e.g. teapots may have very different colors, shapes, and locations of their spouts and handles. We hypothesize that these varied visual patterns can be captured by using a novel representation that arranges common primitives of the patterns in a probabilistic hierarchy, thus compactly capturing possible compositional variations, and then enforces contextual constraints on the appearances of the parts, thus modeling the conditional photometric and geometric relationships of the object parts. We combine a Stochastic Context Free Grammar (SCFG), which captures the long-range compositional variations of a pattern, with a Markov Random Field (MRF), which captures the short-range constraints between neighboring pattern primitives, to create our model. We also present a minimax entropy framework for automatically learning which contextual constraints are most relevant for modeling a type of pattern and estimating their parameters. Finally, we present a novel Markov Chain Monte Carlo (MCMC) algorithm called Clustering Cooperative and Competitive Constraints (C 4) for efficiently performing Bayesian inference with our model. C4 is a method for minimizing energy functions defined on graphs that we will use to combine bottom-up and top-down information to find the best interpretation of an image. We show experiments on learning models of a number of manmade object categories and of aerial images and demonstrate that our algorithms automatically learn models that accurately capture the statistical nature of the patterns we are modeling. We also show that our model can be used for inference in new images, allowing it to identify objects in challenging scenarios.

Latent Topic Visual Language Model for Object Categorization.

Scale-invariant visual language modeling for object categorization

Visual language modeling for image classification.

A hierarchical and contextual model for learning and recognizing highly variant visual categories

Small variance among different subcategories Artic Tern Caspian Tern Large variance in the same subcategory Common Tern Salty Backed Gull Salty Backed Gull Salty Backed Gull

Latent Dirichlet classification:A new method for object detection

A Novel Context-Aware Topic Model For Category Discovery In Natural Scenes

Which Looks Like Which: Exploring Inter-Class Relationships In Fine-Grained Visual Categorization

Unveiling the Mystery of Visual Attributes of Concrete and Abstract Concepts: Variability, Nearest Neighbors, and Challenging Categories

Structural Context for Object Categorization.

Semi-supervised topic modeling for image annotation.

Context-LGM: Leveraging Object-Context Relation for Context-Aware Object Recognition

Investigating the Role of Attribute Context in Vision-Language Models for Object Recognition and Detection

Multi-modal Auto-regressive Modeling via Visual Words

Concept Drift and Long-Tailed Distribution in Fine-Grained Visual Categorization: Benchmark and Method

Visually-Augmented Language Modeling

Localization vs. Semantics: Visual Representations in Unimodal and Multimodal Models

Learning Compositional Models for Object Categories from Small Sample Sets

Topic models for image annotation and text illustration

Multimodal visual dictionary learning via heterogeneous latent semantic sparse coding

Understanding bag-of-words model: a statistical framework