Abstract:In this dissertation we present a hierarchical and contextual model for representing image patterns (manmade objects and aerial images) that are highly variant from instance to instance. These types of patterns are difficult to model because objects within the same class may have very different photometric and geometric properties and/or compositions of parts, e.g. teapots may have very different colors, shapes, and locations of their spouts and handles. We hypothesize that these varied visual patterns can be captured by using a novel representation that arranges common primitives of the patterns in a probabilistic hierarchy, thus compactly capturing possible compositional variations, and then enforces contextual constraints on the appearances of the parts, thus modeling the conditional photometric and geometric relationships of the object parts. We combine a Stochastic Context Free Grammar (SCFG), which captures the long-range compositional variations of a pattern, with a Markov Random Field (MRF), which captures the short-range constraints between neighboring pattern primitives, to create our model. We also present a minimax entropy framework for automatically learning which contextual constraints are most relevant for modeling a type of pattern and estimating their parameters. Finally, we present a novel Markov Chain Monte Carlo (MCMC) algorithm called Clustering Cooperative and Competitive Constraints (C 4) for efficiently performing Bayesian inference with our model. C4 is a method for minimizing energy functions defined on graphs that we will use to combine bottom-up and top-down information to find the best interpretation of an image. We show experiments on learning models of a number of manmade object categories and of aerial images and demonstrate that our algorithms automatically learn models that accurately capture the statistical nature of the patterns we are modeling. We also show that our model can be used for inference in new images, allowing it to identify objects in challenging scenarios.

Vocabulary Hierarchy Optimisation Based on Spatial Context and Category Information

Visual Vocabulary Optimization with Spatial Context for Image Annotation and Classification

Nonlinear Discrete Cross-Modal Hashing for Visual-Textual Data

Vocabulary Hierarchy Optimization for Effective and Transferable Retrieval

Improved Visual Vocabularies for Scene Classification of High Resolution Remote Sensing Imagery in Urban Areas

Vocabulary Hierarchy Optimization and Transfer for Scalable Image Search

Visual Words Refining Exploiting Spatial Co-Occurrence Table

Object Categorization Using Context from Multi-Spatial Levels

Incremental Visual Objects Clustering with the Growing Vocabulary Tree

Building Descriptive and Discriminative Visual Codebook for Large-Scale Image Applications.

Integrated Feature Selection and Higher-Order Spatial Feature Extraction for Object Categorization

High-Order Topology Modeling of Visual Words for Image Classification

Algorithms of Vocabulary Tree Hierarchical Semantic Model Image Retrieval

Image retrieval algorithm based on improved visual dictionary model

Constructing Vocabulary Ensembles by Different Clustering Algorithms for Object Categorization

Building Pair-Wise Visual Word Tree For Efficent Image Re-Ranking

Hierarchical Fine-Grained Visual Classification Leveraging Consistent Hierarchical Knowledge

Visual Coding In A Semantic Hierarchy

Hierarchical Multi-Vlad For Image Retrieval

A hierarchical and contextual model for learning and recognizing highly variant visual categories

Evaluating Bag-of-visual-words Representations in Scene Classification