Abstract:Compositional zero-shot learning (CZSL) aims to recognize novel compositions of attributes and objects learned from seen compositions. Previous works disentangle attribute and object by extracting shared and exclusive parts between image pairs sharing the same attribute (object), as well as aligning them with pretrained word embeddings to improve unseen attribute-object recognition. Despite the significant achievements of existing efforts, they are hampered by three limitations: (1) the efficacy of disentanglement is compromised due to the influence of the background and the intricate entanglement of attribute with object in the same parts. (2) existing word embeddings fail to capture complex multimodal semantic information. (3) overconfidence exhibited by existing models in seen compositions hinders their generalization to novel compositions. Being aware of these, we propose a novel framework named Multimodal Large Language Model (MLLM) embeddings and attribute smoothing guided disentanglement (TRIDENT) for CZSL. First, we leverage feature adaptive aggregation modules to mitigate the impact of background, and utilize learnable condition masks to capture multigranularity features for disentanglement. Then, the last hidden states of MLLM are employed as word embeddings for their superior representation capabilities. Moreover, we propose attribute smoothing with auxiliary attributes generated by Large Language Model (LLM) for seen compositions, addressing the issue of overconfidence by encouraging the model to learn more attributes in one given composition. Extensive experiments demonstrate that TRIDENT achieves state-of-the-art performance on three benchmarks.

Kent feature embedding for classification of compositional data with zeros

A classification framework for multivariate compositional data with Dirichlet feature embedding

Learning Exemplar-Represented Manifolds in Latent Space for Classification.

Decompose Learning: Combine Feature Extraction and Classification

Learning to Embed Seen/Unseen Compositions based on Graph Networks

Dirichlet component analysis: feature extraction for compositional data

Dirichlet composition distribution for compositional data with zero components: An application to fluorescence in situ hybridization (FISH) detection of chromosome

Composite classification methods on composition identification

Disentangling Before Composing: Learning Invariant Disentangled Features for Compositional Zero-Shot Learning

Compositional Zero-shot Learning Via Progressive Language-based Observations

Cross-scene Hyperspectral Image Classification Based on DWT and Manifold-Constrained Subspace Learning

Cross-composition Feature Disentanglement for Compositional Zero-shot Learning

Multiple linear regression modeling for compositional data

Decomposition and Extraction: A New Framework for Visual Classification

A New Method to Capturing Compositional Knowledge in Linguistic Space

Word Embedding Composition for Data Imbalances in Sentiment and Emotion Classification

A causal view of compositional zero-shot recognition

Principal component analysis for compositional data vectors

Leveraging MLLM Embeddings and Attribute Smoothing for Compositional Zero-Shot Learning

Mutual Balancing in State-Object Components for Compositional Zero-Shot Learning.

A Zero-Inflated Logistic Normal Multinomial Model for Extracting Microbial Compositions