Unified Lexical Representation for Interpretable Visual-Language Alignment

Yifan Li,Yikai Wang,Yanwei Fu,Dongyu Ru,Zheng Zhang,Tong He

2024-07-25

Abstract:Visual-Language Alignment (VLA) has gained a lot of attention since CLIP's groundbreaking work. Although CLIP performs well, the typical direct latent feature alignment lacks clarity in its representation and similarity scores. On the other hand, lexical representation, a vector whose element represents the similarity between the sample and a word from the vocabulary, is a natural sparse representation and interpretable, providing exact matches for individual words. However, lexical representations is difficult to learn due to no ground-truth supervision and false-discovery issues, and thus requires complex design to train effectively. In this paper, we introduce LexVLA, a more interpretable VLA framework by learning a unified lexical representation for both modalities without complex design. We use DINOv2 as our visual model for its local-inclined features and Llama 2, a generative language model, to leverage its in-context lexical prediction ability. To avoid the false discovery, we propose an overuse penalty to refrain the lexical representation from falsely frequently activating meaningless words. We demonstrate that these two pre-trained uni-modal models can be well-aligned by fine-tuning on modest multi-modal dataset and avoid intricate training configurations. On cross-modal retrieval benchmarks, LexVLA, trained on the CC-12M multi-modal dataset, outperforms baselines fine-tuned on larger datasets (e.g., YFCC15M) and those trained from scratch on even bigger datasets (e.g., 1.1B data, including CC-12M). We conduct extensive experiments to analyze LexVLA.

Computer Vision and Pattern Recognition,Artificial Intelligence,Computation and Language,Machine Learning

What problem does this paper attempt to address?

### The Problem Addressed by the Paper

This paper aims to address the interpretability issue in Visual-Language Alignment (VLA). Although existing VLA models like CLIP perform excellently in terms of performance, their direct latent feature alignment lacks clear representation and similarity scoring. On the other hand, lexical representation is a naturally sparse representation that can provide exact matches for individual words, thereby improving interpretability. However, learning lexical representation faces challenges such as unsupervised signals and false discoveries, requiring complex training designs.

To this end, this paper proposes LexVLA, which achieves a more interpretable VLA framework by learning unified lexical representations. Specifically, LexVLA leverages pre-trained unimodal models (such as DINOv2 and Llama 2) and introduces an overuse penalty to avoid false activation of irrelevant terms, thereby achieving effective alignment on a small amount of multimodal data. Experimental results show that LexVLA outperforms baseline models in cross-modal retrieval tasks, especially when trained on smaller-scale datasets. Additionally, LexVLA introduces a new evaluation metric, PatchDis, to measure the interpretability of visual lexical representations at the patch level.