Abstract:Concept Bottleneck Models (CBMs) provide interpretable prediction by introducing an intermediate Concept Bottleneck Layer (CBL), which encodes human-understandable concepts to explain models' decision. Recent works proposed to utilize Large Language Models (LLMs) and pre-trained Vision-Language Models (VLMs) to automate the training of CBMs, making it more scalable and automated. However, existing approaches still fall short in two aspects: First, the concepts predicted by CBL often mismatch the input image, raising doubts about the faithfulness of interpretation. Second, it has been shown that concept values encode unintended information: even a set of random concepts could achieve comparable test accuracy to state-of-the-art CBMs. To address these critical limitations, in this work, we propose a novel framework called Vision-Language-Guided Concept Bottleneck Model (VLG-CBM) to enable faithful interpretability with the benefits of boosted performance. Our method leverages off-the-shelf open-domain grounded object detectors to provide visually grounded concept annotation, which largely enhances the faithfulness of concept prediction while further improving the model performance. In addition, we propose a new metric called Number of Effective Concepts (NEC) to control the information leakage and provide better interpretability. Extensive evaluations across five standard benchmarks show that our method, VLG-CBM, outperforms existing methods by at least 4.27% and up to 51.09% on accuracy at NEC=5, and by at least 0.45% and up to 29.78% on average accuracy across different NECs, while preserves both faithfulness and interpretability of the learned concepts as demonstrated in extensive experiments.

Concept Bottleneck Large Language Models

Crafting Large Language Models for Enhanced Interpretability

Label-Free Concept Bottleneck Models

Bayesian Concept Bottleneck Models with LLM Priors

VLG-CBM: Training Concept Bottleneck Models with Vision-Language Guidance

Concept Bottleneck Language Models For protein design

Towards Concept-Aware Large Language Models

Concept-Oriented Deep Learning with Large Language Models

Large Concept Models: Language Modeling in a Sentence Representation Space

Aligning Visual and Semantic Interpretability through Visually Grounded Concept Bottleneck Models

Interpretable-by-Design Text Understanding with Iteratively Generated Concept Bottleneck

Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification

Cabbage Sweeter than Cake? Analysing the Potential of Large Language Models for Learning Conceptual Spaces

Sparse Linear Concept Discovery Models

Exploring Concept Depth: How Large Language Models Acquire Knowledge at Different Layers?

Improving Concept Alignment in Vision-Language Concept Bottleneck Models

Coarse-to-Fine Concept Bottleneck Models

Conceptual Learning via Embedding Approximations for Reinforcing Interpretability and Transparency

Exploring Information Processing in Large Language Models: Insights from Information Bottleneck Theory

Sparsity-Guided Holistic Explanation for LLMs with Interpretable Inference-Time Intervention

Supervised Knowledge Makes Large Language Models Better In-context Learners