Abstract:Concept Bottleneck Models (CBM) are inherently interpretable models that factor model decisions into human-readable concepts. They allow people to easily understand why a model is failing, a critical feature for high-stakes applications. CBMs require manually specified concepts and often under-perform their black box counterparts, preventing their broad adoption. We address these shortcomings and are first to show how to construct high-performance CBMs without manual specification of similar accuracy to black box models. Our approach, Language Guided Bottlenecks (LaBo), leverages a language model, GPT-3, to define a large space of possible bottlenecks. Given a problem domain, LaBo uses GPT-3 to produce factual sentences about categories to form candidate concepts. LaBo efficiently searches possible bottlenecks through a novel submodular utility that promotes the selection of discriminative and diverse information. Ultimately, GPT-3's sentential concepts can be aligned to images using CLIP, to form a bottleneck layer. Experiments demonstrate that LaBo is a highly effective prior for concepts important to visual recognition. In the evaluation with 11 diverse datasets, LaBo bottlenecks excel at few-shot classification: they are 11.7% more accurate than black box linear probes at 1 shot and comparable with more data. Overall, LaBo demonstrates that inherently interpretable models can be widely applied at similar, or better, performance than black box approaches.

What problem does this paper attempt to address?

The paper attempts to address the problem of improving the performance of interpretable models in image classification tasks, especially in scenarios with limited data. Specifically, the paper proposes a method called LaBo (Language Guided Bottlenecks), which aims to construct high-performance and interpretable image classification models by leveraging large language models (such as GPT-3) to automatically generate concept bottlenecks. ### Main Problems and Goals: 1. **Improve the performance of interpretable models**: Traditional interpretable models (such as Concept Bottleneck Models, CBMs) are easy to understand but usually perform worse than black box models. The goal of the paper is to develop a new method that allows interpretable models to achieve performance comparable to or even better than black box models while maintaining high interpretability. 2. **Reduce manual annotation costs**: Traditional CBMs require manual design and annotation of concepts, which is time-consuming and expensive. LaBo reduces the reliance on manual annotation by automatically generating concepts from large language models. 3. **Adapt to different data scales**: LaBo can significantly improve model performance in scenarios with limited data. As the amount of data increases, LaBo remains competitive. ### Solution: - **Generate concepts using large language models**: LaBo generates descriptive sentences about specific categories by querying large language models like GPT-3. These sentences are further processed into candidate concepts. - **Submodular optimization for concept selection**: A submodular optimization function is used to select the most distinctive and diverse concepts from the candidate pool, forming the concept bottleneck layer. - **Align with vision-language models**: Pre-trained vision-language models (such as CLIP) are used to align text concepts with image features and compute concept scores. - **Linear layer for label prediction**: A linear layer maps the concept scores to the final classification labels. During training, the weight matrix is initialized with language model priors to improve performance in few-shot scenarios. ### Experimental Results: - **Performance on multiple datasets**: LaBo was evaluated on 11 different image classification datasets, including common objects, fine-grained objects, actions, textures, skin tumors, and satellite images. The experimental results show that LaBo significantly outperforms black box linear probes in few-shot scenarios and remains competitive as the data volume increases. - **Comparison with existing methods**: LaBo not only outperforms other interpretable methods (such as PCBM and CompDL) in terms of performance but also achieves performance comparable to black box models without sacrificing interpretability. ### Conclusion: The paper demonstrates the effectiveness of the LaBo method, proving that high-performance and interpretable image classification models can be constructed by automatically generating concepts from large language models. This method not only improves model performance but also reduces the reliance on manual annotation, making it suitable for various image classification tasks.

Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification

Learning Bottleneck Concepts in Image Classification

VLG-CBM: Training Concept Bottleneck Models with Vision-Language Guidance

Interpretable-by-Design Text Understanding with Iteratively Generated Concept Bottleneck

Label-Free Concept Bottleneck Models

Robust and Interpretable Medical Image Classifiers via Concept Bottleneck Models

Sparse Concept Bottleneck Models: Gumbel Tricks in Contrastive Learning

Concept Bottleneck Large Language Models

Coarse-to-Fine Concept Bottleneck Models

Beyond Concept Bottleneck Models: How to Make Black Boxes Intervenable?

Concept Bottleneck Models Without Predefined Concepts

AdaCBM: An Adaptive Concept Bottleneck Model for Explainable and Accurate Diagnosis

Can we Constrain Concept Bottleneck Models to Learn Semantically Meaningful Input Features?

Conceptual Learning via Embedding Approximations for Reinforcing Interpretability and Transparency

Stochastic Concept Bottleneck Models

Reading Is Believing: Revisiting Language Bottleneck Models for Image Classification

Improving Concept Alignment in Vision-Language Concept Bottleneck Models

CLIP-QDA: An Explainable Concept Bottleneck Model

Concept Bottleneck Language Models For protein design

Explain via Any Concept: Concept Bottleneck Model with Open Vocabulary Concepts