Abstract:Visual Question Answering is a challenging task, as it requires seamless interaction between perceptual, linguistic, and background knowledge systems. While the recent progress of visual and natural language models like BLIP has led to improved performance on this task, we lack understanding of the ability of such models to perform on different kinds of questions and reasoning types. As our initial analysis of BLIP-family models revealed difficulty with answering fine-detail questions, we investigate the following question: Can visual cropping be employed to improve the performance of state-of-the-art visual question answering models on fine-detail questions? Given the recent success of the BLIP-family models, we study a zero-shot and a fine-tuned BLIP model. We define three controlled subsets of the popular VQA-v2 benchmark to measure whether cropping can help model performance. Besides human cropping, we devise two automatic cropping strategies based on multi-modal embedding by CLIP and BLIP visual QA model gradients. Our experiments demonstrate that the performance of BLIP model variants can be significantly improved through human cropping, and automatic cropping methods can produce comparable benefits. A deeper dive into our findings indicates that the performance enhancement is more pronounced in zero-shot models than in fine-tuned models and more salient with smaller bounding boxes than larger ones. We perform case studies to connect quantitative differences with qualitative observations across question types and datasets. Finally, we see that the cropping enhancement is robust, as we gain an improvement of 4.59% (absolute) in the general VQA-random task by simply inputting a concatenation of the original and gradient-based cropped images. We make our code available to facilitate further innovation on visual cropping methods for question answering.

ParamCrop: Parametric Cubic Cropping for Video Contrastive Learning

SemanticCrop: Boosting Contrastive Learning Via Semantic-Cropped Views.

Enhancing Contrastive Learning Inspired by the Philosophy of "The Blind Men and the Elephant"

Towards perceptual video cropping with curve fitting

Crafting Better Contrastive Views for Siamese Representation Learning

Parametric Augmentation for Time Series Contrastive Learning

Discriminate Clearer to Rank Better: Image Cropping by Amplifying View-Wise Differences.

Compressed Video Contrastive Learning.

CCL4Rec: Contrast over Contrastive Learning for Micro-video Recommendation

Enhancing Contrastive Learning with Efficient Combinatorial Positive Pairing

Adaptive Multi-head Contrastive Learning

Video Contrastive Learning with Global Context

Image Cropping with Spatial-Aware Feature and Rank Consistency

RegionCL: Can Simple Region Swapping Contribute to Contrastive Learning?

Contrastive Multiview Coding

Find Beauty in the Rare: Contrastive Composition Feature Clustering for Nontrivial Cropping Box Regression

CLSA: A Contrastive Learning Framework with Selective Aggregation for Video Rescaling

Unsupervised Action Localization Crop in Video Retargeting for 3D ConvNets

MetAug: Contrastive Learning via Meta Feature Augmentation

Using Visual Cropping to Enhance Fine-Detail Question Answering of BLIP-Family Models

What makes for good views for contrastive learning