Abstract:Texture, a significant visual attribute in images, has been extensively investigated across various image recognition applications. Convolutional Neural Networks (CNNs), which have been successful in many computer vision tasks, are currently among the best texture analysis approaches. On the other hand, Vision Transformers (ViTs) have been surpassing the performance of CNNs on tasks such as object recognition, causing a paradigm shift in the field. However, ViTs have so far not been scrutinized for texture recognition, hindering a proper appreciation of their potential in this specific setting. For this reason, this work explores various pre-trained ViT architectures when transferred to tasks that rely on textures. We review 21 different ViT variants and perform an extensive evaluation and comparison with CNNs and hand-engineered models on several tasks, such as assessing robustness to changes in texture rotation, scale, and illumination, and distinguishing color textures, material textures, and texture attributes. The goal is to understand the potential and differences among these models when directly applied to texture recognition, using pre-trained ViTs primarily for feature extraction and employing linear classifiers for evaluation. We also evaluate their efficiency, which is one of the main drawbacks in contrast to other methods. Our results show that ViTs generally outperform both CNNs and hand-engineered models, especially when using stronger pre-training and tasks involving in-the-wild textures (images from the internet). We highlight the following promising models: ViT-B with DINO pre-training, BeiTv2, and the Swin architecture, as well as the EfficientFormer as a low-cost alternative. In terms of efficiency, although having a higher number of GFLOPs and parameters, ViT-B and BeiT(v2) can achieve a lower feature extraction time on GPUs compared to ResNet50.

Visualization Comparison of Vision Transformers and Convolutional Neural Networks

Do Vision Transformers See Like Convolutional Neural Networks?

A Comparative Survey of Vision Transformers for Feature Extraction in Texture Analysis

Convolutional Embedding Makes Hierarchical Vision Transformer Stronger

DctViT: Discrete Cosine Transform Meet Vision Transformers

Improving Vision Transformers by Revisiting High-Frequency Components

A Comprehensive Study of Vision Transformers on Dense Prediction Tasks

ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias

CRViT: Vision transformer advanced by causality and inductive bias for image recognition

ViR:the Vision Reservoir

A Unified and Biologically Plausible Relational Graph Representation of Vision Transformers

Vision Conformer: Incorporating Convolutions into Vision Transformer Layers

A Unified and Biologically-Plausible Relational Graph Representation of Vision Transformers

Unified Visual Transformer Compression

Visualizing and Comparing Convolutional Neural Networks

Training Vision Transformers with only 2040 Images.

Pretrained ViTs Yield Versatile Representations For Medical Images

ReViT: Enhancing Vision Transformers Feature Diversity with Attention Residual Connections

DeepViT: Towards Deeper Vision Transformer

CF-ViT: A General Coarse-to-Fine Method for Vision Transformer

Denoising Vision Transformers