Understanding Test-Time Augmentation

Masanari Kimura

2024-02-10

Abstract:Test-Time Augmentation (TTA) is a very powerful heuristic that takes advantage of data augmentation during testing to produce averaged output. Despite the experimental effectiveness of TTA, there is insufficient discussion of its theoretical aspects. In this paper, we aim to give theoretical guarantees for TTA and clarify its behavior.

Machine Learning

What problem does this paper attempt to address?

Test-Time Augmentation Meets Variational Bayes

Masanari Kimura,Howard Bondell

2024-09-19

Abstract:Data augmentation is known to contribute significantly to the robustness of machine learning models. In most instances, data augmentation is utilized during the training phase. Test-Time Augmentation (TTA) is a technique that instead leverages these data augmentations during the testing phase to achieve robust predictions. More precisely, TTA averages the predictions of multiple data augmentations of an instance to produce a final prediction. Although the effectiveness of TTA has been empirically reported, it can be expected that the predictive performance achieved will depend on the set of data augmentation methods used during testing. In particular, the data augmentation methods applied should make different contributions to performance. That is, it is anticipated that there may be differing degrees of contribution in the set of data augmentation methods used for TTA, and these could have a negative impact on prediction performance. In this study, we consider a weighted version of the TTA based on the contribution of each data augmentation. Some variants of TTA can be regarded as considering the problem of determining the appropriate weighting. We demonstrate that the determination of the coefficients of this weighted TTA can be formalized in a variational Bayesian framework. We also show that optimizing the weights to maximize the marginal log-likelihood suppresses candidates of unwanted data augmentations at the test phase.

Machine Learning,Artificial Intelligence
Improved Text Classification via Test-Time Augmentation

Helen Lu,Divya Shanmugam,Harini Suresh,John Guttag

DOI: https://doi.org/10.48550/arXiv.2206.13607

2022-06-28

Abstract:Test-time augmentation -- the aggregation of predictions across transformed examples of test inputs -- is an established technique to improve the performance of image classification models. Importantly, TTA can be used to improve model performance post-hoc, without additional training. Although test-time augmentation (TTA) can be applied to any data modality, it has seen limited adoption in NLP due in part to the difficulty of identifying label-preserving transformations. In this paper, we present augmentation policies that yield significant accuracy improvements with language models. A key finding is that augmentation policy design -- for instance, the number of samples generated from a single, non-deterministic augmentation -- has a considerable impact on the benefit of TTA. Experiments across a binary classification task and dataset show that test-time augmentation can deliver consistent improvements over current state-of-the-art approaches.

Machine Learning,Computation and Language
Feature Augmentation based Test-Time Adaptation

Younggeol Cho,Youngrae Kim,Junho Yoon,Seunghoon Hong,Dongman Lee

2024-10-18

Abstract:Test-time adaptation (TTA) allows a model to be adapted to an unseen domain without accessing the source data. Due to the nature of practical environments, TTA has a limited amount of data for adaptation. Recent TTA methods further restrict this by filtering input data for reliability, making the effective data size even smaller and limiting adaptation potential. To address this issue, We propose Feature Augmentation based Test-time Adaptation (FATA), a simple method that fully utilizes the limited amount of input data through feature augmentation. FATA employs Normalization Perturbation to augment features and adapts the model using the FATA loss, which makes the outputs of the augmented and original features similar. FATA is model-agnostic and can be seamlessly integrated into existing models without altering the model architecture. We demonstrate the effectiveness of FATA on various models and scenarios on ImageNet-C and Office-Home, validating its superiority in diverse real-world conditions.

Computer Vision and Pattern Recognition
STTA: enhanced text classification via selective test-time augmentation

Haoyu Xiong,Xinchun Zhang,Leixin Yang,Yu Xiang,Yaping Zhang

DOI: https://doi.org/10.7717/peerj-cs.1757

2023-12-20

PeerJ Computer Science

Abstract:Test-time augmentation (TTA) is a well-established technique that involves aggregating transformed examples of test inputs during the inference stage. The goal is to enhance model performance and reduce the uncertainty of predictions. Despite its advantages of not requiring additional training or hyperparameter tuning, and being applicable to any existing model, TTA is still in its early stages in the field of NLP. This is partly due to the difficulty of discerning the contribution of different transformed samples, which can negatively impact predictions. In order to address these issues, we propose Selective Test-Time Augmentation, called STTA, which aims to select the most beneficial transformed samples for aggregation by identifying reliable samples. Furthermore, we analyze and empirically verify why TTA is sensitive to some text data augmentation methods and reveal why some data augmentation methods lead to erroneous predictions. Through extensive experiments, we demonstrate that STTA is a simple and effective method that can produce promising results in various text classification tasks.

computer science, information systems, artificial intelligence, theory & methods
Better Aggregation in Test-Time Augmentation

Divya Shanmugam,Davis Blalock,Guha Balakrishnan,John Guttag

DOI: https://doi.org/10.48550/arXiv.2011.11156

2021-10-12

Abstract:Test-time augmentation -- the aggregation of predictions across transformed versions of a test input -- is a common practice in image classification. Traditionally, predictions are combined using a simple average. In this paper, we present 1) experimental analyses that shed light on cases in which the simple average is suboptimal and 2) a method to address these shortcomings. A key finding is that even when test-time augmentation produces a net improvement in accuracy, it can change many correct predictions into incorrect predictions. We delve into when and why test-time augmentation changes a prediction from being correct to incorrect and vice versa. Building on these insights, we present a learning-based method for aggregating test-time augmentations. Experiments across a diverse set of models, datasets, and augmentations show that our method delivers consistent improvements over existing approaches.

Computer Vision and Pattern Recognition
Boosting Anomaly Detection Using Unsupervised Diverse Test-Time Augmentation

Seffi Cohen,Niv Goldshlager,Lior Rokach,Bracha Shapira

DOI: https://doi.org/10.1016/j.ins.2023.01.081

IF: 8.1

2023-01-21

Information Sciences

Abstract:Anomaly detection is a well-known task that involves the identification of abnormal events that occur relatively infrequently. Methods for improving anomaly detection performance have been widely studied. However, no studies utilizing test-time augmentation (TTA) for anomaly detection in tabular data have been performed. TTA involves aggregating the predictions of several synthetic versions of a given test sample; TTA produces different points of view for a specific test instance and might decrease its prediction bias. We propose the Test-Time Augmentation for anomaly Detection (TTAD) technique, a TTA-based method aimed at improving anomaly detection performance. TTAD augments a test instance based on its nearest neighbors; various methods, including the k-Means centroid and SMOTE methods, are used to produce the augmentations. Our technique utilizes a Siamese network to learn an advanced distance metric when retrieving a test instance's neighbors. Our experiments show that using our TTA technique significantly improves the performance of anomaly detection algorithms, as evidenced by the higher AUC results achieved on all datasets evaluated. Specifically, we observed average improvements of 0.037 AUC (3.7%) using Autoencoder, 0.016 AUC (1.6%) using OC-SVM, and 0.023 AUC (2.3%) using LOF.
Test-Time Augmentation for 3D Point Cloud Classification and Segmentation

Tuan-Anh Vu,Srinjay Sarkar,Zhiyuan Zhang,Binh-Son Hua,Sai-Kit Yeung

2023-11-22

Abstract:Data augmentation is a powerful technique to enhance the performance of a deep learning task but has received less attention in 3D deep learning. It is well known that when 3D shapes are sparsely represented with low point density, the performance of the downstream tasks drops significantly. This work explores test-time augmentation (TTA) for 3D point clouds. We are inspired by the recent revolution of learning implicit representation and point cloud upsampling, which can produce high-quality 3D surface reconstruction and proximity-to-surface, respectively. Our idea is to leverage the implicit field reconstruction or point cloud upsampling techniques as a systematic way to augment point cloud data. Mainly, we test both strategies by sampling points from the reconstructed results and using the sampled point cloud as test-time augmented data. We show that both strategies are effective in improving accuracy. We observed that point cloud upsampling for test-time augmentation can lead to more significant performance improvement on downstream tasks such as object classification and segmentation on the ModelNet40, ShapeNet, ScanObjectNN, and SemanticKITTI datasets, especially for sparse point clouds.

Computer Vision and Pattern Recognition
Test-Time Augmentation for Traveling Salesperson Problem

Ryo Ishiyama,Takahiro Shirakawa,Seiichi Uchida,Shinnosuke Matsuo

2024-05-08

Abstract:We propose Test-Time Augmentation (TTA) as an effective technique for addressing combinatorial optimization problems, including the Traveling Salesperson Problem. In general, deep learning models possessing the property of invariance, where the output is uniquely determined regardless of the node indices, have been proposed to learn graph structures efficiently. In contrast, we interpret the permutation of node indices, which exchanges the elements of the distance matrix, as a TTA scheme. The results demonstrate that our method is capable of obtaining shorter solutions than the latest models. Furthermore, we show that the probability of finding a solution closer to an exact solution increases depending on the augmentation size.

Machine Learning,Artificial Intelligence
Anomaly Detection with Test Time Augmentation and Consistency Evaluation

Haowei He,Jiaye Teng,Yang Yuan

DOI: https://doi.org/10.48550/arxiv.2206.02345

2022-01-01

Abstract: Deep neural networks are known to be vulnerable to unseen data: they may wrongly assign high confidence stcores to out-distribuion samples. Recent works try to solve the problem using representation learning methods and specific metrics. In this paper, we propose a simple, yet effective post-hoc anomaly detection algorithm named Test Time Augmentation Anomaly Detection (TTA-AD), inspired by a novel observation. Specifically, we observe that in-distribution data enjoy more consistent predictions for its original and augmented versions on a trained network than out-distribution data, which separates in-distribution and out-distribution samples. Experiments on various high-resolution image benchmark datasets demonstrate that TTA-AD achieves comparable or better detection performance under dataset-vs-dataset anomaly detection settings with a 60%~90\% running time reduction of existing classifier-based algorithms. We provide empirical verification that the key to TTA-AD lies in the remaining classes between augmented features, which has long been partially ignored by previous works. Additionally, we use RUNS as a surrogate to analyze our algorithm theoretically.
Active Test-Time Adaptation: Theoretical Analyses and An Algorithm

Shurui Gui,Xiner Li,Shuiwang Ji

2024-04-08

Abstract:Test-time adaptation (TTA) addresses distribution shifts for streaming test data in unsupervised settings. Currently, most TTA methods can only deal with minor shifts and rely heavily on heuristic and empirical studies.

Machine Learning,Artificial Intelligence
Efficient Improvement of Classification Accuracy via Selective Test-Time Augmentation

Jongwook Son,Seokho Kang

DOI: https://doi.org/10.1016/j.ins.2023.119148

IF: 8.1

2023-05-18

Information Sciences

Abstract:Test-time augmentation (TTA) is typically used in image classification tasks to improve the classification accuracy of convolutional neural networks (CNNs). In the inference phase, TTA aggregates the predictions of a CNN over multiple random transformations of a query image to obtain the final prediction. However, TTA incurs a high computational cost owing to its effectiveness. In this study, we propose a selective-TTA method for reducing computational costs while maintaining the high classification accuracy. Given a query image, we quantify the predictive uncertainty of the CNN on the image to determine whether TTA should be performed. We perform TTA to obtain an enhanced prediction only when the uncertainty is high. Experimental results on benchmark datasets demonstrated that the proposed method significantly reduced average computational costs while attaining comparable or higher classification accuracy than conventional full-TTA. When the proportion of performing TTA was set to 10%, the computational cost was reduced by 90% with no performance degradation on most datasets.

computer science, information systems
Bag of Tricks for Fully Test-Time Adaptation

Saypraseuth Mounsaveng,Florent Chiaroni,Malik Boudiaf,Marco Pedersoli,Ismail Ben Ayed

2023-11-09

Abstract:Fully Test-Time Adaptation (TTA), which aims at adapting models to data drifts, has recently attracted wide interest. Numerous tricks and techniques have been proposed to ensure robust learning on arbitrary streams of unlabeled data. However, assessing the true impact of each individual technique and obtaining a fair comparison still constitutes a significant challenge. To help consolidate the community's knowledge, we present a categorization of selected orthogonal TTA techniques, including small batch normalization, stream rebalancing, reliable sample selection, and network confidence calibration. We meticulously dissect the effect of each approach on different scenarios of interest. Through our analysis, we shed light on trade-offs induced by those techniques between accuracy, the computational power required, and model complexity. We also uncover the synergy that arises when combining techniques and are able to establish new state-of-the-art results.

Machine Learning,Computer Vision and Pattern Recognition
On Pitfalls of Test-Time Adaptation

Hao Zhao,Yuejiang Liu,Alexandre Alahi,Tao Lin

2023-06-06

Abstract:Test-Time Adaptation (TTA) has recently emerged as a promising approach for tackling the robustness challenge under distribution shifts. However, the lack of consistent settings and systematic studies in prior literature hinders thorough assessments of existing methods. To address this issue, we present TTAB, a test-time adaptation benchmark that encompasses ten state-of-the-art algorithms, a diverse array of distribution shifts, and two evaluation protocols. Through extensive experiments, our benchmark reveals three common pitfalls in prior efforts. First, selecting appropriate hyper-parameters, especially for model selection, is exceedingly difficult due to online batch dependency. Second, the effectiveness of TTA varies greatly depending on the quality and properties of the model being adapted. Third, even under optimal algorithmic conditions, none of the existing methods are capable of addressing all common types of distribution shifts. Our findings underscore the need for future research in the field to conduct rigorous evaluations on a broader set of models and shifts, and to re-examine the assumptions behind the empirical success of TTA. Our code is available at \url{<a class="link-external link-https" href="https://github.com/lins-lab/ttab" rel="external noopener nofollow">this https URL</a>}.

Machine Learning,Artificial Intelligence
Confidence-based and sample-reweighted test-time adaptation

Hao Yang,Min Wang,Zhengfei Yu,Hang Zhang,Jinshen Jiang,Yun Zhou

DOI: https://doi.org/10.1016/j.knosys.2023.111164

IF: 8.139

2023-11-08

Knowledge-Based Systems

Abstract:Recently, Test Time Adaptation (TTA) has emerged as a solution to real-world challenges posed by inconsistencies between training and testing distributions. This technique refines a pretrained model for a target domain using only unlabeled test data. However, most existing TTA methods assign equal weights to all samples and are dominated by entropy minimization for adaptation. In this paper, we propose a novel confidence-based optimization strategy and theoretically show that the proposed method tends to yield larger gradients than entropy-based methods and has the potential to enhance performance. Moreover, we show that the importance of samples is frequently underestimated and propose a novel truncation function that assigns an adaptive weight to each sample. The proposed method, named CSTTA, consists of a novel confidence-based optimization strategy and sample-reweighted strategy. It aims to better utilize the sample information for quicker adaptation to new scenarios. Extensive experiments on three digital datasets (CIFAR10-C, CIFAR100-C, and ImageNet-C) and a real-world dataset (ImageNet-3DCC) demonstrate the effectiveness of the proposed method.

computer science, artificial intelligence
Intelligent Multi-View Test Time Augmentation

Efe Ozturk,Mohit Prabhushankar,Ghassan AlRegib

2024-06-13

Abstract:In this study, we introduce an intelligent Test Time Augmentation (TTA) algorithm designed to enhance the robustness and accuracy of image classification models against viewpoint variations. Unlike traditional TTA methods that indiscriminately apply augmentations, our approach intelligently selects optimal augmentations based on predictive uncertainty metrics. This selection is achieved via a two-stage process: the first stage identifies the optimal augmentation for each class by evaluating uncertainty levels, while the second stage implements an uncertainty threshold to determine when applying TTA would be advantageous. This methodological advancement ensures that augmentations contribute to classification more effectively than a uniform application across the dataset. Experimental validation across several datasets and neural network architectures validates our approach, yielding an average accuracy improvement of 1.73% over methods that use single-view images. This research underscores the potential of adaptive, uncertainty-aware TTA in improving the robustness of image classification in the presence of viewpoint variations, paving the way for further exploration into intelligent augmentation strategies.

Image and Video Processing,Machine Learning
Generalized Robust Test-Time Adaptation in Continuous Dynamic Scenarios

Shuang Li,Longhui Yuan,Binhui Xie,Tao Yang

DOI: https://doi.org/10.48550/arxiv.2310.04714

2023-01-01

Abstract: Test-time adaptation (TTA) adapts the pre-trained models to test distributions during the inference phase exclusively employing unlabeled test data streams, which holds great value for the deployment of models in real-world applications. Numerous studies have achieved promising performance on simplistic test streams, characterized by independently and uniformly sampled test data originating from a fixed target data distribution. However, these methods frequently prove ineffective in practical scenarios, where both continual covariate shift and continual label shift occur simultaneously, i.e., data and label distributions change concurrently and continually over time. In this study, a more challenging Practical Test-Time Adaptation (PTTA) setup is introduced, which takes into account the concurrent presence of continual covariate shift and continual label shift, and we propose a Generalized Robust Test-Time Adaptation (GRoTTA) method to effectively address the difficult problem. We start by steadily adapting the model through Robust Parameter Adaptation to make balanced predictions for test samples. To be specific, firstly, the effects of continual label shift are eliminated by enforcing the model to learn from a uniform label distribution and introducing recalibration of batch normalization to ensure stability. Secondly, the continual covariate shift is alleviated by employing a source knowledge regularization with the teacher-student model to update parameters. Considering the potential information in the test stream, we further refine the balanced predictions by Bias-Guided Output Adaptation, which exploits latent structure in the feature space and is adaptive to the imbalanced label distribution. Extensive experiments demonstrate GRoTTA outperforms the existing competitors by a large margin under PTTA setting, rendering it highly conducive for adoption in real-world applications.
MITA: Bridging the Gap between Model and Data for Test-time Adaptation

Yige Yuan,Bingbing Xu,Teng Xiao,Liang Hou,Fei Sun,Huawei Shen,Xueqi Cheng

2024-10-12

Abstract:Test-Time Adaptation (TTA) has emerged as a promising paradigm for enhancing the generalizability of models. However, existing mainstream TTA methods, predominantly operating at batch level, often exhibit suboptimal performance in complex real-world scenarios, particularly when confronting outliers or mixed distributions. This phenomenon stems from a pronounced over-reliance on statistical patterns over the distinct characteristics of individual instances, resulting in a divergence between the distribution captured by the model and data characteristics. To address this challenge, we propose Meet-In-The-Middle based Test-Time Adaptation ($\textbf{MITA}$), which introduces energy-based optimization to encourage mutual adaptation of the model and data from opposing directions, thereby meeting in the middle. MITA pioneers a significant departure from traditional approaches that focus solely on aligning the model to the data, facilitating a more effective bridging of the gap between model's distribution and data characteristics. Comprehensive experiments with MITA across three distinct scenarios (Outlier, Mixture, and Pure) demonstrate its superior performance over SOTA methods, highlighting its potential to significantly enhance generalizability in practical applications.

Machine Learning,Computer Vision and Pattern Recognition
Diffusion-Enhanced Test-time Adaptation with Text and Image Augmentation

Chun-Mei Feng,Yuanyang He,Jian Zou,Salman Khan,Huan Xiong,Zhen Li,Wangmeng Zuo,Rick Siow Mong Goh,Yong Liu

2024-12-13

Abstract:Existing test-time prompt tuning (TPT) methods focus on single-modality data, primarily enhancing images and using confidence ratings to filter out inaccurate images. However, while image generation models can produce visually diverse images, single-modality data enhancement techniques still fail to capture the comprehensive knowledge provided by different modalities. Additionally, we note that the performance of TPT-based methods drops significantly when the number of augmented images is limited, which is not unusual given the computational expense of generative augmentation. To address these issues, we introduce IT3A, a novel test-time adaptation method that utilizes a pre-trained generative model for multi-modal augmentation of each test sample from unknown new domains. By combining augmented data from pre-trained vision and language models, we enhance the ability of the model to adapt to unknown new test data. Additionally, to ensure that key semantics are accurately retained when generating various visual and text enhancements, we employ cosine similarity filtering between the logits of the enhanced images and text with the original test data. This process allows us to filter out some spurious augmentation and inadequate combinations. To leverage the diverse enhancements provided by the generation model across different modals, we have replaced prompt tuning with an adapter for greater flexibility in utilizing text templates. Our experiments on the test datasets with distribution shifts and domain gaps show that in a zero-shot setting, IT3A outperforms state-of-the-art test-time prompt tuning methods with a 5.50% increase in accuracy.

Computer Vision and Pattern Recognition
Reliable Test-Time Adaptation via Agreement-on-the-Line

Eungyeup Kim,Mingjie Sun,Aditi Raghunathan,Zico Kolter

DOI: https://doi.org/10.48550/arXiv.2310.04941

IF: 5.414

2023-10-07

Machine Learning

Abstract:Test-time adaptation (TTA) methods aim to improve robustness to distribution shifts by adapting models using unlabeled data from the shifted test distribution. However, there remain unresolved challenges that undermine the reliability of TTA, which include difficulties in evaluating TTA performance, miscalibration after TTA, and unreliable hyperparameter tuning for adaptation. In this work, we make a notable and surprising observation that TTAed models strongly show the agreement-on-the-line phenomenon (Baek et al., 2022) across a wide range of distribution shifts. We find such linear trends occur consistently in a wide range of models adapted with various hyperparameters, and persist in distributions where the phenomenon fails to hold in vanilla models (i.e., before adaptation). We leverage these observations to make TTA methods more reliable in three perspectives: (i) estimating OOD accuracy (without labeled data) to determine when TTA helps and when it hurts, (ii) calibrating TTAed models without label information, and (iii) reliably determining hyperparameters for TTA without any labeled validation data. Through extensive experiments, we demonstrate that various TTA methods can be precisely evaluated, both in terms of their improvements and degradations. Moreover, our proposed methods on unsupervised calibration and hyperparameters tuning for TTA achieve results close to the ones assuming access to ground-truth labels, in terms of both OOD accuracy and calibration error.
Robust Test-Time Adaptation in Dynamic Scenarios

Longhui Yuan,Binhui Xie,Shuang Li

2023-03-24

Abstract:Test-time adaptation (TTA) intends to adapt the pretrained model to test distributions with only unlabeled test data streams. Most of the previous TTA methods have achieved great success on simple test data streams such as independently sampled data from single or multiple distributions. However, these attempts may fail in dynamic scenarios of real-world applications like autonomous driving, where the environments gradually change and the test data is sampled correlatively over time. In this work, we explore such practical test data streams to deploy the model on the fly, namely practical test-time adaptation (PTTA). To do so, we elaborate a Robust Test-Time Adaptation (RoTTA) method against the complex data stream in PTTA. More specifically, we present a robust batch normalization scheme to estimate the normalization statistics. Meanwhile, a memory bank is utilized to sample category-balanced data with consideration of timeliness and uncertainty. Further, to stabilize the training procedure, we develop a time-aware reweighting strategy with a teacher-student model. Extensive experiments prove that RoTTA enables continual testtime adaptation on the correlatively sampled data streams. Our method is easy to implement, making it a good choice for rapid deployment. The code is publicly available at <a class="link-external link-https" href="https://github.com/BIT-DA/RoTTA" rel="external noopener nofollow">this https URL</a>

Computer Vision and Pattern Recognition

Understanding Test-Time Augmentation

Test-Time Augmentation Meets Variational Bayes

Improved Text Classification via Test-Time Augmentation

Feature Augmentation based Test-Time Adaptation

STTA: enhanced text classification via selective test-time augmentation

Better Aggregation in Test-Time Augmentation

Boosting Anomaly Detection Using Unsupervised Diverse Test-Time Augmentation

Test-Time Augmentation for 3D Point Cloud Classification and Segmentation

Test-Time Augmentation for Traveling Salesperson Problem

Anomaly Detection with Test Time Augmentation and Consistency Evaluation

Active Test-Time Adaptation: Theoretical Analyses and An Algorithm

Efficient Improvement of Classification Accuracy via Selective Test-Time Augmentation

Bag of Tricks for Fully Test-Time Adaptation

On Pitfalls of Test-Time Adaptation

Confidence-based and sample-reweighted test-time adaptation

Intelligent Multi-View Test Time Augmentation

Generalized Robust Test-Time Adaptation in Continuous Dynamic Scenarios

MITA: Bridging the Gap between Model and Data for Test-time Adaptation

Diffusion-Enhanced Test-time Adaptation with Text and Image Augmentation

Reliable Test-Time Adaptation via Agreement-on-the-Line

Robust Test-Time Adaptation in Dynamic Scenarios