Abstract:Active learning is a framework for supervised learning to improve the predictive performance by adaptively annotating a small number of samples. To realize efficient active learning, both an acquisition function that determines the next datum and a stopping criterion that determines when to stop learning should be considered. In this study, we propose a stopping criterion based on error stability, which guarantees that the change in generalization error upon adding a new sample is bounded by the annotation cost and can be applied to any Bayesian active learning. We demonstrate that the proposed criterion stops active learning at the appropriate timing for various learning models and real datasets.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is: in Active Learning (AL), how to determine an appropriate stopping criterion to effectively balance the annotation cost and model performance. Specifically, most traditional AL methods adopt a fixed - budget strategy, that is, stop learning when the amount of annotated data reaches a preset value. However, this method is prone to over - sampling or under - sampling problems, thus affecting the generalization ability of the model. Therefore, a stopping criterion based on the generalization error is required to more reasonably terminate the AL process.
### Main contributions of the paper:
1. **Universal stopping criterion**: A stopping criterion based on error stability is proposed, which is applicable to any Bayesian active learning algorithm. This criterion decides when to stop learning by controlling the change of the generalization error, and its high correlation with the generalization error can be verified through experiments.
2. **Bound of generalization error differences**: It is proved that under the PAC - Bayesian framework, the differences between the expected generalization errors can be bounded. Different from traditional PAC - Bayesian learning, this bound does not assume that samples are independent of each other, so it is more suitable for AL scenarios. In addition, when the posterior distribution converges, the proposed bound also tends to zero.
3. **Application to multiple Bayesian AL algorithms**: It is shown that this criterion can work effectively in multiple Bayesian active learning algorithms, including Bayesian linear regression, Bayesian logistic regression, Gaussian process regression, and Bayesian deep learning based on Dropout. In particular, the analytical expressions of the KL divergence between GP posteriors and its bound are derived.
### Main ideas:
- By introducing the concept of error stability, a new stopping criterion is defined. This criterion does not require specific assumptions about the model and can adapt to non - i.i.d. data.
- A normalized threshold range [0, 1] is proposed, making it easier for users to set parameters without considering specific sampling costs.
In short, this paper aims to provide a theoretically - based and highly - practical active learning stopping criterion to help researchers and practitioners better control resource allocation and model performance in the AL process.