Abstract:We provide algorithms for regression with adversarial responses under large classes of non-i.i.d. instance sequences, on general separable metric spaces, with provably minimal assumptions. We also give characterizations of learnability in this regression context. We consider universal consistency which asks for strong consistency of a learner without restrictions on the value responses. Our analysis shows that such an objective is achievable for a significantly larger class of instance sequences than stationary processes, and unveils a fundamental dichotomy between value spaces: whether finite-horizon mean estimation is achievable or not. We further provide optimistically universal learning rules, i.e., such that if they fail to achieve universal consistency, any other algorithms will fail as well. For unbounded losses, we propose a mild integrability condition under which there exist algorithms for adversarial regression under large classes of non-i.i.d. instance sequences. In addition, our analysis also provides a learning rule for mean estimation in general metric spaces that is consistent under adversarial responses without any moment conditions on the sequence, a result of independent interest.
What problem does this paper attempt to address?
The main problem that this paper attempts to solve is how to achieve consistent learning performance on a wide range of non - i.i.d. (non - independent and identically distributed) instance sequences when facing regression tasks with adversarial responses. Specifically, the authors focus on the regression problem in general separable metric spaces and aim to design learning algorithms that can maintain strong consistency under various conditions, especially in adversarial environments. These problems include:
1. **Extension of Consistency**: Researchers hope to understand for which types of data sequences universal consistency can be achieved, even when the value responses in these data sequences can be adversarially selected. In particular, they want to explore whether the learner can achieve strong consistency without being limited to value responses.
2. **Dichotomy of the Value Space**: The authors reveal a fundamental dichotomy of the value space, that is, whether the mean estimate can be achieved in a finite time. This depends on the properties of the value space and the loss function, such as whether the F - TiME (Finite - Time Mean Estimation) property is satisfied.
3. **Optimistic Universal Learning Rule**: The paper proposes an optimistic universal learning rule, that is, if this rule fails to achieve universal consistency, then any other algorithm will also fail. This means that when there is an algorithm that can achieve the goal, the proposed rule will be the best choice.
4. **Learning under Unbounded Loss**: For unbounded loss functions, the authors propose a mild integrability condition, so that there are still algorithms suitable for adversarial regression under a wide range of non - i.i.d. instance sequences. This part also includes an analysis of mean estimation in general metric spaces, showing a learning rule that can achieve consistency without any moment conditions under adversarial responses.
5. **Handling of Adversarial Responses**: The paper also considers a slightly more general case of adversarial responses, where the value \(Y_t\) can not only depend arbitrarily on the instance sequence \(X\), but also on the randomness and predictions used by the learning algorithm in the past. This setting is non - trivial, especially for randomized algorithms.
6. **The Case without Empirical Integrability Assumption**: Finally, the paper explores how to deal with the problem of unbounded loss without the empirical integrability assumption. The results show that for unbounded loss, universal learning is very limited, and universal learning may not be feasible unless the responses satisfy certain specific conditions.
Through the above research, the paper not only provides a series of new theoretical results, but also proposes practical learning algorithms that can guarantee good performance under a wide range of conditions, especially in the face of adversarial environments.