Abstract:Recent advances in machine learning have led to increased interest in reproducing kernel Banach spaces (RKBS) as a more general framework that extends beyond reproducing kernel Hilbert spaces (RKHS). These works have resulted in the formulation of representer theorems under several regularized learning schemes. However, little is known about an optimization method that encompasses these results in this setting. This paper addresses a learning problem on Banach spaces endowed with a reproducing kernel, focusing on efficient optimization within RKBS. To tackle this challenge, we propose an algorithm based on mirror descent (MDA). Our approach involves an iterative method that employs gradient steps in the dual space of the Banach space using the reproducing kernel.
We analyze the convergence properties of our algorithm under various assumptions and establish two types of results: first, we identify conditions under which a linear convergence rate is achievable, akin to optimization in the Euclidean setting, and provide a proof of the linear rate; second, we demonstrate a standard convergence rate in a constrained setting. Moreover, to instantiate this algorithm in practice, we introduce a novel family of RKBSs with $p$-norm ($p \neq 2$), characterized by both an explicit dual map and a kernel.
What problem does this paper attempt to address?
### What problem does this paper attempt to solve?
This paper aims to solve the problem of efficient optimization in Reproducing Kernel Banach Spaces (RKBS). Specifically, the author focuses on how to design an optimization algorithm in non - Hilbertian Banach spaces to achieve efficient function learning and minimize the optimization error.
#### Background and Motivation
1. **Limitations of Existing Methods**:
- Most of the current machine - learning research focuses on Reproducing Kernel Hilbert Spaces (RKHS). Although these spaces have a representation theorem for the optimal solution, they may lack sufficient expressive power in terms of approximation error.
- Although previous work has proposed Reproducing Kernel Banach Spaces (RKBS) to expand the framework, effective optimization methods for these spaces are still less studied.
2. **Balance between Optimization Error and Approximation Error**:
- When choosing the model class \( F \), it is usually necessary to find a balance point between the optimization error and the approximation error. Traditional kernel methods are mainly carried out in RKHS. Although they can reduce the approximation error, they may not be able to fully capture the information in complex data structures.
3. **Advantages of Banach Spaces**:
- Banach spaces provide more abundant geometric structures and norm choices, which may improve the approximation performance. Especially when dealing with complex non - linear data, the flexibility of Banach spaces is particularly important.
#### Main Contributions of the Paper
1. **Propose an Algorithm Based on Mirror Descent**:
- The author proposes a method based on the Mirror Descent Algorithm (MDA) specifically for optimization problems in Reproducing Kernel Banach Spaces. This method achieves effective optimization by performing gradient steps in the dual space of the Banach space and mapping the results back to the original space.
2. **Convergence Analysis**:
- The convergence properties of the proposed algorithm under different conditions are studied. In particular, it is proved that under certain assumptions, the algorithm can achieve a linear convergence rate, similar to optimization methods in Euclidean spaces. In addition, the standard convergence rate is also shown under constraint conditions.
3. **Instantiation and Application**:
- A specific instance of Reproducing Kernel Banach Space, namely RKBS with \( p \)-norm, is proposed, and the corresponding dual maps are introduced. This provides a theoretical basis and technical means for practical applications.
4. **Theoretical Guarantee**:
- It is assumed that RKBS is reflexive, that is, the dual space of its dual space is isomorphic to itself, ensuring the theoretical validity of the algorithm. At the same time, it is proved that under the conditions of strong convexity and smoothness, MDA can achieve linear convergence.
#### Summary
In general, this paper fills the gap in effective optimization methods in Reproducing Kernel Banach Spaces by introducing an optimization algorithm based on mirror descent, providing a new idea and tool for solving the learning problems of high - dimensional complex data.