Abstract:This paper develops a novel mathematical framework for collaborative learning by means of geometrically inspired kernel machines which includes statements on the bounds of generalisation and approximation errors, and sample complexity. For classification problems, this approach allows us to learn bounded geometric structures around given data points and hence solve the global model learning problem in an efficient way by exploiting convexity properties of the related optimisation problem in a Reproducing Kernel Hilbert Space (RKHS). In this way, we can reduce classification problems to determining the closest bounded geometric structure from a given data point. Further advantages that come with our solution is that our approach does not require clients to perform multiple epochs of local optimisation using stochastic gradient descent, nor require rounds of communication between client/server for optimising the global model. We highlight that numerous experiments have shown that the proposed method is a competitive alternative to the state-of-the-art.

What problem does this paper attempt to address?

The problems that this paper attempts to solve mainly focus on the collaborative learning challenges in the federated learning environment, specifically including the following aspects: 1. **Theoretical analysis framework**: The paper aims to construct a theoretical analysis framework for collaborative learning methods applicable to distributed and statistically heterogeneous data. This framework needs to be able to: - Calculate the boundaries of generalization error and approximation error. - Determine the minimum number of training samples required to reduce the risk of the objective function, with a probability of at least \(1 - \delta\) (for any \(\delta\in(0,1)\)) such that the risk is lower than \(\epsilon\) (for any \(\epsilon > 0\)). 2. **Global model optimization problem**: In the federated learning environment, the paper hopes to solve the global model optimization problem without multiple rounds of communication between the client and the server. Specifically, traditional federated learning methods usually require the client to perform multiple local optimizations using local data and communicate with the server in multiple rounds to update the global model. The paper attempts to avoid these communication overheads. 3. **Kernel - machine - based learning method**: The paper explores whether the kernel machine learning theory can provide a competitive and computationally efficient alternative to optimization methods based on stochastic gradient descent. Traditional federated learning methods usually rely on gradient descent, which may bring communication and computational bottlenecks in practical applications. ### Specific problem statements - **Q1**: Can a theoretical analysis framework be constructed for handling collaborative learning of distributed and statistically heterogeneous data without making assumptions about the data distribution? This framework should be able to calculate the boundaries of generalization error and approximation error and determine the minimum number of training samples required to reduce the risk of the objective function. - **Q2**: Can the global model optimization problem be solved in the federated learning environment without multiple rounds of communication between the client and the server? - **Q3**: Can the kernel machine learning theory provide a competitive and computationally efficient alternative to optimization methods based on stochastic gradient descent? ### Solutions The paper proposes a method based on geometrically - inspired kernel machines (KAHM) and solves the above problems through the following steps: 1. **Define the KAHM - induced kernel function**: A new kernel function \(K_c\) is introduced, which measures the distance similarity between two data points relative to the training samples. 2. **Define the data - dependent hypothesis space**: The hypothesis space is defined as the convex hull in the reproducing kernel Hilbert space (RKHS) related to the KAHM - induced kernel function. 3. **Calculate the upper bound of the Rademacher complexity of the hypothesis space**: The upper bound of the Rademacher complexity of the hypothesis space is provided, which helps to derive the upper bound of the generalization error. 4. **Derive the upper bound of the generalization error**: Using the upper bound of the Rademacher complexity, the upper bound of the generalization error of the hypothesis space is derived. 5. **Formulate the global model learning problem**: The global model learning problem is formulated as an optimization problem, which is carried out on an appropriate subset of the hypothesis space. 6. **Derive the learning solution using the convex hull form of the hypothesis space**: Using the convex hull form of the hypothesis space and practical assumptions, a learning solution that does not require estimating model parameters is derived. 7. **Derive the upper bound of the approximation error of the objective function**: Using the upper bound of the generalization error, the upper bound of the approximation error of the objective function is derived. 8. **Calculate the sample complexity**: Using the upper bound of the risk of the approximation error of the objective function, the sample complexity is calculated. 9. **Provide a deterministic analysis of the solution**: Using the upper bound of the KAHM - induced distance function, the solution is analyzed and explained from the perspective of the training sample distance. Through these steps, the paper provides a unified framework for designing and analyzing collaborative learning methods based on geometrically - inspired kernel machines, thereby solving the key problems in federated learning.

Geometrically Inspired Kernel Machines for Collaborative Learning Beyond Gradient Descent

Generalized Convexity-Based Inexact Projection Method for Multiple Kernel Learning

Geometric Learning with Positively Decomposable Kernels

Collaborative and Geometric Multi-Kernel Learning for Multi-Class Classification

Optimizing Kernel Machines using Deep Learning

Every Model Learned by Gradient Descent Is Approximately a Kernel Machine

High-performance Kernel Machines with Implicit Distributed Optimization and Randomization

A Kernel Perspective on Distillation-based Collaborative Learning

Geometric Collaborative Filtering with Convergence

Beyond Euclid: An Illustrated Guide to Modern Machine Learning with Geometric, Topological, and Algebraic Structures

Hierarchical-Hyperplane Kernels for Actively Learning Gaussian Process Models of Nonstationary Systems

Limited Memory Online Gradient Descent for Kernelized Pairwise Learning with Dynamic Averaging

Learning Distance Metric for Support Vector Machine: A Multiple Kernel Learning Approach.

On Mitigating the Utility-Loss in Differentially Private Learning: A New Perspective by a Geometrically Inspired Kernel Approach

Consensus Local Graph for Multiple Kernel Clustering

An Online Multiple Kernel Parallelizable Learning Scheme

Overlapping Cover Local Regression Machines

Randomized Geometric Algebra Methods for Convex Neural Networks

Reduced Multiple Empirical Kernel Learning Machine

A Geometrical Analysis of Kernel Ridge Regression and its Applications