Abstract:Most metric learning algorithms, as well as Fisher's Discriminant Analysis (FDA), optimize some cost function of different measures of within-and between-class distances. On the other hand, Support Vector Machines(SVMs) and several Multiple Kernel Learning (MKL) algorithms are based on the SVM large margin theory. Recently, SVMs have been analyzed from SVM and metric learning, and to develop new algorithms that build on the strengths of each. Inspired by the metric learning interpretation of SVM, we develop here a new metric-learning based SVM framework in which we incorporate metric learning concepts within SVM. We extend the optimization problem of SVM to include some measure of the within-class distance and along the way we develop a new within-class distance measure which is appropriate for SVM. In addition, we adopt the same approach for MKL and show that it can be also formulated as a Mahalanobis metric learning problem. Our end result is a number of SVM/MKL algorithms that incorporate metric learning concepts. We experiment with them on a set of benchmark datasets and observe important predictive performance improvements.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is to combine Support Vector Machine (SVM) and Multiple Kernel Learning (MKL) with metric learning in order to improve classification performance. Specifically:
1. **Combining SVM and metric learning**: Traditional SVM achieves classification by maximizing the inter - class distance (i.e., margin), but ignores the optimization of the intra - class distance. The author proposes a new SVM framework based on metric learning, which takes into account both the inter - class distance and the intra - class distance during the optimization process. This not only improves the classification performance of the model, but also introduces the concept of metric learning into SVM.
2. **Introducing a new intra - class distance metric**: The author defines a new intra - class distance metric, which is especially suitable for SVM, and incorporates it into the optimization problem. This new metric method aims to minimize the distance between samples of the same class, thereby improving the compactness of the model for intra - class samples.
3. **Extension to Multiple Kernel Learning (MKL)**: The author also applies the above ideas to MKL and proposes a MKL framework based on metric learning. Under this framework, not only the margin is optimized, but also the intra - class distance metric is introduced, further enhancing the performance of multiple kernel learning.
4. **Unified perspective**: The paper provides a unified perspective, which links SVM, SVM based on metric learning, metric learning algorithms, and Fisher Discriminant Analysis (FDA) through the concepts of inter - class distance and intra - class distance, revealing the internal relationships among these methods.
In summary, the main objective of this paper is to improve the optimization processes of SVM and MKL by introducing the idea of metric learning, thereby enhancing classification performance. Through experimental verification, the author shows the significant performance improvement of this new framework on multiple benchmark datasets.
### Formula presentation
- **SVM optimization problem**:
\[
\min_{w, \xi, b} \frac{1}{2} \|w\|^2 + C \sum_{i = 1}^l \xi_i
\]
\[
\text{s.t. } y_i(w^T x_i + b) \geq 1 - \xi_i, \quad \forall i
\]
- **SVM optimization problem based on metric learning**:
\[
\min_{w, b, \xi, \eta} \frac{1}{2} \|w\|^2 + C_1 \sum_{i = 1}^n \xi_i + C_2 \sum_{i = 1}^n \eta_i
\]
\[
\text{s.t. } 1 - \xi_i \leq y_i(w^T x_i + b) \leq 1 + \epsilon + \eta_i, \quad \forall i
\]
\[
\xi_i \geq 0, \eta_i \geq 0, \quad \forall i
\]
- **Mahalanobis distance**:
\[
d_M^2(x_i, x_j)=(x_i - x_j)^T M (x_i - x_j)
\]
where \( M \) is a positive semi - definite matrix.
Through these formulas, the paper shows how to introduce the idea of metric learning in SVM and MKL to achieve better classification results.