Abstract:We introduce Universum learning for multiclass problems and propose a novel formulation for multiclass universum SVM (MU-SVM). We also propose an analytic span bound for model selection with almost 2-4x faster computation times than standard resampling techniques. We empirically demonstrate the efficacy of the proposed MUSVM formulation on several real world datasets achieving > 20% improvement in test accuracies compared to multi-class SVM.
What problem does this paper attempt to address?
This paper attempts to address several key challenges in multi - classification problems:
1. **Extending Universum Learning to Multi - classification Scenarios**:
- The main objective of the paper is to extend the Universum learning method from binary - classification problems to multi - classification problems. In the multi - classification case, in addition to the labeled training data, a set of unlabeled Universum samples are introduced, which do not belong to any of the known classes. This method aims to utilize additional knowledge to improve the generalization ability of the classifier.
2. **Proposing a New Multi - classification Universum SVM (MU - SVM) Model**:
- The authors propose a new multi - classification Universum SVM (Multiclass Universum SVM, MU - SVM) model. This model combines the standard SVM loss function and a special loss function for Universum samples (i.e., ∆ - insensitive loss). In this way, MU - SVM can not only handle multi - classification problems but also effectively utilize the prior knowledge provided by unlabeled Universum samples.
3. **Providing an Efficient Model Selection Method**:
- The paper introduces a new definition of Span and derives a leave - one - out (l.o.o.) error bound. This error bound can be used for model selection and is more computationally efficient compared to traditional resampling techniques (such as cross - validation). Specifically, this method can provide reliable error estimates for model selection while maintaining a relatively low computational complexity.
4. **Empirical Analysis and Performance Improvement**:
- Through experiments on multiple real - world datasets, the paper shows that MU - SVM has a significant improvement in test accuracy compared to traditional multi - classification SVMs. In particular, under certain types of Universum samples, the performance improvement of MU - SVM can reach more than 20%.
### Formula Summary
- **MU - SVM Optimization Problem**:
\[
\min_{w_1, \ldots, w_L, \xi, \zeta} \frac{1}{2} \sum_{l = 1}^{L} \|w_l\|^2 + C \sum_{i = 1}^{n} \xi_i + C^* \sum_{i' = 1}^{m} \sum_{k = 1}^{L} \zeta_{i'k}
\]
Constraints:
\[
\begin{aligned}
& (w_{y_i} - w_l)^T x_i \geq e_{il} - \xi_i, & \forall i = 1, \ldots, n, \; l = 1, \ldots, L \\
& |(w_k^T x_{i'} - \max_{l = 1,\ldots,L} w_l^T x_{i'})| \leq \Delta + \zeta_{i'k}, & \forall i' = 1, \ldots, m, \; k = 1, \ldots, L \\
& \zeta_{i'k} \geq 0, & \forall i' = 1, \ldots, m, \; k = 1, \ldots, L
\end{aligned}
\]
- **Analytical Expression of Leave - One - Out Error Bound**:
\[
R_{\text{l.o.o}} \leq \frac{1}{n} (|\Psi_1| + |\Psi_2|)
\]
where,
\[
\begin{aligned}
\Psi_1 &= \{ t \in SV_2 \cap T \} \\
\Psi_2 &= \left\{ t \in SV_1 \cap T \mid S_t \cdot \max\left(\sqrt{2D}, \frac{1}{\sqrt{C}}\right) \geq 1 \right\}
\end{aligned}