Joint & Progressive Learning from High-Dimensional Data for Multi-Label Classification

Danfeng Hong,Naoto Yokoya,Jian Xu,Xiaoxiang Zhu

DOI: https://doi.org/10.48550/arXiv.1808.05110

2018-08-15

Abstract:Despite the fact that nonlinear subspace learning techniques (e.g. manifold learning) have successfully applied to data representation, there is still room for improvement in explainability (explicit mapping), generalization (out-of-samples), and cost-effectiveness (linearization). To this end, a novel linearized subspace learning technique is developed in a joint and progressive way, called \textbf{j}oint and \textbf{p}rogressive \textbf{l}earning str\textbf{a}teg\textbf{y} (J-Play), with its application to multi-label classification. The J-Play learns high-level and semantically meaningful feature representation from high-dimensional data by 1) jointly performing multiple subspace learning and classification to find a latent subspace where samples are expected to be better classified; 2) progressively learning multi-coupled projections to linearly approach the optimal mapping bridging the original space with the most discriminative subspace; 3) locally embedding manifold structure in each learnable latent subspace. Extensive experiments are performed to demonstrate the superiority and effectiveness of the proposed method in comparison with previous state-of-the-art methods.

Machine Learning

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is the representation and classification of high - dimensional data in multi - label classification tasks. Specifically, the author points out that although current non - linear subspace learning techniques (such as manifold learning) have achieved certain success in data representation, there is still room for improvement in interpretability (explicit mapping), generalization ability (handling out - of - sample data), and cost - effectiveness (linearization).

To solve these problems, the author proposes a new linearized subspace learning technique, called the Joint and Progressive Learning strategy (J - Play), and applies it to multi - label classification. The main goal of J - Play is to learn high - level and semantically meaningful feature representations from high - dimensional data, which is achieved in the following ways:

1. **Jointly perform multiple subspace learning and classification**: Look for a latent subspace in which samples can be better classified.
2. **Progressive learning of multi - coupled projections**: Linearly approximate the optimal mapping from the original space to the most discriminative subspace.
3. **Locally embed the manifold structure**: Maintain the local topological structure in each learnable latent subspace.

Through these methods, J - Play aims to improve the discriminative ability of features and classification performance while overcoming the limitations of traditional subspace learning methods. Experimental results show that J - Play outperforms existing advanced methods on multiple datasets, especially on high - dimensional data such as hyperspectral images and face images.

### Formula Summary

- **Label Matrix \(Y\)**:
\[
y_{kt}=\begin{cases}
1 & \text{if } y_k \text{ belongs to the }t\text{-th class}\\
0 & \text{otherwise}
\end{cases}
\]

- **Reconstruction Loss Term \(\Upsilon(\{\Theta_l\}_{l = 1}^m)\)**:
\[
\Upsilon(\{\Theta_l\}_{l = 1}^m)=\sum_{l = 1}^m\|X_{l - 1}-\Theta_l^T\Theta_lX_{l - 1}\|_F^2
\]

- **Prediction Loss Term \(E(P,\{\Theta_l\}_{l = 1}^m)\)**:
\[
E(P,\{\Theta_l\}_{l = 1}^m)=\|Y - P\Theta_m\cdots\Theta_l\cdots\Theta_1X\|_F^2
\]

- **Local Manifold Regularization \(\Phi(\{\Theta_l\}_{l = 1}^m)\)**:
\[
\Phi(\{\Theta_l\}_{l = 1}^m)=\sum_{l = 1}^m\text{tr}(\Theta_lX_{l - 1}LX_{l - 1}^T\Theta_l^T)
\]

- **Regression Coefficient Regularization \(\Psi(P)\)**:
\[
\Psi(P)=\|P\|_F^2
\]

These formulas together form the core part of the J - Play model, ensuring the effectiveness and robustness of the model.