Abstract:One of the main challenges for feature representation in deep learning-based classification is the design of appropriate loss functions that exhibit strong discriminative power. The classical softmax loss does not explicitly encourage discriminative learning of features. A popular direction of research is to incorporate margins in well-established losses in order to enforce extra intra-class compactness and inter-class separability, which, however, were developed through heuristic means, as opposed to rigorous mathematical principles. In this work, we attempt to address this limitation by formulating the principled optimization objective as learning towards the largest margins. Specifically, we firstly propose to employ the class margin as the measure of inter-class separability, and the sample margin as the measure of intra-class compactness. Accordingly, to encourage discriminative representation of features, the loss function should promote the largest possible margins for both classes and samples. Furthermore, we derive a generalized margin softmax loss to draw general conclusions for the existing margin-based losses. Not only does this principled framework offer new perspectives to understand and interpret existing margin-based losses, but it also provides new insights that can guide the design of new tools, including \textit{sample margin regularization} and \textit{largest margin softmax loss} for class balanced cases, and \textit{zero centroid regularization} for class imbalanced cases. Experimental results demonstrate the effectiveness of our strategy for multiple tasks including visual classification, imbalanced classification, person re-identification, and face verification.

A Refined Margin Distribution Analysis for Forest Representation Learning

Margin Distribution Analysis

Improving generalization of deep neural networks by leveraging margin distribution

InterFace:Adjustable Angular Margin Inter-class Loss for Deep Face Recognition

Optimal Margin Distribution Network

Adaptive Embedding and Distribution Re-margin for Long-Tail Recognition

A Margin-Maximizing Fine-Grained Ensemble Method

A Refined Margin Analysis for Boosting Algorithms Via Equilibrium Margin

On the Optimization of Margin Distribution

Learning Towards The Largest Margins.

Improving Deep Forest by Confidence Screening.

Multi-Label Learning with Deep Forest

Improve Deep Forest with Learnable Layerwise Augmentation Policy Schedule

Large Margin Discriminative Loss for Classification

Laplacian Margin Distribution Boosting for Learning from Sparsely Labeled Data

Improved Sample Complexities for Deep Networks and Robust Classification via an All-Layer Margin

Boosting Through Optimization of Margin Distributions

New forest-based approaches for sufficient dimension reduction

Margin distribution and structural diversity guided ensemble pruning

Riemann-Lebesgue Forest for Regression

The Role of Depth, Width, and Tree Size in Expressiveness of Deep Forest