Metric Learning from Relative Comparisons by Minimizing Squared Residual

Eric Yi Liu,Zhishan Guo,Xiang Zhang,Vladimir Jojic,Wei Wang

DOI: https://doi.org/10.1109/ICDM.2012.38

2012-01-01

Abstract:Recent studies [1] -- [5] have suggested using constraints in the form of relative distance comparisons to represent domain knowledge: d(a, b)

What problem does this paper attempt to address?

Metric Learning Via Penalized Optimization.

Hao Huang,Yanan Peng,Ting Gan,Weiping Tu,Ruiting Zhou,Sai Wu

DOI: https://doi.org/10.1145/3447548.3467369

2021-01-01

Abstract:Metric learning aims to project original data into a new space, where data points can be classified more accurately using kNN or similar types of classification algorithms. To avoid trivial learning results such as indistinguishably projecting the data onto a line, many existing approaches formulate metric learning as a constrained optimization problem, like finding a metric that minimizes the distance between data points from the same class, with a constraint of ensuring a certain separation for data points from different classes, and then they approximate the optimal solution to the constrained optimization in an iterative way. In order to improve the classification accuracy as much as possible, we try to find a metric that is able to minimize the intra-class distance and maximize the inter-class distance simultaneously. Towards this, we formulate metric learning as a penalized optimization problem, and provide design guideline, paradigms with a general formula, as well as two representative instantiations for the penalty term. In addition, we provide an analytical solution for the penalized optimization, with which costly computation can be avoid, and more importantly, there is no need to worry about the convergence rates or approximation ratios any more. Extensive experiments on real-world data sets are conducted, and the results verify the effectiveness and efficiency of our approach.
Metric Learning from Limited Pairwise Preference Comparisons

Zhi Wang,Geelon So,Ramya Korlakai Vinayak

2024-07-13

Abstract:We study metric learning from preference comparisons under the ideal point model, in which a user prefers an item over another if it is closer to their latent ideal item. These items are embedded into $\mathbb{R}^d$ equipped with an unknown Mahalanobis distance shared across users. While recent work shows that it is possible to simultaneously recover the metric and ideal items given $\mathcal{O}(d)$ pairwise comparisons per user, in practice we often have a limited budget of $o(d)$ comparisons. We study whether the metric can still be recovered, even though it is known that learning individual ideal items is now no longer possible. We show that in general, $o(d)$ comparisons reveal no information about the metric, even with infinitely many users. However, when comparisons are made over items that exhibit low-dimensional structure, each user can contribute to learning the metric restricted to a low-dimensional subspace so that the metric can be jointly identified. We present a divide-and-conquer approach that achieves this, and provide theoretical recovery guarantees and empirical validation.

Machine Learning
Constrained Metric Learning Via Distance Gap Maximization

Wei Liu,Xinmei Tian,Dacheng Tao,Jianzhuang Liu

DOI: https://doi.org/10.1609/aaai.v24i1.7701

2010-01-01

Abstract:Vectored data frequently occur in a variety of fields, which are easy to handle since they can be mathematically abstracted as points residing in a Euclidean space. An appropriate distance metric in the data space is quite demanding for a great number of applications. In this paper, we pose robust and tractable metric learning under pairwise constraints that are expressed as similarity judgements between data pairs. The major features of our approach include: 1) it maximizes the gap between the average squared distance among dissimilar pairs and the average squared distance among similar pairs; 2) it is capable of propagating similar constraints to all data pairs; and 3) it is easy to implement in contrast to the existing approaches using expensive optimization such as semidefinite programming. Our constrained metric learning approach has widespread applicability without being limited to particular backgrounds. Quantitative experiments are performed for classification and retrieval tasks, uncovering the effectiveness of the proposed approach.
Learning a Distance Metric by Empirical Loss Minimization

Wei Bian,Dacheng Tao

DOI: https://doi.org/10.5591/978-1-57735-516-8/ijcai11-202

2011-01-01

Abstract:In this paper, we study the problem of learning a metric and propose a loss function based metric learning framework, in which the metric is estimated by minimizing an empirical risk over a training set. With mild conditions on the instance distribution and the used loss function, we prove that the empirical risk converges to its expected counterpart at rate of root- n . In addition, with the assumption that the best metric that minimizes the expected risk is bounded, we prove that the learned metric is consistent. Two example algorithms are presented by using the proposed loss function based metric learning framework, each of which uses a log loss function and a smoothed hinge loss function, respectively. Experimental results suggest the effectiveness of the proposed algorithms.
Kullback–Leibler Divergence Metric Learning

Shuyi Ji,Zizhao Zhang,Shihui Ying,Liejun Wang,Xibin Zhao,Yue Gao

DOI: https://doi.org/10.1109/tcyb.2020.3008248

IF: 11.8

2020-01-01

IEEE Transactions on Cybernetics

Abstract:The Kullback-Leibler divergence (KLD), which is widely used to measure the similarity between two distributions, plays an important role in many applications. In this article, we address the KLD metric-learning task, which aims at learning the best KLD-type metric from the distributions of datasets. Concretely, first, we extend the conventional KLD by introducing a linear mapping and obtain the best KLD to well express the similarity of data distributions by optimizing such a linear mapping. It improves the expressivity of data distribution, which means it makes the distributions in the same class close and those in different classes far away. Then, the KLD metric learning is modeled by a minimization problem on the manifold of all positive-definite matrices. To deal with this optimization task, we develop an intrinsic steepest descent method, which preserves the manifold structure of the metric in the iteration. Finally, we apply the proposed method along with ten popular metric-learning approaches on the tasks of 3-D object classification and document classification. The experimental results illustrate that our proposed method outperforms all other methods.
Deep Localized Metric Learning.

Yueqi Duan,Jiwen Lu,Jianjiang Feng,Jie Zhou

DOI: https://doi.org/10.1109/TCSVT.2017.2711015

IF: 5.859

2018-01-01

IEEE Transactions on Circuits and Systems for Video Technology

Abstract:Metric learning has been widely used in many visual analysis applications, which learns new distance metrics to measure the similarities of samples effectively. Conventional metric learning methods learn a single linear Mahalanobis metric, yet such linear projections are not powerful enough to capture the nonlinear relationships. Recently, deep metric learning approaches, such as discriminative de...
Distance-Dependent Metric Learning.

Xianhong Chen,Liang He,Can Xu,Jia Liu

DOI: https://doi.org/10.1109/lsp.2019.2891913

2019-01-01

IEEE Signal Processing Letters

Abstract:In most existing metric learning methods, the data pairs are equally treated without considering their diversity. In fact, for pairs with different distances, the main purpose of the metric acted on them has some differences. If the pairs have smaller distance, metric should focus more on expanding the negative pairs, which are more easily misjudged. While if the pairs have larger distance, metric should focus more on shrinking the positive pairs. However, mast metric learning methods neglect these differences. In this letter, we propose a distance-dependent metric learning ((DML)-M-2) method. It partitions data pairs into different clusters according to the l(2) distance between them. Each cluster is associated with a Mahalanobis metric that learns the pairs' distance. This not only allows us to make each metric more targeted and adapt to the data diversity flexibly, but also avoids the problem of computing the distance between points assigned to different clusters, which happens in some local metric learning methods. (DML)-M-2 is further extended to D-3 ML to embrace the nonlinear capacity of neural network. Experiments on UCI datasets and speaker recognition i-vector machine learning challenge show that the proposed methods are superior to other metric learning methods.
Geometric Mean Metric Learning

Pourya Habib Zadeh,Reshad Hosseini,Suvrit Sra

DOI: https://doi.org/10.48550/arXiv.1607.05002

2016-07-18

Abstract:We revisit the task of learning a Euclidean metric from data. We approach this problem from first principles and formulate it as a surprisingly simple optimization problem. Indeed, our formulation even admits a closed form solution. This solution possesses several very attractive properties: (i) an innate geometric appeal through the Riemannian geometry of positive definite matrices; (ii) ease of interpretability; and (iii) computational speed several orders of magnitude faster than the widely used LMNN and ITML methods. Furthermore, on standard benchmark datasets, our closed-form solution consistently attains higher classification accuracy.

Machine Learning
Learning Multiple Local Metrics: Global Consideration Helps.

Han-Jia Ye,D C Zhan,Nan Li,Yuan Jiang

DOI: https://doi.org/10.1109/TPAMI.2019.2901675

IF: 23.6

2020-01-01

IEEE Transactions on Pattern Analysis and Machine Intelligence

Abstract:Learning distance metric between objects provides a better measurement for their relative comparisons. Due to the complex properties inside or between heterogeneous objects, multiple local metrics become an essential representation tool to depict various local characteristics of examples. Different from existing methods building more than one local metric directly, however in this paper, we emphas...
What Makes Objects Similar: A Unified Multi-Metric Learning Approach.

Han-Jia Ye,De-Chuan Zhan,Xue-Min Si,Yuan Jiang,Zhi-Hua Zhou

DOI: https://doi.org/10.1109/tpami.2018.2829192

IF: 23.6

2018-01-01

IEEE Transactions on Pattern Analysis and Machine Intelligence

Abstract:Linkages are essentially determined by similarity measures that may be derived from multiple perspectives. For example, spatial linkages are usually generated based on localities of heterogeneous data. Semantic linkages, however, can come from even more properties, such as different physical meanings behind social relations. Many existing metric learning models focus on spatial linkages but leave the rich semantic factors unconsidered. We propose a Unified Multi-Metric Learning (Um$^2$2l) framework to exploit multiple types of metrics with respect to overdetermined similarities between linkages. In Um$^2$2l, types of combination operators are introduced for distance characterization from multiple perspectives, and thus can introduce flexibilities for representing and utilizing both spatial and semantic linkages. Besides, we propose a uniform solver for Um$^2$2l, and the theoretical analysis reflects the generalization ability of Um$^2$2l as well. Extensive experiments on diverse applications exhibit the superior classification performance and comprehensibility of Um$^2$2l. Visualization results also validate its ability to physical meanings discovery.
Efficient Dual Approach to Distance Metric Learning

Chunhua Shen,Junae Kim,Fayao Liu,Lei Wang,Anton van den Hengel

DOI: https://doi.org/10.1109/tnnls.2013.2275170

IF: 14.255

2014-01-01

IEEE Transactions on Neural Networks and Learning Systems

Abstract:Distance metric learning is of fundamental interest in machine learning because the employed distance metric can significantly affect the performance of many learning methods. Quadratic Mahalanobis metric learning is a popular approach to the problem, but typically requires solving a semidefinite programming (SDP) problem, which is computationally expensive. The worst case complexity of solving an SDP problem involving a matrix variable of size D×D with O(D) linear constraints is about O(D(6.5)) using interior-point methods, where D is the dimension of the input data. Thus, the interior-point methods only practically solve problems exhibiting less than a few thousand variables. Because the number of variables is D(D+1)/2, this implies a limit upon the size of problem that can practically be solved around a few hundred dimensions. The complexity of the popular quadratic Mahalanobis metric learning approach thus limits the size of problem to which metric learning can be applied. Here, we propose a significantly more efficient and scalable approach to the metric learning problem based on the Lagrange dual formulation of the problem. The proposed formulation is much simpler to implement, and therefore allows much larger Mahalanobis metric learning problems to be solved. The time complexity of the proposed method is roughly O(D(3)), which is significantly lower than that of the SDP approach. Experiments on a variety of data sets demonstrate that the proposed method achieves an accuracy comparable with the state of the art, but is applicable to significantly larger problems. We also show that the proposed method can be applied to solve more general Frobenius norm regularized SDP problems approximately.
Constrained Empirical Risk Minimization Framework for Distance Metric Learning.

Wei Bian,Dacheng Tao

DOI: https://doi.org/10.1109/tnnls.2012.2198075

IF: 14.255

2012-01-01

IEEE Transactions on Neural Networks and Learning Systems

Abstract:Distance metric learning (DML) has received increasing attention in recent years. In this paper, we propose a constrained empirical risk minimization framework for DML. This framework enriches the state-of-the-art studies on both theoretic and algorithmic aspects. Theoretically, we comprehensively analyze the generalization by bounding the sample and the approximation errors with respect to the best model. Algorithmically, we carefully derive an optimal gradient descent by using Nesterov's method, and provide two example algorithms that utilize the logarithmic loss and the smoothed hinge loss, respectively. We evaluate the new framework on data classification and image retrieval experiments. Results show that the new framework has competitive performance compared with the representative DML algorithms, including Xing's method, large margin nearest neighbor classifier, neighborhood component analysis, and regularized metric learning.
Metric Learning with Adaptive Density Discrimination

Oren Rippel,Manohar Paluri,Piotr Dollar,Lubomir Bourdev

DOI: https://doi.org/10.48550/arXiv.1511.05939

IF: 5.414

2015-11-18

Machine Learning

Abstract:Distance metric learning (DML) approaches learn a transformation to a representation space where distance is in correspondence with a predefined notion of similarity. While such models offer a number of compelling benefits, it has been difficult for these to compete with modern classification algorithms in performance and even in feature extraction. In this work, we propose a novel approach explicitly designed to address a number of subtle yet important issues which have stymied earlier DML algorithms. It maintains an explicit model of the distributions of the different classes in representation space. It then employs this knowledge to adaptively assess similarity, and achieve local discrimination by penalizing class distribution overlap. We demonstrate the effectiveness of this idea on several tasks. Our approach achieves state-of-the-art classification results on a number of fine-grained visual recognition datasets, surpassing the standard softmax classifier and outperforming triplet loss by a relative margin of 30-40%. In terms of computational performance, it alleviates training inefficiencies in the traditional triplet loss, reaching the same error in 5-30 times fewer iterations. Beyond classification, we further validate the saliency of the learnt representations via their attribute concentration and hierarchy recovery properties, achieving 10-25% relative gains on the softmax classifier and 25-50% on triplet loss in these tasks.
Metric Learning for Tag Recommendation: Tackling Data Sparsity and Cold Start Issues

Yuanshuai Luo,Rui Wang,Yaxin Liang,Ankai Liang,Wenyi Liu

2024-11-10

Abstract:With the rapid growth of digital information, personalized recommendation systems have become an indispensable part of Internet services, especially in the fields of e-commerce, social media, and online entertainment. However, traditional collaborative filtering and content-based recommendation methods have limitations in dealing with data sparsity and cold start problems, especially in the face of largescale heterogeneous data, which makes it difficult to meet user expectations. This paper proposes a new label recommendation algorithm based on metric learning, which aims to overcome the challenges of traditional recommendation systems by learning effective distance or similarity metrics to capture the subtle differences between user preferences and item features. Experimental results show that the algorithm outperforms baseline methods including local response metric learning (LRML), collaborative metric learning (CML), and adaptive tensor factorization (ATF) based on adversarial learning on multiple evaluation metrics. In particular, it performs particularly well in the accuracy of the first few recommended items, while maintaining high robustness and maintaining high recommendation accuracy.

Information Retrieval,Machine Learning
Bounded-Distortion Metric Learning

Renjie Liao,Jianping Shi,Ziyang Ma,Jun Zhu,Jiaya Jia

DOI: https://doi.org/10.48550/arxiv.1505.02377

2015-01-01

Abstract:Metric learning aims to embed one metric space into another to benefit tasks like classification and clustering. Although a greatly distorted metric space has a high degree of freedom to fit training data, it is prone to overfitting and numerical inaccuracy. This paper presents {\it bounded-distortion metric learning} (BDML), a new metric learning framework which amounts to finding an optimal Mahalanobis metric space with a bounded-distortion constraint. An efficient solver based on the multiplicative weights update method is proposed. Moreover, we generalize BDML to pseudo-metric learning and devise the semidefinite relaxation and a randomized algorithm to approximately solve it. We further provide theoretical analysis to show that distortion is a key ingredient for stability and generalization ability of our BDML algorithm. Extensive experiments on several benchmark datasets yield promising results.
Empirical Risk Minimization for Metric Learning Using Privileged Information

Xun Yang,Meng Wang,Luming Zhang,Dacheng Tao

2016-01-01

Abstract:Traditional metric learning methods usually make decisions based on a fixed threshold, which may result in a suboptimal metric when the inter-class and inner-class variations are complex. To address this issue, in this paper we propose an effective metric learning method by exploiting privileged information to relax the fixed threshold under the empirical risk minimization framework. Privileged information describes useful high-level semantic information that is only available during training. Our goal is to improve the performance by incorporating privileged information to design a locally adaptive decision function. We jointly learn two distance metrics by minimizing the empirical loss penalizing the difference between the distance in the original space and that in the privileged space. The distance in the privileged space functions as a locally adaptive decision threshold, which can guide the decision making like a teacher. We optimize the objective function using the Accelerated Proximal Gradient approach to obtain a global optimum solution. Experiment results show that by leveraging privileged information, our proposed method can achieve satisfactory performance.
Regularized Large Margin Distance Metric Learning.

Ya Li,Xinmei Tian,Dacheng Tao

DOI: https://doi.org/10.1109/icdm.2016.0129

2016-01-01

Abstract:Distance metric learning plays an important role in many applications, such as classification and clustering. In this paper, we propose a novel distance metric learning using two hinge losses in the objective function. One is the constraint of the pairs which makes the similar pairs (the same label) closer and the dissimilar (different labels) pairs separated as far as possible. The other one is the constraint of the triplets which makes the largest distance between pairs intra the class larger than the smallest distance between pairs inter the classes. Previous works only consider one of the two kinds of constraints. Additionally, different from the triplets used in previous works, we just need a small amount of such special triplets. This improves the efficiency of our proposed method. Consider the situation in which we might not have enough labeled samples, we extend the proposed distance metric learning into a semi-supervised learning framework. Experiments are conducted on several landmark datasets and the results demonstrate the effectiveness of our proposed method.
Semi-supervised Sparse Metric Learning Using Alternating Linearization Optimization

Wei Liu,Shiqian Ma,Dacheng Tao,Jianzhuang Liu,Peng Liu

DOI: https://doi.org/10.1145/1835804.1835947

2010-01-01

Abstract:In plenty of scenarios, data can be represented as vectors and then mathematically abstracted as points in a Euclidean space. Because a great number of machine learning and data mining applications need proximity measures over data, a simple and universal distance metric is desirable, and metric learning methods have been explored to produce sensible distance measures consistent with data relationship. However, most existing methods suffer from limited labeled data and expensive training. In this paper, we address these two issues through employing abundant unlabeled data and pursuing sparsity of metrics, resulting in a novel metric learning approach called semi-supervised sparse metric learning. Two important contributions of our approach are: 1) it propagates scarce prior affinities between data to the global scope and incorporates the full affinities into the metric learning; and 2) it uses an efficient alternating linearization method to directly optimize the sparse metric. Compared with conventional methods, ours can effectively take advantage of semi-supervision and automatically discover the sparse metric structure underlying input data patterns. We demonstrate the efficacy of the proposed approach with extensive experiments carried out on six datasets, obtaining clear performance gains over the state-of-the-arts.
Sparse online relative similarity learning

Dezhong Yao,Peilin Zhao,Chen Yu,Hai Jin,Bin Li

DOI: https://doi.org/10.48550/arXiv.2104.07501

IF: 5.414

2021-04-15

Machine Learning

Abstract:For many data mining and machine learning tasks, the quality of a similarity measure is the key for their performance. To automatically find a good similarity measure from datasets, metric learning and similarity learning are proposed and studied extensively. Metric learning will learn a Mahalanobis distance based on positive semi-definite (PSD) matrix, to measure the distances between objectives, while similarity learning aims to directly learn a similarity function without PSD constraint so that it is more attractive. Most of the existing similarity learning algorithms are online similarity learning method, since online learning is more scalable than offline learning. However, most existing online similarity learning algorithms learn a full matrix with d 2 parameters, where d is the dimension of the instances. This is clearly inefficient for high dimensional tasks due to its high memory and computational complexity. To solve this issue, we introduce several Sparse Online Relative Similarity (SORS) learning algorithms, which learn a sparse model during the learning process, so that the memory and computational cost can be significantly reduced. We theoretically analyze the proposed algorithms, and evaluate them on some real-world high dimensional datasets. Encouraging empirical results demonstrate the advantages of our approach in terms of efficiency and efficacy.
Boundary-restricted metric learning

Shuo Chen,Chen Gong,Xiang Li,Jian Yang,Gang Niu,Masashi Sugiyama

DOI: https://doi.org/10.1007/s10994-023-06380-3

IF: 5.414

2023-09-20

Machine Learning

Abstract:Metric learning aims to learn a distance metric to properly measure the similarities between pairwise examples. Most existing learning algorithms are designed to reduce intra-class distances and meanwhile enlarge inter-class distances by critically introducing a margin between intra-class and inter-class distances. However, such learning objectives may yield boundless (distance) metric space, because their enlargements on inter-class distances are usually unconstrained. In this case, excessively enlarged inter-class distances would relatively reduce the ratio of margin to the whole distance range (i.e., the margin-range-ratio), and thus being against the initial large-margin purpose for discriminating the similarities of data pairs. To address this issue, we propose a new boundary-restricted metric (BRM), which confines the metric space by a restriction function. Such a restriction function is monotonous and gradually converges to an upper bound, which suppresses excessively large distances of data pairs and concurrently maintains the reliable discriminability. After that, the learned metric can be successfully restricted in a finite region, and thereby avoiding the reduction of margin-range-ratio. Theoretically, we prove that BRM tightens the generalization error bound of the traditional learning model without sacrificing the fitting capability or destroying the topological property of the learned metric, which implies that BRM makes a good bias-variance tradeoff for the metric learning task. Extensive experiments on toy data and real-world datasets validate the superiority of our approach over the state-of-the-art metric learning methods.

computer science, artificial intelligence

Metric Learning from Relative Comparisons by Minimizing Squared Residual

Metric Learning Via Penalized Optimization.

Metric Learning from Limited Pairwise Preference Comparisons

Constrained Metric Learning Via Distance Gap Maximization

Learning a Distance Metric by Empirical Loss Minimization

Kullback–Leibler Divergence Metric Learning

Deep Localized Metric Learning.

Distance-Dependent Metric Learning.

Geometric Mean Metric Learning

Learning Multiple Local Metrics: Global Consideration Helps.

What Makes Objects Similar: A Unified Multi-Metric Learning Approach.

Efficient Dual Approach to Distance Metric Learning

Constrained Empirical Risk Minimization Framework for Distance Metric Learning.

Metric Learning with Adaptive Density Discrimination

Metric Learning for Tag Recommendation: Tackling Data Sparsity and Cold Start Issues

Bounded-Distortion Metric Learning

Empirical Risk Minimization for Metric Learning Using Privileged Information

Regularized Large Margin Distance Metric Learning.

Semi-supervised Sparse Metric Learning Using Alternating Linearization Optimization

Sparse online relative similarity learning

Boundary-restricted metric learning