Distance Metric Learning by Knowledge Embedding

YG Zhang,CS Zhang,D Zhang

DOI: https://doi.org/10.1016/s0031-3203(03)00218-8

IF: 8

2004-01-01

Pattern Recognition

Abstract:This paper presents an algorithm which learns a distance metric from a data set by knowledge embedding and uses the new distance metric to solve nonlinear pattern recognition problems such a clustering.

What problem does this paper attempt to address?

Clustering In Knowledge Embedded Space

Yg Zhang,Cs Zhang,S Wang

DOI: https://doi.org/10.1007/978-3-540-39857-8_43

2003-01-01

Abstract:Cluster analysis is a fundamental technique in pattern recognition. It is difficult to cluster data on complex data sets. This paper presents a new algorithm for clustering. There are three key ideas in the algorithm: using mutual neighborhood graphs to discover knowledge and cluster data; using eigenvalues of local covariance matrixes to express knowledge and form a knowledge embedded space; and using a denoising trick in knowledge embedded space to implement clustering. Essentially, it learns a new distance metric by knowledge embedding and makes clustering become easier under this distance metric. The experiment results show that the algorithm can construct a quality neighborhood graph from a complex and noisy data set and well solve clustering problems.
Kernel Density Metric Learning

Yujie He,Wenlin Chen,Yixin Chen,Yi Mao

DOI: https://doi.org/10.1109/icdm.2013.153

2013-01-01

Abstract:This paper introduces a supervised metric learning algorithm, called kernel density metric learning (KDML), which is easy to use and provides nonlinear, probability-based distance measures. KDML constructs a direct nonlinear mapping from the original input space into a feature space based on kernel density estimation. The nonlinear mapping in KDML embodies established distance measures between probability density functions, and leads to correct classification on datasets for which linear metric learning methods would fail. It addresses the severe challenge to kNN when features are from heterogeneous domains and, as a result, the Euclidean or Mahalanobis distance between original feature vectors is not meaningful. Existing metric learning algorithms can then be applied to the KDML features. We also propose an integrated optimization algorithm that learns not only the Mahalanobis matrix but also kernel bandwidths, the only hyper-parameters in the nonlinear mapping. KDML can naturally handle not only numerical features, but also categorical ones, which is rarely found in previous metric learning algorithms. Extensive experimental results on various datasets show that KDML significantly improves existing metric learning algorithms in terms of kNN classification accuracy.
Nonlinear Metric Learning with Kernel Density Estimation

Yujie He,Yi Mao,Wenlin Chen,Yixin Chen

DOI: https://doi.org/10.1109/TKDE.2014.2384522

2015-01-01

Abstract:Metric learning, the task of learning a good distance metric, is a key problem in machine learning with ample applications. This paper introduces a novel framework for nonlinear metric learning, called kernel density metric learning (KDML), which is easy to use and provides nonlinear, probability-based distance measures. KDML constructs a direct nonlinear mapping from the original input space into a feature space based on kernel density estimation. The nonlinear mapping in KDML embodies established distance measures between probability density functions, and leads to accurate classification on datasets for which existing linear metric learning methods would fail. It addresses the severe challenge to distance-based classifiers when features are from heterogeneous domains and, as a result, the Euclidean or Mahalanobis distance between original feature vectors is not meaningful. We also propose two ways to determine the kernel bandwidths, including an adaptive local scaling approach and an integrated optimization algorithm that learns the Mahalanobis matrix and kernel bandwidths together. KDML is a general framework that can be combined with any existing metric learning algorithm. As concrete examples, we combine KDML with two leading metric learning algorithms, large margin nearest neighbors (LMNN) and neighborhood component analysis (NCA). KDML can naturally handle not only numerical features, but also categorical ones, which is rarely found in previous metric learning algorithms. Extensive experimental results on various datasets show that KDML significantly improves existing metric learning algorithms in terms of classification accuracy.
Learning Distance Metric for Support Vector Machine: A Multiple Kernel Learning Approach.

Weiqi Zhang,Zifei Yan,Gang Xiao,Hongzhi Zhang,Wangmeng Zuo

DOI: https://doi.org/10.1007/s11063-019-10053-5

IF: 2.565

2019-01-01

Neural Processing Letters

Abstract:Recent work in distance metric learning has significantly improved the performance in k-nearest neighbor classification. However, the learned metric with these methods cannot adapt to the support vector machines (SVM), which are amongst the most popular classification algorithms using distance metrics to compare samples. In order to investigate the possibility to develop a novel model for joint learning distance metric and kernel classifier, in this paper, we provide a new parameterization scheme for incorporating the squared Mahalanobis distance into the Gaussian RBF kernel, and formulate kernel learning into a generalized multiple kernel learning framework, gearing towards SVM classification. We demonstrate the effectiveness of the proposed algorithm on the UCI machine learning datasets of varying sizes and difficulties and two real-world datasets. Experimental results show that the proposed model achieves competitive classification accuracies and comparable execution time by using spectral projected gradient descent optimizer compared with state-of-the-art methods.
Distance Metric Learning from Uncertain Side Information with Application to Automated Photo Tagging

Lei Wu,Steven C. H. Hoi,Rong Jin,Jianke Zhu,Nenghai Yu

DOI: https://doi.org/10.1145/1631272.1631293

2009-01-01

Abstract:Automated photo tagging is essential to make massive unlabeled photos searchable by text search engines. Conventional image annotation approaches, though working reasonably well on small testbeds, are either computationally expensive or inaccurate when dealing with large-scale photo tagging. Recently, with the popularity of networking websites, we observe a massive number of user-tagged referred to as social images, that are available on the web. Unlike traditional web images often contain tags and other user-generated content, which offer a new opportunity to resolve some long-standing challenges in multimedia. In this work, we aim to address the challenge of large-scale automated photo tagging by exploring the images. We present a retrieval based approach for automated photo tagging. To tag a test image, the proposed approach first retrieves k images that share the largest visual similarity with the test image. The tags of the test image are then derived based on the tagging of the similar images. Due to the well-known semantic gap issue, a regular Euclidean distance-based retrieval method often fails to find semantically relevant images. To address the challenge of semantic gap, we propose a novel probabilistic distance metric learning scheme that (1) automatically derives constraints from the uncertain side information, and (2) efficiently learns a distance metric from the derived constraints. We apply the proposed technique to automated photo tagging tasks based on a image testbed with over 200,000 images crawled from Flickr. Encouraging results show that the proposed technique is effective and promising for automated photo tagging.
Distance Metric Learning Via Iterated Support Vector Machines

Wangmeng Zuo,Faqiang Wang,David Zhang,Liang Lin,Yuchi Huang,Deyu Meng,Lei Zhang

DOI: https://doi.org/10.1109/tip.2017.2725578

2015-01-01

Abstract:Distance metric learning aims to learn from the given training data a valid distance metric, with which the similarity between data samples can be more effectively evaluated for classification. Metric learning is often formulated as a convex or nonconvex optimization problem, while most existing methods are based on customized optimizers and become inefficient for large scale problems. In this paper, we formulate metric learning as a kernel classification problem with the positive semi-definite constraint, and solve it by iterated training of support vector machines (SVMs). The new formulation is easy to implement and efficient in training with the off-the-shelf SVM solvers. Two novel metric learning models, namely positive-semidefinite constrained metric learning (PCML) and nonnegative-coefficient constrained metric learning (NCML), are developed. Both PCML and NCML can guarantee the global optimality of their solutions. Experiments are conducted on general classification, face verification, and person re-identification to evaluate our methods. Compared with the state-of-the-art approaches, our methods can achieve comparable classification accuracy and are efficient in training.
Metric learning by discriminant neighborhood embedding

Wei Zhang,Xiangyang Xue,Zichen Sun,Hong Lu,Yue-Fei Guo

DOI: https://doi.org/10.1016/j.patcog.2007.11.023

IF: 8

2008-01-01

Pattern Recognition

Abstract:In this paper, we learn a distance metric in favor of classification task from available labeled samples. Multi-class data points are supposed to be pulled or pushed by discriminant neighbors. We define a discriminant adjacent matrix in favor of classification task and learn a map transforming input data into a new space such that intra-class neighbors become even more nearby while extra-class neighbors become as far away from each other as possible. Our method is non-parametric, non-iterative, and immune to small sample size (SSS) problem. Target dimensionality of the new space is selected by spectral analysis in the proposed method. Experiments on real-world data sets demonstrate the effectiveness of our method.
Measuring Unequal Knowledge Distance by Network Embedding and Multiple Relationships

Keye Wu,Lele Kang,Ziyue Xie,Jia Tina Du,Jianjun Sun

DOI: https://doi.org/10.1002/pra2.987

2023-01-01

Proceedings of the Association for Information Science and Technology

Abstract:ABSTRACTKnowledge distance, representing the dissimilarity between different knowledge units, has been considered as an important dimension of recombination novelty and technological innovation. Previous measurements merely rely on the citation relationship and ignore their directions and weights. To fill this gap, this study proposes a new measurement which not only captures the unequal citation relationship but also integrates multiple information to depict knowledge distance. The results show that our method can accurately portray the knowledge distance in both scientific areas and technical fields.
A scalable algorithm for learning a mahalanobis distance metric

Junae Kim,Chunhua Shen,Lei Wang

DOI: https://doi.org/10.1007/978-3-642-12297-2_29

2009-01-01

Abstract:A distance metric that can accurately reflect the intrinsic characteristics of data is critical for visual recognition tasks An effective solution to defining such a metric is to learn it from a set of training samples In this work, we propose a fast and scalable algorithm to learn a Mahalanobis distance By employing the principle of margin maximization to secure better generalization performances, this algorithm formulates the metric learning as a convex optimization problem with a positive semidefinite (psd) matrix variable Based on an important theorem that a psd matrix with trace of one can always be represented as a convex combination of multiple rank-one matrices, our algorithm employs a differentiable loss function and solves the above convex optimization with gradient descent methods This algorithm not only naturally maintains the psd requirement of the matrix variable that is essential for metric learning, but also significantly cuts down computational overhead, making it much more efficient with the increasing dimensions of feature vectors Experimental study on benchmark data sets indicates that, compared with the existing metric learning algorithms, our algorithm can achieve higher classification accuracy with much less computational load.
Exploring dual information in distance metric learning for clustering

Rodrigo Randel,Daniel Aloise,Alain Hertz

DOI: https://doi.org/10.48550/arXiv.2105.12703

2021-05-27

Abstract:Distance metric learning algorithms aim to appropriately measure similarities and distances between data points. In the context of clustering, metric learning is typically applied with the assist of side-information provided by experts, most commonly expressed in the form of cannot-link and must-link constraints. In this setting, distance metric learning algorithms move closer pairs of data points involved in must-link constraints, while pairs of points involved in cannot-link constraints are moved away from each other. For these algorithms to be effective, it is important to use a distance metric that matches the expert knowledge, beliefs, and expectations, and the transformations made to stick to the side-information should preserve geometrical properties of the dataset. Also, it is interesting to filter the constraints provided by the experts to keep only the most useful and reject those that can harm the clustering process. To address these issues, we propose to exploit the dual information associated with the pairwise constraints of the semi-supervised clustering problem. Experiments clearly show that distance metric learning algorithms benefit from integrating this dual information.

Machine Learning
Scalable Large-Margin Mahalanobis Distance Metric Learning

Chunhua Shen,Junae Kim,Lei Wang

DOI: https://doi.org/10.1109/tnn.2010.2052630

2010-01-01

IEEE Transactions on Neural Networks

Abstract:For many machine learning algorithms such as k-nearest neighbor ( k-NN) classifiers and k-means clustering, often their success heavily depends on the metric used to calculate distances between different data points. An effective solution for defining such a metric is to learn it from a set of labeled training samples. In this work, we propose a fast and scalable algorithm to learn a Mahalanobis distance metric. The Mahalanobis metric can be viewed as the Euclidean distance metric on the input data that have been linearly transformed. By employing the principle of margin maximization to achieve better generalization performances, this algorithm formulates the metric learning as a convex optimization problem and a positive semidefinite (p.s.d.) matrix is the unknown variable. Based on an important theorem that a p.s.d. trace-one matrix can always be represented as a convex combination of multiple rank-one matrices, our algorithm accommodates any differentiable loss function and solves the resulting optimization problem using a specialized gradient descent procedure. During the course of optimization, the proposed algorithm maintains the positive semidefiniteness of the matrix variable that is essential for a Mahalanobis metric. Compared with conventional methods like standard interior-point algorithms or the special solver used in large margin nearest neighbor , our algorithm is much more efficient and has a better performance in scalability. Experiments on benchmark data sets suggest that, compared with state-of-the-art metric learning algorithms, our algorithm can achieve a comparable classification accuracy with reduced computational complexity.
KPML: A Novel Probabilistic Perspective Kernel Mahalanobis Distance Metric Learning Model for Semi-supervised Clustering

Chao Wang,Yongyi Hu,Xiaofeng Gao,Guihai Chen

DOI: https://doi.org/10.1007/978-3-030-59051-2_17

2020-01-01

Abstract:Metric learning aims to transform features of data into another based on some given distance relationships, which may improve the performances of distance-based machine learning models. Most existing methods use the difference between the distance of similar pairs and that of dissimilar pairs as loss functions for training. This kind of loss function may lack interpretability since people can only observe the distance or the difference of the distance, a number with no bounds, but have no idea about how large or small it is. To provide more explanation of these metric learning models, in this paper, we propose the probabilistic theoretical analysis of metric learning, design a special loss function, and propose the Kernelized Probabilistic Metric Learning (KPML) approach. With all the distance values transformed into probabilities, we can, therefore, compare and explain the results of the model. Besides, to effectively make use of both the labeled and unlabeled data to enhance the performance of semi-supervised clustering, we propose a KPML-based approach that leverages metric learning and semi-supervised learning effectively in a novel way. Finally, we use our model to do experiments about kNN-based semi-supervised clustering and the results show that our model significantly outperforms baselines across various datasets.
General Heterogeneous Transfer Distance Metric Learning Via Knowledge Fragments Transfer.

Yong Luo,Yonggang Wen,Tongliang Liu,Dacheng Tao

DOI: https://doi.org/10.24963/ijcai.2017/341

2017-01-01

Abstract:Transfer learning aims to improve the performance of target learning task by leveraging information (or transferring knowledge) from other related tasks. Recently, transfer distance metric learning (TDML) has attracted lots of interests, but most of these methods assume that feature representations for the source and target learning tasks are the same. Hence, they are not suitable for the applications, in which the data are from heterogeneous domains (feature spaces, modalities and even semantics). Although some existing heterogeneous transfer learning (HTL) approaches is able to handle such domains, they lack flexibility in real-world applications, and the learned transformations are often restricted to be linear. We therefore develop a general and flexible heterogeneous TDML (HTDML) framework based on the knowledge fragment transfer strategy. In the proposed HTDML, any (linear or nonlinear) distance metric learning algorithms can be employed to learn the source metric beforehand. Then a set of knowledge fragments are extracted from the pre-learned source metric to help target metric learning. In addition, either linear or nonlinear distance metric can be learned for the target domain. Extensive experiments on both scene classification and object recognition demonstrate superiority of the proposed method.
A Distributed Approach Toward Discriminative Distance Metric Learning

Jun Li,Xun Lin,Xiaoguang Rui,Yong Rui,Dacheng Tao

DOI: https://doi.org/10.1109/tnnls.2014.2377211

IF: 14.255

2015-01-01

IEEE Transactions on Neural Networks and Learning Systems

Abstract:Distance metric learning (DML) is successful in discovering intrinsic relations in data. However, most algorithms are computationally demanding when the problem size becomes large. In this paper, we propose a discriminative metric learning algorithm, develop a distributed scheme learning metrics on moderate-sized subsets of data, and aggregate the results into a global solution. The technique leverages the power of parallel computation. The algorithm of the aggregated DML (ADML) scales well with the data size and can be controlled by the partition. We theoretically analyze and provide bounds for the error induced by the distributed treatment. We have conducted experimental evaluation of the ADML, both on specially designed tests and on practical image annotation tasks. Those tests have shown that the ADML achieves the state-of-the-art performance at only a fraction of the cost incurred by most existing methods.
Knowledge Elicitation using Deep Metric Learning and Psychometric Testing

Lu Yin,Vlado Menkovski,Mykola Pechenizkiy

DOI: https://doi.org/10.1007/978-3-030-67661-2_10

2020-04-14

Abstract:Knowledge present in a domain is well expressed as relationships between corresponding concepts. For example, in zoology, animal species form complex hierarchies; in genomics, the different (parts of) molecules are organized in groups and subgroups based on their functions; plants, molecules, and astronomical objects all form complex taxonomies. Nevertheless, when applying supervised machine learning (ML) in such domains, we commonly reduce the complex and rich knowledge to a fixed set of labels, and induce a model shows good generalization performance with respect to these labels. The main reason for such a reductionist approach is the difficulty in eliciting the domain knowledge from the experts. Developing a label structure with sufficient fidelity and providing comprehensive multi-label annotation can be exceedingly labor-intensive in many real-world applications. In this paper, we provide a method for efficient hierarchical knowledge elicitation (HKE) from experts working with high-dimensional data such as images or videos. Our method is based on psychometric testing and active deep metric learning. The developed models embed the high-dimensional data in a metric space where distances are semantically meaningful, and the data can be organized in a hierarchical structure. We provide empirical evidence with a series of experiments on a synthetically generated dataset of simple shapes, and Cifar 10 and Fashion-MNIST benchmarks that our method is indeed successful in uncovering hierarchical structures.

Machine Learning,Artificial Intelligence
Transferring Knowledge Fragments for Learning Distance Metric from A Heterogeneous Domain

Yong Luo,Yonggang Wen,Tongliang Liu,Dacheng Tao

DOI: https://doi.org/10.1109/TPAMI.2018.2824309

2019-04-08

Abstract:The goal of transfer learning is to improve the performance of target learning task by leveraging information (or transferring knowledge) from other related tasks. In this paper, we examine the problem of transfer distance metric learning (DML), which usually aims to mitigate the label information deficiency issue in the target DML. Most of the current Transfer DML (TDML) methods are not applicable to the scenario where data are drawn from heterogeneous domains. Some existing heterogeneous transfer learning (HTL) approaches can learn target distance metric by usually transforming the samples of source and target domain into a common subspace. However, these approaches lack flexibility in real-world applications, and the learned transformations are often restricted to be linear. This motivates us to develop a general flexible heterogeneous TDML (HTDML) framework. In particular, any (linear/nonlinear) DML algorithms can be employed to learn the source metric beforehand. Then the pre-learned source metric is represented as a set of knowledge fragments to help target metric learning. We show how generalization error in the target domain could be reduced using the proposed transfer strategy, and develop novel algorithm to learn either linear or nonlinear target metric. Extensive experiments on various applications demonstrate the effectiveness of the proposed method.

Machine Learning,Computer Vision and Pattern Recognition
Learning Metrics via Discriminant Kernels and Multidimensional Scaling: Toward Expected Euclidean Representation

Zhihua Zhang

2003-01-01

Abstract:Distance-based methods in machine learning and pattern recognition have to rely on a metric distance between points in the input space. Instead of specifying a metric a pri- ori, we seek to learn the metric from data via kernel methods and multidimensional scal- ing (MDS) techniques. Under the classifica- tion setting, we define discriminant kernels on the joint space of input and output spaces and present a specific family of discriminant kernels. This family of discriminant ker- nels is attractive because the induced met- rics are Euclidean and Fisher separable, and MDS techniques can be used to find the low- dimensional Euclidean representations (also called feature vectors) of the induced met- rics. Since the feature vectors incorporate information from both input points and their corresponding labels and they enjoy Fisher separability, they are appropriate to be used in distance-based classifiers.
Distance-Dependent Metric Learning.

Xianhong Chen,Liang He,Can Xu,Jia Liu

DOI: https://doi.org/10.1109/lsp.2019.2891913

2019-01-01

IEEE Signal Processing Letters

Abstract:In most existing metric learning methods, the data pairs are equally treated without considering their diversity. In fact, for pairs with different distances, the main purpose of the metric acted on them has some differences. If the pairs have smaller distance, metric should focus more on expanding the negative pairs, which are more easily misjudged. While if the pairs have larger distance, metric should focus more on shrinking the positive pairs. However, mast metric learning methods neglect these differences. In this letter, we propose a distance-dependent metric learning ((DML)-M-2) method. It partitions data pairs into different clusters according to the l(2) distance between them. Each cluster is associated with a Mahalanobis metric that learns the pairs' distance. This not only allows us to make each metric more targeted and adapt to the data diversity flexibly, but also avoids the problem of computing the distance between points assigned to different clusters, which happens in some local metric learning methods. (DML)-M-2 is further extended to D-3 ML to embrace the nonlinear capacity of neural network. Experiments on UCI datasets and speaker recognition i-vector machine learning challenge show that the proposed methods are superior to other metric learning methods.
Learning A Mahalanobis Distance Metric for Data Clustering and Classification

Shiming Xiang,Feiping Nie,Changshui Zhang

DOI: https://doi.org/10.1016/j.patcog.2008.05.018

IF: 8

2008-01-01

Pattern Recognition

Abstract:Distance metric is a key issue in many machine learning algorithms. This paper considers a general problem of learning from pairwise constraints in the form of must-links and cannot-links. As one kind of side information, a must-link indicates the pair of the two data points must be in a same class, while a cannot-link indicates that the two data points must be in two different classes. Given must-link and cannot-link information, our goal is to learn a Mahalanobis distance metric. Under this metric, we hope the distances of point pairs in must-links are as small as possible and those of point pairs in cannot-links are as large as possible. This task is formulated as a constrained optimization problem, in which the global optimum can be obtained effectively and efficiently. Finally, some applications in data clustering, interactive natural image segmentation and face pose estimation are given in this paper. Experimental results illustrate the effectiveness of our algorithm.
EMD Metric Learning

Zizhao Zhang,Yubo Zhang,Xibin Zhao,Yue Gao

DOI: https://doi.org/10.1609/aaai.v32i1.11703

2018-01-01

Abstract:Earth Mover's Distance (EMD), targeting at measuring the many-to-many distances, has shown its superiority and been widely applied in computer vision tasks, such as object recognition, hyperspectral image classification and gesture recognition. However, there is still little effort concentrated on optimizing the EMD metric towards better matching performance. To tackle this issue, we propose an EMD metric learning algorithm in this paper. In our method, the objective is to learn a discriminative distance metric for EMD ground distance matrix generation which can better measure the similarity between compared subjects. More specifically, given a group of labeled data from different categories, we first select a subset of training data and then optimize the metric for ground distance matrix generation. Here, both the EMD metric and the EMD flow-network are alternatively optimized until a steady EMD value can be achieved. This method is able to generate a discriminative ground distance matrix which can further improve the EMD distance measurement. We then apply our EMD metric learning method on two tasks, i.e. , multi-view object classification and document classification. The experimental results have shown better performance of our proposed EMD metric learning method compared with the traditional EMD method and the state-of-the-art methods. It is noted that the proposed EMD metric learning method can be also used in other applications.

Distance Metric Learning by Knowledge Embedding

Clustering In Knowledge Embedded Space

Kernel Density Metric Learning

Nonlinear Metric Learning with Kernel Density Estimation

Learning Distance Metric for Support Vector Machine: A Multiple Kernel Learning Approach.

Distance Metric Learning from Uncertain Side Information with Application to Automated Photo Tagging

Distance Metric Learning Via Iterated Support Vector Machines

Metric learning by discriminant neighborhood embedding

Measuring Unequal Knowledge Distance by Network Embedding and Multiple Relationships

A scalable algorithm for learning a mahalanobis distance metric

Exploring dual information in distance metric learning for clustering

Scalable Large-Margin Mahalanobis Distance Metric Learning

KPML: A Novel Probabilistic Perspective Kernel Mahalanobis Distance Metric Learning Model for Semi-supervised Clustering

General Heterogeneous Transfer Distance Metric Learning Via Knowledge Fragments Transfer.

A Distributed Approach Toward Discriminative Distance Metric Learning

Knowledge Elicitation using Deep Metric Learning and Psychometric Testing

Transferring Knowledge Fragments for Learning Distance Metric from A Heterogeneous Domain

Learning Metrics via Discriminant Kernels and Multidimensional Scaling: Toward Expected Euclidean Representation

Distance-Dependent Metric Learning.

Learning A Mahalanobis Distance Metric for Data Clustering and Classification

EMD Metric Learning