Towards using a Single Uniform Metric in Instance-Based Learning

Kai Ming Ting

DOI: https://doi.org/10.1007/3-540-60598-3_52

1995-01-01

Abstract:In instance-based learning, two different metrics are usually used for continuous-valued attributes and nominal attributes. The problem of using different metrics in domains which have both types of attribute has been mitigated by methods such as attribute and instance weightings in instance-based learning.

What problem does this paper attempt to address?

Instance-Ranking: A New Perspective to Consider the Instance Dependency for Classification

Xin Xia,Xiaohu Yang,Shanping Li,Chao Wu

DOI: https://doi.org/10.1007/978-3-642-36778-6_10

2012-01-01

Abstract:Single-label classification refers to the task to predict an instance to be one unique label in a set of labels. Different from single-label classification, for multi-label classification, one instance is associated with one or more labels in a set of labels simultaneously. Various works have focused on the algorithms for those two types of classification. Since the ranking problem is always coexisting with the classification problem, and traditional researches mainly assume the uniform distribution for the instances, in this paper, we propose a new perspective for the ranking problem. With the assumption that the distribution for the instance is not uniform, different instances have different influences for the distribution, the Instance-Ranking algorithm is presented. With the Instance- Ranking algorithm, the famous K-nearest-neighbors (KNN) algorithm is modified to confirm the validity of our algorithm. Lastly, the Instance-Ranking algorithm is combined with the ML.KNN algorithm for multi-label classification. Experiment with different datasets show that our Instance-Ranking algorithm achieves better performance than the original state-of-art algorithm such as KNN and ML.KNN.
Instance Specific Metric Subspace Learning: A Bayesian Approach.

Han-Jia Ye,De-Chuan Zhan,Yuan Jiang

DOI: https://doi.org/10.1609/aaai.v30i1.10264

2016-01-01

Proceedings of the AAAI Conference on Artificial Intelligence

Abstract:Instead of using a uniform metric, instance specific distance learning methods assign multiple metrics for different localities, which take data heterogeneity into consideration. Therefore, they may improve the performance of distance based classifiers, e.g., kNN. Existing methods obtain multiple metrics of test data by either transductively assigning metrics for unlabeled instances or designing distance functions manually, which are with limited generalization ability. In this paper, we propose ISMETS (Instance Specific METric Subspace) framework which can automatically span the whole metric space in a generative manner and is able to inductively learn a specific metric subspace for each instance via inferring the expectation over the metric bases in a Bayesian manner. The whole framework can be solved with Variational Bayes (VB). Experiment on synthetic data shows that the learned results are with good interpretability. Moreover, comprehensive results on real world datasets validate the effectiveness and robustness of ISMETS.
Metric Learning Via Penalized Optimization.

Hao Huang,Yanan Peng,Ting Gan,Weiping Tu,Ruiting Zhou,Sai Wu

DOI: https://doi.org/10.1145/3447548.3467369

2021-01-01

Abstract:Metric learning aims to project original data into a new space, where data points can be classified more accurately using kNN or similar types of classification algorithms. To avoid trivial learning results such as indistinguishably projecting the data onto a line, many existing approaches formulate metric learning as a constrained optimization problem, like finding a metric that minimizes the distance between data points from the same class, with a constraint of ensuring a certain separation for data points from different classes, and then they approximate the optimal solution to the constrained optimization in an iterative way. In order to improve the classification accuracy as much as possible, we try to find a metric that is able to minimize the intra-class distance and maximize the inter-class distance simultaneously. Towards this, we formulate metric learning as a penalized optimization problem, and provide design guideline, paradigms with a general formula, as well as two representative instantiations for the penalty term. In addition, we provide an analytical solution for the penalized optimization, with which costly computation can be avoid, and more importantly, there is no need to worry about the convergence rates or approximation ratios any more. Extensive experiments on real-world data sets are conducted, and the results verify the effectiveness and efficiency of our approach.
Multiple Instance Learning with the Optimal Sub-Pattern Assignment Metric

Quang N. Tran,Ba-Ngu Vo,Dinh Phung,Ba-Tuong Vo,Thuong Nguyen

DOI: https://doi.org/10.48550/arXiv.1703.08933

2017-03-27

Abstract:Multiple instance data are sets or multi-sets of unordered elements. Using metrics or distances for sets, we propose an approach to several multiple instance learning tasks, such as clustering (unsupervised learning), classification (supervised learning), and novelty detection (semi-supervised learning). In particular, we introduce the Optimal Sub-Pattern Assignment metric to multiple instance learning so as to provide versatile design choices. Numerical experiments on both simulated and real data are presented to illustrate the versatility of the proposed solution.

Machine Learning
Discretization of Continuous-Valued Attributes and Instance-Based Learning

Kai Ming Ting

1994-01-01

Abstract: Recent work on discretization of continuous-valued attributes inlearning decision trees has produced some positive results. This paperadopts the idea of discretization of continuous-valued attributesand applies it to instance-based learning (Aha, 1990; Aha, Kibler &Albert, 1991).Our experiments have shown that instance-based learning (IBL)usually performs well in continuous-valued attribute domains andpoorly in nominal attribute domains. Cost and Salzberg (1993) havedevised the modified ...
What Makes Objects Similar: A Unified Multi-Metric Learning Approach.

Han-Jia Ye,De-Chuan Zhan,Xue-Min Si,Yuan Jiang,Zhi-Hua Zhou

DOI: https://doi.org/10.1109/tpami.2018.2829192

IF: 23.6

2018-01-01

IEEE Transactions on Pattern Analysis and Machine Intelligence

Abstract:Linkages are essentially determined by similarity measures that may be derived from multiple perspectives. For example, spatial linkages are usually generated based on localities of heterogeneous data. Semantic linkages, however, can come from even more properties, such as different physical meanings behind social relations. Many existing metric learning models focus on spatial linkages but leave the rich semantic factors unconsidered. We propose a Unified Multi-Metric Learning (Um$^2$2l) framework to exploit multiple types of metrics with respect to overdetermined similarities between linkages. In Um$^2$2l, types of combination operators are introduced for distance characterization from multiple perspectives, and thus can introduce flexibilities for representing and utilizing both spatial and semantic linkages. Besides, we propose a uniform solver for Um$^2$2l, and the theoretical analysis reflects the generalization ability of Um$^2$2l as well. Extensive experiments on diverse applications exhibit the superior classification performance and comprehensibility of Um$^2$2l. Visualization results also validate its ability to physical meanings discovery.
Learning instance specific distances using metric propagation

De-Chuan Zhan,Ming Li,Yu-Feng Li,Zhi-Hua Zhou

DOI: https://doi.org/10.1145/1553374.1553530

2009-01-01

Abstract:In many real-world applications, such as image retrieval, it would be natural to measure the distances from one instance to others using instance specific distance which captures the distinctions from the perspective of the concerned instance. However, there is no complete framework for learning instance specific distances since existing methods are incapable of learning such distances for test instance and unlabeled data. In this paper, we propose the Isd method to address this issue. The key of Isd is metric propagation, that is, propagating and adapting metrics of individual labeled examples to individual unlabeled instances. We formulate the problem into a convex optimization framework and derive efficient solutions. Experiments show that Isd can effectively learn instance specific distances for labeled as well as unlabeled instances. The metric propagation scheme can also be used in other scenarios.
A multi-birth metric learning framework based on binary constraints

QiangQiang Ren,Chao Yuan,Yifeng Zhao,Liming Yang

DOI: https://doi.org/10.1016/j.neunet.2022.07.004

Abstract:Multi-metric learning plays a significant role in improving the generalization of algorithms related to distance metrics since using a single metric is sometimes insufficient to handle complex data. Metric learning can adjust automatically the distance between samples to make the intra-class samples compact while making the inter-class distance as far as possible. To implement this intention better,in this work, we propose a novel multi-metric learning framework based on the pair constraints instead of triple constraints to reduce computational burden. To solve effectively the problem, we first propose a multi-birth metric learning model (termed MBML), where for each class sample, the global metric and a local metric are jointly trained. Both global and local structural information are adapted to better depict sample information. Then two alternating iterative algorithms are developed to optimize the MBML. The convergence of the proposed algorithm and complexity are analyzed theoretically. Moreover, a fast diagonal multi-metric learning method is proposed based on binary constraints, and problem can be reformulated a linear programming, with fast training speed, low the computational burden and the global optimal solutions. Numerical experiments are carried out on different scales and different types of datasets including an artificial data, benchmark datasets and an image database from binary class and multi-class problems. Experiment results confirm the feasibility and effectiveness of the proposed methods.
Learning Multiple Local Metrics: Global Consideration Helps.

Han-Jia Ye,D C Zhan,Nan Li,Yuan Jiang

DOI: https://doi.org/10.1109/TPAMI.2019.2901675

IF: 23.6

2020-01-01

IEEE Transactions on Pattern Analysis and Machine Intelligence

Abstract:Learning distance metric between objects provides a better measurement for their relative comparisons. Due to the complex properties inside or between heterogeneous objects, multiple local metrics become an essential representation tool to depict various local characteristics of examples. Different from existing methods building more than one local metric directly, however in this paper, we emphas...
An Efficient Metric of Automatic Weight Generation for Properties in Instance Matching Technique

Md. Hanif Seddiqui,Rudra Pratap Deb Nath,Masaki Aono

DOI: https://doi.org/10.5121/ijwest.2015.6101

2015-02-12

Abstract:The proliferation of heterogeneous data sources of semantic knowledge base intensifies the need of an automatic instance matching technique. However, the efficiency of instance matching is often influenced by the weight of a property associated to instances. Automatic weight generation is a non-trivial, however an important task in instance matching technique. Therefore, identifying an appropriate metric for generating weight for a property automatically is nevertheless a formidable task. In this paper, we investigate an approach of generating weights automatically by considering hypotheses: (1) the weight of a property is directly proportional to the ratio of the number of its distinct values to the number of instances contain the property, and (2) the weight is also proportional to the ratio of the number of distinct values of a property to the number of instances in a training dataset. The basic intuition behind the use of our approach is the classical theory of information content that infrequent words are more informative than frequent ones. Our mathematical model derives a metric for generating property weights automatically, which is applied in instance matching system to produce re-conciliated instances efficiently. Our experiments and evaluations show the effectiveness of our proposed metric of automatic weight generation for properties in an instance matching technique.

Artificial Intelligence
Metric learning for monotonic classification: turning the space up to the limits of monotonicity

DOI: https://doi.org/10.1007/s10489-024-05371-8

IF: 5.3

2024-03-27

Applied Intelligence

Abstract:This paper presents, for the first time, a distance metric learning algorithm for monotonic classification. Monotonic datasets arise in many real-world applications, where there exist order relations in the input and output variables, and the outputs corresponding to ordered pairs of inputs are also expected to be ordered. Monotonic classification can be addressed through several distance-based classifiers that are able to respect the monotonicity constraints of the data. The performance of distance-based classifiers can be improved with the use of distance metric learning algorithms, which are able to find the distances that best represent the similarities among each pair of data samples. However, learning a distance for monotonic data has an additional drawback: the learned distance may negatively impact the monotonic constraints of the data. In our work, we propose a new model for learning distances that does not corrupt these constraints. This methodology will also be useful in identifying and discarding non-monotonic pairs of samples that may be present in the data due to noise. The experimental analysis conducted, supported by a Bayesian statistical testing, demonstrates that the distances obtained by the proposed method can enhance the performance of several distance-based classifiers in monotonic problems.

computer science, artificial intelligence
Multiple Instance Learning Via Distance Metric Optimization

Haifeng Zhao,Jun Cheng,Jun Jiang,Dacheng Tao

DOI: https://doi.org/10.1109/icip.2013.6738539

2013-01-01

Abstract:Multiple Instance Learning (MIL) has been widely applied in practice, such as drug activity prediction, content-based image retrieval. In MIL, a sample, comprised of a set of instances, is called a bag. Labels are assigned to bags instead of instances. The uncertainty of labels on instances makes MIL different from conventional supervised single instance learning (SIL) tasks. Therefore, it is critical to learn an effective mapping to convert an MIL task to an SIL task. In this paper, we present OptMILES by learning the optimal transformation on the bag-to-instance similarity measure, exploring the optimal distance metric between instances, by an alternating minimization training procedure. We thoroughly evaluate the proposed method on both a synthetic dataset and real world datasets by comparing with representative MIL algorithms. The experimental results suggest the effectiveness of OptMILES.
Two-Level Multi-Task Metric Learning with Application to Multi-Classification

Hong Liu,Xuewu Zhang,Pingping Wu

DOI: https://doi.org/10.1109/icip.2015.7351304

2015-01-01

Abstract:Many metric learning approaches neglect that the real world multi-class problems share strong visual similarities, which can be exploited by learning discriminative models. In this paper, a Two-level Multi-task Metric Learning (TMTL) method is presented to learn a distance measure from equivalence constraints. Multiple features are adopted to represent the image information and learn the distance matrices in the first level. Then the task-specific learning paradigm and multi-task voting mechanism make full use of pairwise equivalence labels, which induces knowledge from anonymous pairs to multi-classification. Experiments are conducted on two challenging benchmarks PubFig and OuluVS for face identification and lipreading respectively. The results demonstrate that our method outperforms the recent multi-task learning approaches and multi-class support vector machine.
Local Large-Margin Multi-Metric Learning for Face and Kinship Verification

Junlin Hu,Jiwen Lu,Junsong Yuan,Yap-Peng Tan

DOI: https://doi.org/10.1109/tcsvt.2017.2691801

IF: 5.859

2017-01-01

IEEE Transactions on Circuits and Systems for Video Technology

Abstract:Metric learning has attracted wide attention in face and kinship verification, and a number of such algorithms have been presented over the past few years. However, most existing metric learning methods learn only one Mahalanobis distance metric from a single feature representation for each face image and cannot make use of multiple feature representations directly. In many face-related tasks, we can easily extract multiple features for a face image to extract more complementary information, and it is desirable to learn distance metrics from these multiple features, so that more discriminative information can be exploited than those learned from individual features. To achieve this, we present a large-margin multi-metric learning (LM3L) method for face and kinship verification, which jointly learns multiple global distance metrics under which the correlations of different feature representations of each sample are maximized, and the distance of each positive pair is less than a low threshold and that of each negative pair is greater than a high threshold. To better exploit the local structures of face images, we also propose a local metric learning and local LM3Lmethods to learn a set of local metrics. Experimental results on three face data sets show that the proposed methods achieve very competitive results compared with the state-of-the-art methods.
MetricMask: Single category instance segmentation by metric learning

Yang Wang,Wanlin Zhou,Qinwei Lv,Guangle Yao

DOI: https://doi.org/10.1016/j.neucom.2022.05.117

IF: 6

2022-08-21

Neurocomputing

Abstract:In this paper, we introduce a novel single category instance segmentation method, termed MetricMask, which can be used by easily embedding it into most off-the-shelf detection and segmentation methods. Currently, instance segmentation methods usually segment each instance after completing the object detection. Our method can implement segmentation for all instances at once by fusing object detection, semantic segmentation, and metric learning. The contributions are threefold, 1. The instance segmentation is converted to three parallel tasks, including bounding box regression, mask regression, and embedding vector of pixel regression. 2. Random sampling metric loss is proposed to optimize embedding vectors, saving GPU memory without losing instance segmentation accuracy. 3. Based on the model output(bounding box, embedding vector of pixel, and the category-level mask), we propose a Metric Operation to segment each instance. Finally, we conduct experiments on two standard datasets, including the COCO person dataset and the ISOD dataset. The experiment results show that our method has excellent competitiveness with other methods. In particular, our method has an excellent performance on the large object and outperforms the existing state-of-the-art competitors on the ISOD dataset. We hope the MetricMask of our proposed will provide a new method for instance segmentation.

computer science, artificial intelligence
On Combining Side Information and Unlabeled Data for Heterogeneous Multi-Task Metric Learning.

Yong Luo,Yonggang Wen,Dacheng Tao

2016-01-01

Abstract:Distance metric learning (DML) is critical for a wide variety of machine learning algorithms and pattern recognition applications. Transfer metric learning (TML) leverages the side information (e.g., similar/dissimilar constraints over pairs of samples) from related domains to help the target metric learning (with limited information). Current TML tools usually assume that different domains exploit the same feature representation, and thus are not applicable to tasks where data are drawn from heterogeneous domains. Heterogeneous transfer learning approaches handle heterogeneous domains by usually learning feature transformations across different domains. The learned transformation can be used to derive a metric, but these approaches are mostly limited by their capability of only handling two domains. This motivates the proposed heterogeneous multi-task metric learning (HMTML) framework for handling multiple domains by combining side information and unlabeled data. Specifically, HMTML learns the metrics for all different domains simultaneously by maximizing their high-order correlation (parameterized by feature covariance of unlabeled data) in a common subspace, which is induced by the transformations derived from the metrics. Extensive experiments on both multi-language text categorization and multi-view social image annotation demonstrate the effectiveness of the proposed method.
Heterogeneous Multitask Metric Learning Across Multiple Domains

Yong Luo,Yonggang Wen,Dacheng Tao

DOI: https://doi.org/10.1109/tnnls.2017.2750321

IF: 14.255

2018-01-01

IEEE Transactions on Neural Networks and Learning Systems

Abstract:Distance metric learning plays a crucial role in diverse machine learning algorithms and applications. When the labeled information in a target domain is limited, transfer metric learning (TML) helps to learn the metric by leveraging the sufficient information from other related domains. Multitask metric learning (MTML), which can be regarded as a special case of TML, performs transfer across all related domains. Current TML tools usually assume that the same feature representation is exploited for different domains. However, in real-world applications, data may be drawn from heterogeneous domains. Heterogeneous transfer learning approaches can be adopted to remedy this drawback by deriving a metric from the learned transformation across different domains. However, they are often limited in that only two domains can be handled. To appropriately handle multiple domains, we develop a novel heterogeneous MTML (HMTML) framework. In HMTML, the metrics of all different domains are learned together. The transformations derived from the metrics are utilized to induce a common subspace, and the high-order covariance among the predictive structures of these domains is maximized in this subspace. There do exist a few heterogeneous transfer learning approaches that deal with multiple domains, but the high-order statistics (correlation information), which can only be exploited by simultaneously examining all domains, is ignored in these approaches. Compared with them, the proposed HMTML can effectively explore such high-order information, thus obtaining more reliable feature transformations and metrics. Effectiveness of our method is validated by the extensive and intensive experiments on text categorization, scene classification, and social image annotation.
Lifelong Metric Learning

Gan Sun,Yang Cong,Ji Liu,Xiaowei Xu

DOI: https://doi.org/10.48550/arXiv.1705.01209

2017-06-12

Abstract:The state-of-the-art online learning approaches are only capable of learning the metric for predefined tasks. In this paper, we consider lifelong learning problem to mimic "human learning", i.e., endowing a new capability to the learned metric for a new task from new online samples and incorporating previous experiences and knowledge. Therefore, we propose a new metric learning framework: lifelong metric learning (LML), which only utilizes the data of the new task to train the metric model while preserving the original capabilities. More specifically, the proposed LML maintains a common subspace for all learned metrics, named lifelong dictionary, transfers knowledge from the common subspace to each new metric task with task-specific idiosyncrasy, and redefines the common subspace over time to maximize performance across all metric tasks. For model optimization, we apply online passive aggressive optimization algorithm to solve the proposed LML framework, where the lifelong dictionary and task-specific partition are optimized alternatively and consecutively. Finally, we evaluate our approach by analyzing several multi-task metric learning datasets. Extensive experimental results demonstrate effectiveness and efficiency of the proposed framework.

Machine Learning,Artificial Intelligence
Local Metric Learning Based on Anchor Points for Multimedia Analysis

Chunbin Gu,Jiajun Bu,Keyue Shi,Zhi Yu,Beidou Wang,Liangcheng Li

DOI: https://doi.org/10.1109/icme.2019.00237

2019-01-01

Abstract:Distance metric learning has been shown to be an effective and efficient method which can lead to significant improvements in classification, clustering, and retrieval. Conventional metric learning methods which apply a global metric to capture the input features and their correlations are unsuitable for heterogeneous data, whose features vary according to different clusters. Thus, a number of local metric learning methods have been proposed to overcome this limitation. Unfortunately, most of them suffer from over-fitting or need the data set to be under a strong assumption, such as Gaussian distribution. To avoid over-fitting caused by learning local metric matrices separately in the previous researches, we propose a novel local metric learning method in which we learn a metric matrix for each pair of instances as linear combinations of basis metric matrices defined on different clusters of the input space without any assumption for data set. We get the parameters of the linear combinations by using the relationship between direct distance and anchor~distance between a pair of data points. Additionally, we design a novel expectation-maximization based optimization method to learn the linear combinations and basis metric matrices. Experimental results show that our method outperforms state-of-the-art methods on various data sets.
Instance Selection Improves Geometric Mean Accuracy: A Study on Imbalanced Data Classification

Ludmila I. Kuncheva,Álvar Arnaiz-González,José-Francisco Díez-Pastor,Iain A. D. Gunn

DOI: https://doi.org/10.48550/arXiv.1804.07155

2018-04-19

Abstract:A natural way of handling imbalanced data is to attempt to equalise the class frequencies and train the classifier of choice on balanced data. For two-class imbalanced problems, the classification success is typically measured by the geometric mean (GM) of the true positive and true negative rates. Here we prove that GM can be improved upon by instance selection, and give the theoretical conditions for such an improvement. We demonstrate that GM is non-monotonic with respect to the number of retained instances, which discourages systematic instance selection. We also show that balancing the distribution frequencies is inferior to a direct maximisation of GM. To verify our theoretical findings, we carried out an experimental study of 12 instance selection methods for imbalanced data, using 66 standard benchmark data sets. The results reveal possible room for new instance selection methods for imbalanced data.

Computer Vision and Pattern Recognition,Machine Learning

Towards using a Single Uniform Metric in Instance-Based Learning

Instance-Ranking: A New Perspective to Consider the Instance Dependency for Classification

Instance Specific Metric Subspace Learning: A Bayesian Approach.

Metric Learning Via Penalized Optimization.

Multiple Instance Learning with the Optimal Sub-Pattern Assignment Metric

Discretization of Continuous-Valued Attributes and Instance-Based Learning

What Makes Objects Similar: A Unified Multi-Metric Learning Approach.

Learning instance specific distances using metric propagation

A multi-birth metric learning framework based on binary constraints

Learning Multiple Local Metrics: Global Consideration Helps.

An Efficient Metric of Automatic Weight Generation for Properties in Instance Matching Technique

Metric learning for monotonic classification: turning the space up to the limits of monotonicity

Multiple Instance Learning Via Distance Metric Optimization

Two-Level Multi-Task Metric Learning with Application to Multi-Classification

Local Large-Margin Multi-Metric Learning for Face and Kinship Verification

MetricMask: Single category instance segmentation by metric learning

On Combining Side Information and Unlabeled Data for Heterogeneous Multi-Task Metric Learning.

Heterogeneous Multitask Metric Learning Across Multiple Domains

Lifelong Metric Learning

Local Metric Learning Based on Anchor Points for Multimedia Analysis

Instance Selection Improves Geometric Mean Accuracy: A Study on Imbalanced Data Classification