From classifiers to discriminators

YangJian,ZhangLei,YangJing-yu,ZhangDavid

IF: 8

2011-01-01

Pattern Recognition

Abstract:The current discriminant analysis method design is generally independent of classifiers, thus the connection between discriminant analysis methods and classifiers is loose. This paper provides a wa...

What problem does this paper attempt to address?

Discriminative analysis of direct linear discriminant analysis for face recognition

赵武锋,沈海斌,严晓浪

DOI: https://doi.org/10.3785/j.issn.1008-973X.2010.08.008

2010-01-01

Abstract:Direct linear discriminant analysis (DLDA) is claimed that it can take advantage of all the information, both within and outside of the within-class scatter matrix's null space. In order to analyze this claim's flaw in theory, the discriminative characteristics of DLDA for face recognition was studied. Since the optimal solution of DLDA is inside the range space of the between-class scatter matrix, a theoretical analysis was unfold via the following three aspects: the relationship between the within-class and between-class scatter matrix's range space, the relationship between the within-class scatter matrix's null space and the between-class scatter matrix's range space, the characteristics of DLDA under keeping all the discriminative vectors. The results show that: in undersampled cases DLDA nearly can not make use of the information inside the null space of the within-class scatter matrix, thus some discriminative information may be lost; DLDA is degenerated as PCA of the between-class scatter matrix with all nonzero principal components if it keeps the complete discriminative vectors found. The comparative results on the face database, ORL and YALE, indicate that DLDA is inferior to the other extensions of linear discriminant analysis in terms of recognition accuracy. Which is consistent with the theoretical analysis.
Computational and Theoretical Analysis of Supervised Dimensionality Reduction

赵武锋,沈海斌,严晓浪

DOI: https://doi.org/10.3785/j.issn.1008-9497.2009.06.015

2009-01-01

Abstract:In the case of small sample size,classical linear discriminant analysis fails due to the singularity of scatter matrices for supervised dimensionality reduction.Many extensions were proposed in the past to overcome this problem,which in general can be classified into three categories: methods based on the null space of the within-class scatter matrix,ones based on the range space of the total scatter matrix and ones based on other subspace.In order to better understand the characteristics of the algorithms of the former two classes,a computational and theoretical analysis was carried out,and concluded that: under a mild condition which holds in many applications involving high-dimensional data,methods based on the null space of the within-class scatter matrix are equivalent to those based on the range space of the total scatter matrix except two differences of constraints for discriminant vectors and implementation procedure.The comparative results on the face database,ORL and YALE,also confirmed the aforementioned conclusion.
Class-Aware Analysis Dictionary Learning for Pattern Classification.

Jiujun Wang,Yanqing Guo,Jun Guo,Xiangyang Luo,Xiangwei Kong

DOI: https://doi.org/10.1109/lsp.2017.2734860

2017-01-01

IEEE Signal Processing Letters

Abstract:Dictionary learning (DL) plays an important role in pattern classification. However, learning a discriminative dictionary has not been well addressed in analysis dictionary learning (ADL). This letter proposes a Class-aware Analysis Dictionary Learning (CADL) model to improve the classification performance of conventional ADL. The objective function of CADL mainly includes two parts to promote the discriminability. The first part aims to learn a discriminative analysis subdictionary for each class instead of a global dictionary for all classes. The learned analysis dictionary is class-aware, generating a block-diagonal coding coefficient matrix. The second part aims to enhance the discrimination of coding coefficients by integrating a max-margin regularization term into our proposed framework. This term ensures the coefficients of different classes to be separated by a max-margin, which boosts the confidence of classification. A theoretical analysis is also given to support the max-margin regularization term from the perspective of preserving the pairwise relations of samples in coding space. We employ an alternating minimization algorithm to iteratively find the convergent solution. By evaluating our method on four pattern classification datasets, we demonstrate the superiority of our CADL method to the state-of-the-art DL methods.
Complete discriminant analysis

HUO HongWen,FENG JuFu

DOI: https://doi.org/10.1360/112011-494

2012-01-01

Abstract:This paper proposes a complete discriminant analysis algorithm to solve the problem that existing linear discriminative analysis algorithms always miss discriminant information when reducing dimensions,and the problem that it is difficult to juggle discriminant information both in rank space and null space.This algorithm makes full use of all discriminant information in all space,by constructing a new subspace and its discriminant matrix.The dimension of the new subspace is low and the algorithm is simple and costless.The experimental results demonstrate that this algorithm has better performance and efficiency than traditional discriminant analysis algorithms.
A procedure of linear discrimination analysis with detected sparsity structure for high-dimensional multi-class classification

Shan Luo,Zehua Chen

DOI: https://doi.org/10.1016/j.jmva.2020.104641

IF: 1.387

2020-01-01

Journal of Multivariate Analysis

Abstract:In this article, we consider discrimination analyses in high-dimensional cases where the dimension of the predictor vector diverges with the sample size in a theoretical setting. The emphasis is on the case where the number of classes is bigger than two. We first deal with the asymptotic misclassification rates of linear discrimination rules under various conditions. In practical high-dimensional classification problems, it is reasonable to assume certain sparsity conditions on the class means and the common precision matrix. Our theoretical study reveals that with known sparsity structures an asymptotically optimal linear discrimination rule can be constructed. Motivated by the theoretical result, we propose a linear discrimination rule constructed based on estimated sparsity structures which is dubbed as linear discrimination with detected sparsity (LDwDS). The asymptotic optimality of LDwDS is established. Numerical studies are carried out for the comparison of LDwDS with other existing methods. The numerical studies include a comprehensive simulation study and two real data analyses. The numerical studies demonstrate that the LDwDS has an edge in terms of misclassification rate over all the other methods under consideration in the comparison. (C) 2020 Elsevier Inc. All rights reserved.
Constrained Linear Discrimination Analysis and Face Recognition

LI Xiao-run,Guangzhou Zhao,ZHAO Liao-ying

2008-01-01

Abstract:A constrained linear discrimination analysis method was proposed for the feature extraction in the pattern recognition of problems with high dimension and small samples. Applying whitening process and Gram-Schimdt orthogonalization and orthogonal subspace projection, an optimal transformation matrix was designed to minimize the ratio of intra-class distance to inter-class distance while imposing the constraint that different class centers after transformation are along specifically directions that are orthogonal each other. For the small sample problem of face recognition, the whitening process was realized by singular value decomposition. The experimental results using the ORL and the UMIST face image database demonstrate that the effectiveness and performance of CLDA is approximate with some Fisher discrimination analysis such as direct Fisher discrimination analysis (DDA) and regularized Fisher discriminant analysis (R-LDA).
Annealed discriminant analysis

Gang Wang,Zhihua Zhang,Frederick H. Lochovsky

DOI: https://doi.org/10.1007/11564096_43

2005-01-01

Abstract:Motivated by the analogies to statistical physics, the deterministic annealing (DA) method has successfully been demonstrated in a variety of applications. In this paper, we explore a new methodology to devise the classifier under the DA method. The differential cost function is derived subject to a constraint on the randomness of the solution, which is governed by the temperature T. While gradually lowering the temperature, we can always find a good solution which can both solve the overfitting problem and avoid poor local optima. Our approach is called annealed discriminant analysis (ADA). It is a general approach, where we elaborate two classifiers, i.e., distance-based and inner product-based, in this paper. The distance-based classifier is an annealed version of linear discriminant analysis (LDA) while the inner product-based classifier is a generalization of penalized logistic regression (PLR). As such, ADA provides new insights into the workings of these two classification algorithms. The experimental results show substantial performance gains over standard learning methods.
A reformative kernel Fisher discriminant analysis

Yong Xu,Jing-yu Yang,Jian Yang

DOI: https://doi.org/10.1016/j.patcog.2003.10.006

IF: 8

2004-01-01

Pattern Recognition

Abstract:A reformative kernel Fisher discriminant method is proposed, which is directly derived from the naive kernel Fisher discriminant analysis with superiority in classification efficiency. In the novel method only a part of training patterns, called “significant nodes”, are necessary to be adopted in classifying one test pattern. A recursive algorithm for selecting “significant nodes”, which is the key of the novel method, is presented in detail. The experiment on benchmarks shows that the novel method is effective and much efficient in classifying.
A discriminative model selection approach and its application to text classification

Lungan Zhang,Liangxiao Jiang,Chaoqun Li

DOI: https://doi.org/10.1007/s00521-017-3151-0

2017-07-15

Neural Computing and Applications

Abstract:Classification is one of the fundamental problems in data mining, in which a classification algorithm attempts to construct a classifier from a given set of training instances with class labels. It is well known that some classification algorithms perform very well on some domains, and poorly on others. For example, NB performs well on some domains, and poorly on others that involve correlated features. C4.5, on the other hand, typically works better than NB on such domains. To integrate their advantages and avoid their disadvantages, many model hybrid approaches, such as model insertion and model combination, are proposed. In this paper, we focus on a novel view and propose a discriminative model selection approach, called discriminative model selection (DMS). DMS discriminatively chooses different single models for different test instances and retains the interpretability of single models. Empirical studies on a collection of 36 classification problems from the University of California at Irvine repository show that our discriminative model selection approach outperforms single models, model insertion approaches and model combination approaches. Besides, we apply the proposed discriminative model selection approach to some state-of-the-art naive Bayes text classifiers and also improve their performance.

computer science, artificial intelligence
A Generalized Form of Fisher Linear Discriminant Function

Zhengdong Cheng,ZHANG Yu-Jin,FAN Xiang

DOI: https://doi.org/10.3969/j.issn.1003-6059.2009.02.002

2009-01-01

Pattern Recognition and Artificial Intelligence

Abstract:A generalized form of Fisher discriminant function is presented.It overcomes the limitations of two common discriminant functions.The presented form uniforms the discriminant functions in two subspaces of the dual subspace discriminant analysis(DSDA).A new orthogonal discriminant vector set is obtained by QR decomposition,and its discriminant property is approximate to that of the Foley-Sammon orthogonal discriminant vector set with smaller computational complexity.The experiments on ORL and JAFFE database show that theory analysis is consistent to the experimental results.
A Structurally Motivated Framework for Discriminant Analysis

Bo Yang,Songcan Chen,Xindong Wu

DOI: https://doi.org/10.1007/s10044-011-0228-8

2011-01-01

Abstract:Over the last few years, a lot of algorithms for discriminant analysis (DA) have been developed. Although having different motivations, they all inject structure information in data into their own within- and between-class scatters. However, to our best knowledge, there has not been yet a systematical examination about (1) which structure granularities lurk in data; (2) which structure granularities are utilized in scatters of a DA algorithm; (3) whether new DA algorithms can be developed based on existing structure granularities. In this paper, the established so-called structurally motivated (SM) framework for DA and its unified mathematical formulation of the ratio trace exactly answers them. It categorizes these DA algorithms from the viewpoint of constructing scatters based on different-granularity structures in data, identifies their applicable scenarios for different structure types, and provides insights into developing new DA algorithms. Inspired by the insight, we find that cluster granularity lying in the middle of granularity spectrum in SM framework can still be further utilized and exploited. As a result, the three DA algorithms based on the cluster granularity are derived from the SM framework and from the injection of the cluster structure information into the respective within-class, between-class and joint both scatter matrices for the classical MDA, and these corresponding algorithms are, respectively, called as SWDA, SBDA and SWBDA. The injection of cluster structure information makes the proposed three algorithms able to fit relatively complicated data not only more effectively, but also with the regularization technique obtain more projections than the classical MDA, which is very helpful for more effective DA. Moreover, MDA becomes their special case when the cluster numbers of all classes are set to 1. Our experiments on the benchmarks (face and UCI databases) here show that the proposed algorithms yield encouraging results.
Network linear discriminant analysis

Wei Cai,Guoyu Guan,Rui Pan,Xuening Zhu,Hansheng Wang

DOI: https://doi.org/10.1016/j.csda.2017.07.007

IF: 2.035

2018-01-01

Computational Statistics & Data Analysis

Abstract:Linear discriminant analysis (LDA) is one of the most popularly used classification methods. With the rapid advance of information technology, network data are becoming increasingly available. A novel method called network linear discriminant analysis (NLDA) is proposed to deal with the classification problem for network data. The NLDA model takes both network information and predictive variables into consideration. Theoretically, the misclassification rate is studied and an upper bound is derived under mild conditions. Furthermore, it is observed that real networks are often sparse in structure. As a result, asymptotic performance of NLDA is also obtained under certain sparsity assumptions. In order to evaluate the finite sample performance of the newly proposed methodology, a number of simulation studies are conducted. Lastly, a real data analysis about Sina Weibo is also presented for illustration purpose.
Discriminant analysis for repeated measures data: a review

Lisa M. Lix,Tolulope T. Sajobi,Lisa Lix

DOI: https://doi.org/10.3389/fpsyg.2010.00146

IF: 3.8

2010-01-01

Frontiers in Psychology

Abstract:Discriminant analysis (DA) encompasses procedures for classifying observations into groups (i.e., predictive discriminative analysis) and describing the relative importance of variables for distinguishing amongst groups (i.e., descriptive discriminative analysis). In recent years, a number of developments have occurred in DA procedures for the analysis of data from repeated measures designs. Specifically, DA procedures have been developed for repeated measures data characterized by missing observations and/or unbalanced measurement occasions, as well as high-dimensional data in which measurements are collected repeatedly on two or more variables. This paper reviews the literature on DA procedures for univariate and multivariate repeated measures data, focusing on covariance pattern and linear mixed-effects models. A numeric example illustrates their implementation using SAS software.

psychology, multidisciplinary
Discriminant analysis for perceptionally comparable classes.

Bingpeng Ma,Shiguang Shan,Xilin Chen,Wen Gao

DOI: https://doi.org/10.1109/AFGR.2008.4813422

2008-01-01

Abstract:Abstract Traditional discriminate analysis treats all the involved classes equally in the computation,of the between-class scatter matrix. However, we find that for many vision tasks, the classes to be processed are not equal in per- ception, i.e. a distance metric can be defined between the classes. Typical examples include head pose classification and age estimation. Aiming at this category of classification problem, this paper proposes a novel discriminant analysis method, called Class Distance based Discriminant Analy- sis (CDDA). In CDDA, the perceptional distance between two classes is exploited to weight the outer product in the between-class scatter computation, to concentrate more on the classes difficult to separate. Another novelty of CDDA is that to preserve the within-class local structure of mul- timodal labeled data, the within-class scatter is re-defined by complementing,the similarity of the samples pairs in the nearby classes. The method,is then applied to head pose classification and age estimation problem, and experimen- tal results demonstrate the effectiveness of CDDA.
On Discriminative Semi-Supervised Classification.

Fei Wang,Changshui Zhang

2008-01-01

Abstract:The recent years have witnessed a surge of interests in semi-supervised learning methods. A common strategy for these algorithms is to require that the predicted data labels should be sufficiently smooth with respect to the intrinsic data manifold. In this paper, we argue that rather than penalizing the label smoothness, we can directly punish the discriminality of the classification function to achieve a more powerful predictor, and we derive two specific algorithms: Semi-Supervised Discriminative Regularization (SSDR) and Semi-parametric Discriminative Semi-supervised Classification (SDSC). Finally many experimental results are presented to show the effectiveness of our method.
Discriminative analysis algorithm for linear local tangent spaces

蒋胜利,张军英,许进

DOI: https://doi.org/10.13245/j.hust.2010.10.022

2010-01-01

Abstract:A novel algorithm, called linear local tangent space discriminant analysis (LLTSDA), is proposed for classification, which is motivated by the desire to preserve the local geometry structure of data and to maximize the classification margin of data. The proposed algorithm is a supervised and modified manifold learning algorithm. The matrix of local tangent space alignment preserves the local geometry structure of data in low-dimensional embedded space; while the scatter matrix can ensure the within-class scatter minimization and the between-class scatter maximization; and a trade-off projected vector base can be taken by solving the generalized eigenvalue problem to the summation of above both matrices. Compared with classical classification methods, the experimental results from Yale, UMIST (University of Manchester Institute of Science and Technology) and MIT face databases show that LLTSDA algorithm can extract the more efficient features for face recognition while the dimensionality is reduced, and obtains much higher recognition accuracies and stronger classification power.
Global subclass discriminant analysis

Huan Wan,Hui Wang,Bryan W. Scotney,Jun Liu,Xin Wei

DOI: https://doi.org/10.1016/j.knosys.2023.111010

IF: 8.139

2023-09-22

Knowledge-Based Systems

Abstract:Linear discriminant analysis (LDA) is a powerful supervised dimensionality reduction method for analysing high-dimensional data. However, LDA cannot use locality information in data, which makes LDA degrade dramatically in performance on multimodal data. A number of LDA variants have been proposed to exploit locality information in data, including subclass-based LDAs. We discover a problem with these variants, which is that subclasses are selected on a within-class basis without considering other classes. This causes the loss of important information at class boundaries. In this paper, we present a novel variant of subclass-based LDA, Global Subclass Discriminant Analysis (GSDA). Unlike other subclass-based LDAs, GSDA selects subclasses from global clusters that may cross class boundaries, thus utilising within-class information and between-class information. More specifically, GSDA applies an effective clustering algorithm to the whole data to construct global clusters. It then utilises the local structure refining strategy on these global clusters to construct subclasses. Finally, GSDA learns a representative data subspace by maximising inter-subclass distance and minimising intra-subclass distance simultaneously. GSDA is extensively evaluated on a wide range of public datasets through comparison with the state-of-the-art LDA algorithms. Experimental results demonstrate its superiority in terms of accuracy and run times.

computer science, artificial intelligence
Neighborhood linear discriminant analysis

Fa Zhu,Junbin Gao,Jian Yang,Ning Ye

DOI: https://doi.org/10.1016/j.patcog.2021.108422

IF: 8

2022-03-01

Pattern Recognition

Abstract:Linear Discriminant Analysis (LDA) assumes that all samples from the same class are independently and identically distributed (i.i.d.). LDA may fail in the cases where the assumption does not hold. Particularly when a class contains several clusters (or subclasses), LDA cannot correctly depict the internal structure as the scatter matrices that LDA relies on are defined at the class level. In order to mitigate the problem, this paper proposes a neighborhood linear discriminant analysis (nLDA) in which the scatter matrices are defined on a neighborhood consisting of reverse nearest neighbors. Thus, the new discriminator does not need an i.i.d. assumption. In addition, the neighborhood can be naturally regarded as the smallest subclass, for which it is easier to be obtained than subclass without resorting to any clustering algorithms. The projected directions are sought to make sure that the within-neighborhood scatter as small as possible and the between-neighborhood scatter as large as possible, simultaneously. The experimental results show that nLDA performs significantly better than previous discriminators, such as LDA, LFDA, ccLDA, LM-NNDA, and l 2 , 1 -RLDA.

computer science, artificial intelligence,engineering, electrical & electronic
Sparse tensor discriminant analysis

Zhihui Lai,Yong Xu,Jian Yang,Jinhui Tang,David Zhang

DOI: https://doi.org/10.1109/TIP.2013.2264678

IF: 10.6

2013-01-01

IEEE Transactions on Image Processing

Abstract:The classical linear discriminant analysis has undergone great development and has recently been extended to different cases. In this paper, a novel discriminant subspace learning method called sparse tensor discriminant analysis (STDA) is proposed, which further extends the recently presented multilinear discriminant analysis to a sparse case. Through introducing the $L_{1}$ and $L_{2}$ norms into the objective function of STDA, we can obtain multiple interrelated sparse discriminant subspaces for feature extraction. As there are no closed-form solutions, $k$-mode optimization technique and the $L_{1}$ norm sparse regression are combined to iteratively learn the optimal sparse discriminant subspace along different modes of the tensors. Moreover, each non-zero element in each subspace is selected from the most important variables/factors, and thus STDA has the potential to perform better than other discriminant subspace methods. Extensive experiments on face databases (Yale, FERET, and CMU PIE face databases) and the Weizmann action database show that the proposed STDA algorithm demonstrates the most competitive performance against the compared tensor-based methods, particularly in small sample sizes.
Linear Discriminant Regularized Regression

Xin Bing,Bingqing Li,Marten Wegkamp

2024-02-22

Abstract:Linear Discriminant Analysis (LDA) is an important classification approach. Its simple linear form makes it easy to interpret and it is capable to handle multi-class responses. It is closely related to other classical multivariate statistical techniques, such as Fisher's discriminant analysis, canonical correlation analysis and linear regression. In this paper we strengthen its connection to multivariate response regression by characterizing the explicit relationship between the discriminant directions and the regression coefficient matrix. This key characterization leads to a new regression-based multi-class classification procedure that is flexible enough to deploy any existing structured, regularized, and even non-parametric, regression methods. Moreover, our new formulation is generically easy to analyze compared to existing regression-based LDA procedures. In particular, we provide complete theoretical guarantees for using the widely used $\ell_1$-regularization that has not yet been fully analyzed in the LDA context. Our theoretical findings are corroborated by extensive simulation studies and real data analysis.

Methodology

From classifiers to discriminators

Discriminative analysis of direct linear discriminant analysis for face recognition

Computational and Theoretical Analysis of Supervised Dimensionality Reduction

Class-Aware Analysis Dictionary Learning for Pattern Classification.

Complete discriminant analysis

A procedure of linear discrimination analysis with detected sparsity structure for high-dimensional multi-class classification

Constrained Linear Discrimination Analysis and Face Recognition

Annealed discriminant analysis

A reformative kernel Fisher discriminant analysis

A discriminative model selection approach and its application to text classification

A Generalized Form of Fisher Linear Discriminant Function

A Structurally Motivated Framework for Discriminant Analysis

Network linear discriminant analysis

Discriminant analysis for repeated measures data: a review

Discriminant analysis for perceptionally comparable classes.

On Discriminative Semi-Supervised Classification.

Discriminative analysis algorithm for linear local tangent spaces

Global subclass discriminant analysis

Neighborhood linear discriminant analysis

Sparse tensor discriminant analysis

Linear Discriminant Regularized Regression