Support vector machines classification with a very large-scale taxonomy

LiuTie-Yan,YangYiming,WanHao,ZengHua-Jun,ChenZheng,MaWei-Ying

2005-01-01

Abstract:Very large-scale classification taxonomies typically have hundreds of thousands of categories, deep hierarchies, and skewed category distribution over documents. However, it is still an open questi...

What problem does this paper attempt to address?

Support Vector Machines Classification with a Very Large-Scale Taxonomy

Tie-Yan Liu,Yiming Yang,Hao Wan,Hua-Jun Zeng,Zheng Chen,Wei-Ying Ma

DOI: https://doi.org/10.1145/1089815.1089821

2005-01-01

Abstract:Very large-scale classification taxonomies typically have hundreds of thousands of categories, deep hierarchies, and skewed category distribution over documents. However, it is still an open question whether the state-of-the-art technologies in automated text categorization can scale to (and perform well on) such large taxonomies. In this paper, we report the first evaluation of Support Vector Machines (SVMs) in web-page classification over the full taxonomy of the Yahoo! categories. Our accomplishments include: 1) a data analysis on the Yahoo! taxonomy; 2) the development of a scalable system for large-scale text categorization; 3) theoretical analysis and experimental evaluation of SVMs in hierarchical and non-hierarchical settings for classification; 4) an investigation of threshold tuning algorithms with respect to time complexity and their effect on the classification accuracy of SVMs. We found that, in terms of scalability, the hierarchical use of SVMs is efficient enough for very large-scale classification; however, in terms of effectiveness, the performance of SVMs over the Yahoo! Directory is still far from satisfactory, which indicates that more substantial investigation is needed.
An experimental study on large-scale web categorization.

Tie-Yan Liu,Yiming Yang,Hao Wan,Qian Zhou,Bin Gao,Hua-Jun Zeng,Zheng Chen,Wei-Ying Ma

DOI: https://doi.org/10.1145/1062745.1062891

2005-01-01

Abstract:Taxonomies of the Web typically have hundreds of thousands of categories and skewed category distribution over documents. It is not clear whether existing text classification technologies can perform well on and scale up to such large-scale applications. To understand this, we conducted the evaluation of several representative methods (Support Vector Machines, k-Nearest Neighbor and Naive Bayes) with Yahoo! taxonomies. In particular, we evaluated the effectiveness/efficiency tradeoff in classifiers with hierarchical setting compared to conventional (flat) setting, and tested popular threshold tuning strategies for their scalability and accuracy in large-scale classification problems.
A Multi-Class Large Margin Classifier

Liang Tang,Qi Xuan,Rong Xiong,Tie-jun Wu,Jian Chu

DOI: https://doi.org/10.1631/jzus.a0820122

2009-01-01

Abstract:Currently there are two approaches for a multi-class support vector classifier (SVC). One is to construct and combine several binary classifiers while the other is to directly consider all classes of data in one optimization formulation. For a K-class problem (K > 2), the first approach has to construct at least K classifiers, and the second approach has to solve a much larger optimization problem proportional to K by the algorithms developed so far. In this paper, following the second approach, we present a novel multi-class large margin classifier (MLMC). This new machine can solve K-class problems in one optimization formulation without increasing the size of the quadratic programming (QP) problem proportional to K. This property allows us to construct just one classifier with as few variables in the QP problem as possible to classify multi-class data, and we can gain the advantage of speed from it especially when K is large. Our experiments indicate that MLMC almost works as well as (sometimes better than) many other multi-class SVCs for some benchmark data classification problems, and obtains a reasonable performance in face recognition application on the AR face database.
Deep Classification in Large-Scale Text Hierarchies

Gui-Rong Xue,Dikan Xing,Qiang Yang,Yong Yu

DOI: https://doi.org/10.1145/1390334.1390440

2008-01-01

Abstract:Most classification algorithms are best at categorizing the Web documents into a few categories, such as the top two levels in the Open Directory Project. Such a classification method does not give very detailed topic-related class information for the user because the first two levels are often too coarse. However, classification on a large-scale hierarchy is known to be intractable for many target categories with cross-link relationships among them. In this paper, we propose a novel deep-classification approach to categorize Web documents into categories in a large-scale taxonomy. The approach consists of two stages: a search stage and a classification stage. In the first stage, a category-search algorithm is used to acquire the category candidates for a given document. Based on the category candidates, we prune the large-scale hierarchy to focus our classification effort on a small subset of the original hierarchy. As a result, the classification model is trained on the small subset before being applied to assign the category for a new document. Since the category candidates are sufficiently close to each other in the hierarchy, a statistical-language-model based classifier using n-gram features is exploited. Furthermore, the structure of the taxonomy can be utilized in this stage to improve the performance of classification. We demonstrate the performance of our proposed algorithms on the Open Directory Project with over 130,000 categories. Experimental results show that our proposed approach can reach 51.8% on the measure of Mi-F1 at the 5th level, which is 77.7% improvement over top-down based SVM classification algorithms.
Joint Hierarchical Category Structure Learning and Large-Scale Image Classification

Yanyun Qu,Li Lin,Fumin Shen,Chang Lu,Yang Wu,Yuan Xie,Dacheng Tao

DOI: https://doi.org/10.1109/tip.2016.2615423

IF: 10.6

2017-01-01

IEEE Transactions on Image Processing

Abstract:We investigate the scalable image classification problem with a large number of categories. Hierarchical visual data structures are helpful for improving the efficiency and performance of large-scale multi-class classification. We propose a novel image classification method based on learning hierarchical inter-class structures. Specifically, we first design a fast algorithm to compute the similarity metric between categories, based on which a visual tree is constructed by hierarchical spectral clustering. Using the learned visual tree, a test sample label is efficiently predicted by searching for the best path over the entire tree. The proposed method is extensively evaluated on the ILSVRC2010 and Caltech 256 benchmark datasets. The experimental results show that our method obtains significantly better category hierarchies than other state-of-the-art visual treebased methods and, therefore, much more accurate classification.
Novel Design of Decision-Tree-Based Support Vector Machines Multi-Class Classifier

Liaoying Zhao,Xiaorun Li,Guangzhou Zhao

DOI: https://doi.org/10.1007/978-3-540-74205-0_90

2007-01-01

Abstract:Designing the hierarchical structure is a key issue for the decision-tree-based (DTB) support vector machines multi-class classification. Inter-class separability is an important basis for designing the hierarchical structure. A new method based on vector projection is proposed to measure inter-class separability. Furthermore, two different DTB support vector multi-class classifiers are designed based on the inter-class separability: one is in the structure of DTB-balanced branches and another is in the structure of DTB-one against all. Experiment results on three large-scale data sets indicate that the proposed method speeds up the decision-tree-based support vector machines multi-class classifiers and yields higher precision.
Multi-Label Requirements Classification with Large Taxonomies

Waleed Abdeen,Michael Unterkalmsteiner,Krzysztof Wnuk,Alexandros Chirtoglou,Christoph Schimanski,Heja Goli

2024-06-13

Abstract:Classification aids software development activities by organizing requirements in classes for easier access and retrieval. The majority of requirements classification research has, so far, focused on binary or multi-class classification. Multi-label classification with large taxonomies could aid requirements traceability but is prohibitively costly with supervised training. Hence, we investigate zero-short learning to evaluate the feasibility of multi-label requirements classification with large taxonomies. We associated, together with domain experts from the industry, 129 requirements with 769 labels from taxonomies ranging between 250 and 1183 classes. Then, we conducted a controlled experiment to study the impact of the type of classifier, the hierarchy, and the structural characteristics of taxonomies on the classification performance. The results show that: (1) The sentence-based classifier had a significantly higher recall compared to the word-based classifier; however, the precision and F1-score did not improve significantly. (2) The hierarchical classification strategy did not always improve the performance of requirements classification. (3) The total and leaf nodes of the taxonomies have a strong negative correlation with the recall of the hierarchical sentence-based classifier. We investigate the problem of multi-label requirements classification with large taxonomies, illustrate a systematic process to create a ground truth involving industry participants, and provide an analysis of different classification pipelines using zero-shot learning.

Software Engineering
Hierarchical Taxonomy Preparation for Text Categorization Using Consistent Bipartite Spectral Graph Copartitioning

B Gao,TY Liu,G Feng,T Qin,QS Cheng,WY Ma

DOI: https://doi.org/10.1109/tkde.2005.147

IF: 9.235

2005-01-01

IEEE Transactions on Knowledge and Data Engineering

Abstract:Multiclass classification has been investigated for many years in the literature. Recently, the scales of real-world multiclass classification applications have become larger and larger. For example, there are hundreds of thousands of categories employed in the Open Directory Project (ODP) and the Yahoo! directory. In such cases, the scalability of classification methods turns out to be a major concern. To tackle this problem, hierarchical classification is proposed and widely adopted to get better trade-off between effectiveness and efficiency. Unfortunately, many data sets are not explicitly organized in hierarchical forms and, therefore, hierarchical classification cannot be used directly. In this paper, we propose a novel algorithm to automatically mine a hierarchical structure from the flat taxonomy of a data corpus as a preparation for the adoption of hierarchical classification. In particular, we first compute matrices to represent the relations among categories, documents, and terms. And, then, we cocluster the three substances at different scales through consistent bipartite spectral graph copartitioning, which is formulated as a generalized singular value decomposition problem. At last, a hierarchical taxonomy is constructed from the category clusters. Our experiments showed that the proposed algorithm could discover very reasonable taxonomy hierarchy and help improve the classification accuracy.
Using Support Vector Machines for Mining Regression Classes in Large Data Sets

ZH Sun,LX Gao,YX Sun

DOI: https://doi.org/10.1109/tencon.2002.1181221

2002-01-01

Abstract:Support vector machines (SVM) overcome the limit of the maximum-likelihood. method that only applies to very limited set of the density functions. They can estimate simultaneously the regression classes in the mixture data set. The validity of the SVM was demonstrated in experiments. The results indicate that the SVM can estimate the regression classes in the mixture data set with noise.
Large-scale support vector machine classification with redundant data reduction

Xiang-Jun Shen,Lei Mu,Zhen Li,Hao-Xiang Wu,Jian-Ping Gou,Xin Chen

DOI: https://doi.org/10.1016/j.neucom.2014.10.102

IF: 6

2016-01-01

Neurocomputing

Abstract:Large-scale image classification has shown great importance in object recognition and image retrieval as the vast amounts of social multimedia sharing on the networks. While the time and memory requirements for SVM training surge with an increase in the sample size, which makes SVM impractical even for a moderate problem as the number of training data reaches to the extent of hundreds of thousands. To solve this problem, many specially designed algorithms are proposed such as clustering-based SVM training which attempts to remove the clustered data points that lie far away from support vectors. In this paper, we further explore that there exist clustered and scattered data points in a cluster. The clustered data points that lie around the clustering centroid are the dense data points, which are in the inner layer of a cluster. Those data points are viewed as having no SVs and removed. While the scattered data points are the sparse data points in the outside layer of a cluster. Those data points are viewed as having SVs and thus reserved. The Fisher Discriminant Ratio is employed to determine a boundary between the clustered and scattered data points in one cluster, which is computed based on the distance densities of data points to the cluster centroid. The redundant clustered data points in clusters are thus removed to speed up SVM training process. Several experimental results show that our proposed method has good classification accuracy while training time is significantly reduced. The training time in our proposed method only accounts for about 17 percent of the time in LIBSVM on the large data set of Covertype.

computer science, artificial intelligence
Learning Multi-Layer Coarse-to-fine Representations for Large-Scale Image Classification

Ji Zhang,Kuizhi Mei,Yu Zheng,Jianping Fan

DOI: https://doi.org/10.1016/j.patcog.2019.02.024

IF: 8

2019-01-01

Pattern Recognition

Abstract:•Inter-category visual and semantic correlations are exploited.•Large numbers of structural image classes are organized hierarchically.•Hierarchical multi-task SVMs are trained over the visual-semantic tree.•The visual-semantic tree and CNNs are integrated as another framework.•We perform our experiments on 10k image categories for algorithm evaluation.
Review on Hierarchical Learning Methods for Large-Scale Classification Task

Qinghua Hu,Yu Wang,Yucan Zhou,Zekai Hong,Yuhua Qian,Jiye Liang

DOI: https://doi.org/10.1360/n112017-00246

2018-01-01

Abstract:Hierarchical classification is a task that uses hierarchy of categories in data. It can handle large-scale data. In recent years, significant research has emerged in this field, which is receiving increasingly more attention. In this paper, we first introduce the definition of hierarchical classification and thereafter review the important studies on several basic issues in large-scale hierarchical classification tasks based on different problem-solving strategies. First, we define the hierarchy formally and introduce some hierarchical evaluation metrics. Second, we explain how to construct the hierarchy, how to learn classifiers and perform feature selection using the information in the hierarchy, and how to design stopping strategies and introduce some representative studies on each issue. Finally, we summarize the features of large-scale hierarchical classification task and discuss the possible future work in this field.
Hierarchical Support Vector Machine Based Structural Classification with Fused Hierarchies

Shuo Zhao,Yahong Han,Quan Zou,Qinghua Hu

DOI: https://doi.org/10.1016/j.neucom.2016.05.072

IF: 6

2016-01-01

Neurocomputing

Abstract:In this paper, we consider the problem of hierarchical image classification with multiple semantic views of object categories. A novel method is proposed for computing an image-semantic measure by determining the weights for the semantic similarity among the concepts of each view. After obtaining the new image-semantic measure, we construct a semantic hierarchy with the existing method called TRUST-ME. For the hierarchical classification, we translate the classification task with a learned taxonomy into a structured support vector machine (SVM) learning framework. We demonstrate our method on VOC2010 and a subset of the Animals with Attributes dataset, and show that the structured SVM using the weighted semantic hierarchy provides better accuracy.
Large scale semi-supervised linear SVM with stochastic gradient descent

Xin Zhou,Conghui Zhu,Sheng Li,Mo Yu

DOI: https://doi.org/10.12733/jcisP0590

2013-01-01

Journal of Computational Information Systems

Abstract:Semi-supervised learning tries to employ a large collection of unlabeled data and a few labeled examples for improving generalization performance, which has been proved meaningful in real-world applications. The bottleneck of exiting semi-supervised approaches lies in over long training time due to the large scale unlabeled data. In this article we introduce a novel method for semi-supervised linear support vector machine based on average stochastic gradient descent, which significantly enhances the training speed of S3VM over existing toolkits, such as SVMlight-TSVM, CCCP-TSVM and SVMlin. We evaluate our method on text categorization and sentiment classification respectively, which indicates its efficiency on large scale semi-supervised tasks. © 2013 Binary Information Press.
Large-Scale Patent Classification With Min-Max Modular Support Vector Machines

Xiao-lei Chu,Chao Ma,Jing Li,Bao-liang Lu,Masao Utiyama,Hitoshi Isahara

DOI: https://doi.org/10.1109/IJCNN.2008.4634369

2008-01-01

Abstract:Patent classification is a large-scale, hierarchical, imbalanced, multi-label problem. The number of samples in a real-world patent classification typically exceeds one million, and this number increases every year. An effective patent classifier must be able to deal with this situation. This paper discusses the use of min-max modular support vector machine (M-3-SVM) to deal with large-scale patent classification problems. The method includes three steps: decomposing a large-scale and imbalanced patent classification problem into a group of relatively smaller and more balanced two-class subproblems which are independent of each other, learning these subproblems using support vector machines (SVMs) in parallel, and combining all of the trained SVMs according to the minimization and the maximization rules. M-3-SVM has two attractive features which are urgently needed to deal with large-scale patent classification problems. First, it can be realized in a massively parallel form. Second, it can be built up incrementally. Results from experiments using the NTCIR-5 patent data set, which contains more than two million patents, have confirmed these two attractive features, and demonstrate that M-3-SVM outperforms conventional SVMs in terms of both training time and generalization performance.
Hierarchical Classification of Large-Scale Patient Records for Automatic Treatment Stratification

Kuizhi Mei,Jinye Peng,Ling Gao,Naiquan (Nigel) Zheng,Jianping Fan

DOI: https://doi.org/10.1109/jbhi.2015.2414876

IF: 7.7

2015-01-01

IEEE Journal of Biomedical and Health Informatics

Abstract:In this paper, a hierarchical learning algorithm is developed for classifying large-scale patient records, e.g., categorizing large-scale patient records into large numbers of known patient categories (i.e., thousands of known patient categories) for automatic treatment stratification. Our hierarchical learning algorithm can leverage tree structure to train more discriminative max-margin classifiers for high-level nodes and control interlevel error propagation effectively. By ruling out unlikely groups of patient categories (i.e., irrelevant high-level nodes) at an early stage, our hierarchical approach can achieve log-linear computational complexity, which is very attractive for big data applications. Our experiments on one specific medical domain have demonstrated that our hierarchical approach can achieve very competitive results on both classification accuracy and computational efficiency as compared with other state-of-the-art techniques.
Can Large Language Models Serve as Effective Classifiers for Hierarchical Multi-Label Classification of Scientific Documents at Industrial Scale?

Seyed Amin Tabatabaei,Sarah Fancher,Michael Parsons,Arian Askari

2024-12-06

Abstract:We address the task of hierarchical multi-label classification (HMC) of scientific documents at an industrial scale, where hundreds of thousands of documents must be classified across thousands of dynamic labels. The rapid growth of scientific publications necessitates scalable and efficient methods for classification, further complicated by the evolving nature of taxonomies--where new categories are introduced, existing ones are merged, and outdated ones are deprecated. Traditional machine learning approaches, which require costly retraining with each taxonomy update, become impractical due to the high overhead of labelled data collection and model adaptation. Large Language Models (LLMs) have demonstrated great potential in complex tasks such as multi-label classification. However, applying them to large and dynamic taxonomies presents unique challenges as the vast number of labels can exceed LLMs' input limits. In this paper, we present novel methods that combine the strengths of LLMs with dense retrieval techniques to overcome these challenges. Our approach avoids retraining by leveraging zero-shot HMC for real-time label assignment. We evaluate the effectiveness of our methods on SSRN, a large repository of preprints spanning multiple disciplines, and demonstrate significant improvements in both classification accuracy and cost-efficiency. By developing a tailored evaluation framework for dynamic taxonomies and publicly releasing our code, this research provides critical insights into applying LLMs for document classification, where the number of classes corresponds to the number of nodes in a large taxonomy, at an industrial scale.

Artificial Intelligence
Survey of Applying Support Vector Machines to Handle Large-scale Problems

文益民,王耀南,吕宝粮,陈义明

DOI: https://doi.org/10.3969/j.issn.1002-137x.2009.07.004

2009-01-01

Computer Science

Abstract:Being applied to handling large-scale problems,support vector machines(SVMs) needs longer training time and larger memory.The paper analyzed the limitation of SVMs,classified the algorithms of applying SVMs to handle large-scale problems into seven types,and made profound and comprehensive analysis of each kind of algorithm.Moreover,some issues valuable for future exploration in this area were indicated and discussed.
One-class support vector machines for large-scale data sets

Zhibo Xiao,Huangang Wang,Yingchao Xiao,Wenli Xu

DOI: https://doi.org/10.3969/j.issn.1001-0505.2013.S1.043

2013-01-01

Abstract:A method to train one-class support vector machine (OCSVM) on the large-scale data sets is proposed. The proposed method selects inner points representing the distribution characteristics of the original large-scale data sets based on the principle of k-nearest neighbor, and generates the edge points using the inner points selected. A new data set is formed by combining these two kinds of points to train OCSVM. The new data set not only reduces the volume of the original large-scale data set greatly, but also maintains the distribution characteristics of the original data set. Thus the problems faced by OCSVM on the large-scale data sets, such as long training time, complicated models and low predicting speed, are effectively solved. Finally, the experiment is conducted on typical data sets to illustrate the effectiveness of the method proposed.

Support vector machines classification with a very large-scale taxonomy

Support Vector Machines Classification with a Very Large-Scale Taxonomy

An experimental study on large-scale web categorization.

A Multi-Class Large Margin Classifier

Deep Classification in Large-Scale Text Hierarchies

Joint Hierarchical Category Structure Learning and Large-Scale Image Classification

Novel Design of Decision-Tree-Based Support Vector Machines Multi-Class Classifier

Multi-Label Requirements Classification with Large Taxonomies

Hierarchical Taxonomy Preparation for Text Categorization Using Consistent Bipartite Spectral Graph Copartitioning

Using Support Vector Machines for Mining Regression Classes in Large Data Sets

Large-scale support vector machine classification with redundant data reduction

Learning Multi-Layer Coarse-to-fine Representations for Large-Scale Image Classification

Review on Hierarchical Learning Methods for Large-Scale Classification Task

Hierarchical Support Vector Machine Based Structural Classification with Fused Hierarchies

Large scale semi-supervised linear SVM with stochastic gradient descent

Large-Scale Patent Classification With Min-Max Modular Support Vector Machines

Hierarchical Classification of Large-Scale Patient Records for Automatic Treatment Stratification

Can Large Language Models Serve as Effective Classifiers for Hierarchical Multi-Label Classification of Scientific Documents at Industrial Scale?

Survey of Applying Support Vector Machines to Handle Large-scale Problems

One-class support vector machines for large-scale data sets