Mixtures of Unsupervised Lexicon Classification

Peratham Wiriyathammabhum

DOI: https://doi.org/10.48550/arXiv.2405.17116

2024-05-27

Abstract:This paper presents a mixture version of the method-of-moment unsupervised lexicon classification by an incorporation of a Dirichlet process.

Computation and Language,Artificial Intelligence,Computer Vision and Pattern Recognition,Machine Learning

What problem does this paper attempt to address?

A Mixture Language Model for the Classification of Chinese Online Reviews.

Ming Jiang,Jian Wang,Xingqi Wang,Jingfan Tang,Chunming Wu

DOI: https://doi.org/10.1504/ijict.2015.066017

2015-01-01

International Journal of Information and Communication Technology

Abstract:In this essay, we propose an unsupervised topic and sentiment mixture model (LSS model) mainly on the basis of LDA model and combining with the sentiment factor, as an approach to sentiment classification of online reviews. There is an extra constraint that all words in a sentence are generated from one topic and one sentiment. Which means the hypothesis of the model is a word in a sentence is of one meaning only. LSS model is totally unsupervised and it needs no labelled corpora or any other sentiment seed words. Experiments show that LSS model performs better than JST model and ASUM model. The F-1 value of sentiment classification is 8% higher than ASUM model and 12.5% than JST model.
Mixtures of Dirichlet-Multinomial distributions for supervised and unsupervised classification of short text data

Laura Anderlucci,Cinzia Viroli

DOI: https://doi.org/10.1007/s11634-020-00399-3

2020-05-25

Advances in Data Analysis and Classification

Abstract:Topic detection in short textual data is a challenging task due to its representation as high-dimensional and extremely sparse document-term matrix. In this paper we focus on the problem of classifying textual data on the base of their (unique) topic. For unsupervised classification, a popular approach called Mixture of Unigrams consists in considering a mixture of multinomial distributions over the word counts, each component corresponding to a different topic. The multinomial distribution can be easily extended by a Dirichlet prior to the compound mixtures of Dirichlet-Multinomial distributions, which is preferable for sparse data. We propose a gradient descent estimation method for fitting the model, and investigate supervised and unsupervised classification performance on real empirical problems.

statistics & probability
A probability for classification based on the mixture of Dirichlet process model

Ruth Fuentes-Garcia,Ramses H Mena,Stephen G Walker

DOI: https://doi.org/10.48550/arXiv.0905.0210

2009-05-02

Abstract:In this paper, we provide an explicit probability distribution for classification purposes. It is derived from the Bayesian nonparametric mixture of Dirichlet process model, but with suitable modifications which remove unsuitable aspects of the classification based on this model. The resulting approach then more closely resembles a classical hierarchical grouping rule in that it depends on sums of squares of neighboring values. The proposed probability model for classification relies on a simulation algorithm which will be based on a reversible MCMC algorithm for determining the probabilities, and we provide numerical illustrations comparing with alternative ideas for classification.

Applications
Label Correlation Mixture Model: A Supervised Generative Approach to Multilabel Spoken Document Categorization

Zhiyang He,Ji Wu,Tao Li

DOI: https://doi.org/10.1109/tetc.2014.2377559

2015-01-01

IEEE Transactions on Emerging Topics in Computing

Abstract:Multilabel categorization, which is more difficult but practical than the conventional binary and multiclass categorization, has received a great deal of attention in recent years. This paper proposes a novel probabilistic generative model, label correlation mixture model (LCMM), to depict the multiply labeled documents, which can be used for multilabel spoken document categorization as well as multilabel text categorization. In LCMM, labels and topics have the one-to-one correspondences. The LCMM consists of two important components: 1) a label correlation model and 2) a multilabel conditioned document model. The label correlation model formulates the generating process of labels where the dependences between the labels are taken into account. We also propose an efficient algorithm for calculating the probability of generating an arbitrary subset of labels. The multilabel conditioned document model can be regarded as a supervised label mixture model, in which labels for a document are known. Each label is characterized by distributions over words. For the parameter learning of the multilabel conditioned document model, in addition to maximum likelihood estimation, a discriminative approach based on the minimum classification error rate training is proposed. To evaluate LCMM, extensive multilabel categorization experiments are conducted on a spoken document data set and three standard text data sets. The experimental results in comparison with other competitive methods demonstrate the effectiveness of LCMM.
Dirichlet Mixture Allocation for Multiclass Document Collections Modeling

Wei Bian,Dacheng Tao

DOI: https://doi.org/10.1109/ICDM.2009.102

2009-01-01

Abstract:Topic model, latent Dirichlet allocation (LDA), is an effective tool for statistical analysis of large collections of documents. In LDA, each document is modeled as a mixture of topics and the topic proportions are generated from the unimodal Dirichlet distribution prior. When a collection of documents are drawn from multiple classes, this unimodal prior is insufficient for data fitting. To solve this problem, we exploit the multimodal Dirichlet mixture prior, and propose the Dirichlet mixture allocation (DMA). We report experiments on the popular TDT2 Corpus demonstrating that DMA models a collection of documents more precisely than LDA when the documents are obtained from multiple classes.
Unsupervised Learning of Dirichlet Process Mixture Models with Missing Data

Xunan Zhang,Shiji Song,Lei Zhu,Keyou You,Cheng Wu

DOI: https://doi.org/10.1007/s11432-015-5429-0

2015-01-01

Science China Information Sciences

Abstract:This study presents a novel approach to unsupervised learning for clustering with missing data.We first extend a finite mixture model to the infinite case by considering Dirichlet process mixtures, which can automatically determine the number of mixture components or clusters. Furthermore, we view the missing features as latent variables and compute the posterior distributions using the variational Bayesian expectation maximization algorithm, which optimizes the evidence lower bound on the complete-data log marginal likelihood. We demonstrate the performance on several artificial data sets with missing values. The experimental results indicate that the proposed method outperforms some classic imputation methods. We finally present an application to seabed hydrothermal sulfide color images analysis problem.
Multi-label Text Classification Based on the Label Correlation Mixture Model.

Zhiyang He,Ji Wu,Ping Lv

DOI: https://doi.org/10.3233/ida-163055

IF: 1.7

2017-01-01

Intelligent Data Analysis

Abstract:In the current paper, we propose a probabilistic generative model, the label correlation mixture model (LCMM), to depict multi-labeled document data, which can be utilized for multi-label text classification. LCMM assumes two stochastic generative processes, which correspond to two submodels: 1) a label correlation model; and 2) a label mixture model. The former model formulates labels' generative process, in which a label correlation network is created to depict the dependency between labels. Moreover, we present an efficient inference algorithm for calculating the generative probability of a multi-label class. Furthermore, in order to optimize the label correlation network, we propose a parameter-learning algorithm based on gradient descent. The second submodel in the LCMM depicts the generative process of words in a document with the given labels. Different traditional mixture models can be adopted in this generative process, such as the mixture of language models, or topic models. In the multi-label classification stage, we propose a two-step strategy to most efficiently utilize the LCMM based on the framework of Bayes decision theory. We conduct extensive multi-label classification experiments on three standard text data sets. The experimental results show significant performance improvements comparing to existing approaches. For example, the improvements on accuracy and macro F-score measures in the OHSUMED data set achieve 28.3% and 37.0%, respectively. These performance enhancements demonstrate the effectiveness of the proposed models and solutions.
Hierarchical mixtures of Unigram models for short text clustering: the role of Beta-Liouville priors

Massimo Bilancia,Samuele Magro

2024-10-29

Abstract:This paper presents a variant of the Multinomial mixture model tailored for the unsupervised classification of short text data. Traditionally, the Multinomial probability vector in this hierarchical model is assigned a Dirichlet prior distribution. Here, however, we explore an alternative prior - the Beta-Liouville distribution - which offers a more flexible correlation structure than the Dirichlet. We examine the theoretical properties of the Beta-Liouville distribution, focusing on its conjugacy with the Multinomial likelihood. This property enables the derivation of update equations for a CAVI (Coordinate Ascent Variational Inference) variational algorithm, facilitating the approximate posterior estimation of model parameters. Additionally, we propose a stochastic variant of the CAVI algorithm that enhances scalability. The paper concludes with data examples that demonstrate effective strategies for setting the Beta-Liouville hyperparameters.

Machine Learning,Computation
Nonparametric Bayesian Methods (Dirichlet Process Mixtures)

Jun Zhu

2011-01-01

Abstract:Page 1. Jun Zhu dcszj@mail.tsinghua.edu.cn State Key Lab of Intelligent Technology & Systems Tsinghua University November 8, 2011 Nonparametric Bayesian Methods (Dirichlet Process Mixtures) [80240603 Advanced Machine Learning, Fall, 2012] Page 2. Recap. of Topic Models A Bayesian Admixture Model ❑ Mixture of mixtures Inference with variational approximation and Gibbs sampling Page 3. Latent Dirichlet Allocation -- a generative story for documents A Bayesian mixture model with topical bases Each document is a random mixture over topics; Each word is generated by ONE topic image, jpg, gif, file, color, file, images, files, format ground, wire, power, wiring, current, circuit, Topic #1 Topic #2 Document #1: gif jpg image current file color images ground power file current format file formats circuit gif images Document #2: wire currents file format ground power image format wire circuit …
Hierarchical mixture of discriminative Generalized Dirichlet classifiers

Elvis Togban,Djemel Ziou

2024-05-03

Abstract:This paper presents a discriminative classifier for compositional data. This classifier is based on the posterior distribution of the Generalized Dirichlet which is the discriminative counterpart of Generalized Dirichlet mixture model. Moreover, following the mixture of experts paradigm, we proposed a hierarchical mixture of this classifier. In order to learn the models parameters, we use a variational approximation by deriving an upper-bound for the Generalized Dirichlet mixture. To the best of our knownledge, this is the first time this bound is proposed in the literature. Experimental results are presented for spam detection and color space identification.

Machine Learning
Mixture of Mixups for Multi-label Classification of Rare Anuran Sounds

Ilyass Moummad,Nicolas Farrugia,Romain Serizel,Jeremy Froidevaux,Vincent Lostanlen

2024-06-22

Abstract:Multi-label imbalanced classification poses a significant challenge in machine learning, particularly evident in bioacoustics where animal sounds often co-occur, and certain sounds are much less frequent than others. This paper focuses on the specific case of classifying anuran species sounds using the dataset AnuraSet, that contains both class imbalance and multi-label examples. To address these challenges, we introduce Mixture of Mixups (Mix2), a framework that leverages mixing regularization methods Mixup, Manifold Mixup, and MultiMix. Experimental results show that these methods, individually, may lead to suboptimal results; however, when applied randomly, with one selected at each training iteration, they prove effective in addressing the mentioned challenges, particularly for rare classes with few occurrences. Further analysis reveals that Mix2 is also proficient in classifying sounds across various levels of class co-occurrences.

Sound,Machine Learning,Audio and Speech Processing
Label correlation mixture model for multi-label text categorization

Zhiyang He,Ji Wu,Ping Lv

DOI: https://doi.org/10.1109/SLT.2014.7078554

2014-01-01

Abstract:Multi-label text categorization is more difficult but practical than the conventional binary or multi-class text categorization. This paper propose a novel probabilistic generative model, label correlation mixture model (LCMM), to depict the multiple labeled documents, which can be used for multi-label text categorization. In LCMM, labels and topics have the one-to-one correspondences. LCMM consists of two parts: label correlation model and multi-label conditioned document model. The former one formulates the generating process of labels and the dependencies between the labels are taken into account. We also propose an efficient algorithm for calculating the probability of generating an arbitrary subset of labels. Multi-label conditioned document model can be regarded as a supervised label mixture model, in which the labels for a document are known. To evaluate LCMM, multi-label text categorization experiments on three standard text data sets are performed. The experimental results demonstrate the effectiveness of LCMM, comparing to other reported methods.
MixLab: An Informative Semi-supervised Method for Multi-label Classification

Ye Qiu,Xiaolong Gong,Zhiyi Ma,Xi Chen

DOI: https://doi.org/10.1007/978-3-030-60450-9_40

2020-01-01

Abstract:Multi-label classification is an intensively studied topic in data analysis. In spite of the considerable improvements, recent deep learning-based methods overlook the existence of unlabeled data, which consumes too much time on instance annotation. To circumvent this difficulty, semi-supervised multi-label classification aims to exploit the readily-available unlabeled data to help build multi-label classification model. To make full use of labeled and unlabeled data, this paper propose a novel approach named MixLab, encourages the model classifications to be accurate with label-correlated information and consistency regularization. It utilizes label correlations to enhance predicted labels for augmented unlabeled instances as targets and regularizes predictions to be consistent with this targets. We empirically validate the effectiveness of our framework by extensive experiments on four real datasets of textual content.
Unsupervised Bilingual Lexicon Induction Via Latent Variable Models.

Zi-Yi Dou,Zhi-Hao Zhou,Shujian Huang

DOI: https://doi.org/10.18653/v1/d18-1062

2018-01-01

Abstract:Bilingual lexicon extraction has been studied for decades and most previous methods have relied on parallel corpora or bilingual dictionaries. Recent studies have shown that it is possible to build a bilingual dictionary by aligning monolingual word embedding spaces in an unsupervised way. With the recent advances in generative models, we propose a novel approach which builds cross-lingual dictionaries via latent variable models and adversarial training with no parallel corpora. To demonstrate the effectiveness of our approach, we evaluate our approach on several language pairs and the experimental results show that our model could achieve competitive and even superior performance compared with several state-of-the-art models.
Cross-Lingual Mixture Model for Sentiment Classification.

Xinfan Meng,Furu Wei,Xiaohua Liu,Ming Zhou,Ge Xu,Houfeng Wang

2012-01-01

Abstract:The amount of labeled sentiment data in English is much larger than that in other languages. Such a disproportion arouse interest in cross-lingual sentiment classification, which aims to conduct sentiment classification in the target language (e.g. Chinese) using labeled data in the source language (e.g. English). Most existing work relies on machine translation engines to directly adapt labeled data from the source language to the target language. This approach suffers from the limited coverage of vocabulary in the machine translation results. In this paper, we propose a generative cross-lingual mixture model ( CLMM ) to leverage unlabeled bilingual parallel data. By fitting parameters to maximize the likelihood of the bilingual parallel data, the proposed model learns previously unseen sentiment words from the large bilingual parallel data and improves vocabulary coverage significantly. Experiments on multiple data sets show that CLMM is consistently effective in two settings: (1) labeled data in the target language are unavailable; and (2) labeled data in the target language are also available.
Modelling the Lexicon in Unsupervised Part of Speech Induction

Greg Dubbin,Phil Blunsom

DOI: https://doi.org/10.48550/arXiv.1402.6516

2014-02-26

Abstract:Automatically inducing the syntactic part-of-speech categories for words in text is a fundamental task in Computational Linguistics. While the performance of unsupervised tagging models has been slowly improving, current state-of-the-art systems make the obviously incorrect assumption that all tokens of a given word type must share a single part-of-speech tag. This one-tag-per-type heuristic counters the tendency of Hidden Markov Model based taggers to over generate tags for a given word type. However, it is clearly incompatible with basic syntactic theory. In this paper we extend a state-of-the-art Pitman-Yor Hidden Markov Model tagger with an explicit model of the lexicon. In doing so we are able to incorporate a soft bias towards inducing few tags per type. We develop a particle filter for drawing samples from the posterior of our model and present empirical results that show that our model is competitive with and faster than the state-of-the-art without making any unrealistic restrictions.

Computation and Language
Topic-based mixture language modelling

YOSHIHIKO GOTOH,STEVE RENALS

DOI: https://doi.org/10.1017/s1351324900002278

IF: 1.841

1999-12-01

Natural Language Engineering

Abstract:This paper describes an approach for constructing a mixture of language models based on simple statistical notions of semantics using probabilistic models developed for information retrieval. The approach encapsulates corpus-derived semantic information and is able to model varying styles of text. Using such information, the corpus texts are clustered in an unsupervised manner and a mixture of topic-specific language models is automatically created. The principal contribution of this work is to characterise the document space resulting from information retrieval techniques and to demonstrate the approach for mixture language modelling. A comparison is made between manual and automatic clustering in order to elucidate how the global content information is expressed in the space. We also compare (in terms of association with manual clustering and language modelling accuracy) alternative term-weighting schemes and the effect of singular value decomposition dimension reduction (latent semantic analysis). Test set perplexity results using the British National Corpus indicate that the approach can improve the potential of statistical language modelling. Using an adaptive procedure, the conventional model may be tuned to track text data with a slight increase in computational cost.

computer science, artificial intelligence,linguistics
Combining Supervised Learning and Reinforcement Learning for Multi-Label Classification Tasks with Partial Labels

Zixia Jia,Junpeng Li,Shichuan Zhang,Anji Liu,Zilong Zheng

2024-06-24

Abstract:Traditional supervised learning heavily relies on human-annotated datasets, especially in data-hungry neural approaches. However, various tasks, especially multi-label tasks like document-level relation extraction, pose challenges in fully manual annotation due to the specific domain knowledge and large class sets. Therefore, we address the multi-label positive-unlabelled learning (MLPUL) problem, where only a subset of positive classes is annotated. We propose Mixture Learner for Partially Annotated Classification (MLPAC), an RL-based framework combining the exploration ability of reinforcement learning and the exploitation ability of supervised learning. Experimental results across various tasks, including document-level relation extraction, multi-label image classification, and binary PU learning, demonstrate the generalization and effectiveness of our framework.

Computation and Language,Artificial Intelligence
Reinforced mixture learning

Yuan Le,Fan Zhou,Yang Bai

DOI: https://doi.org/10.1016/j.neunet.2023.05.018

IF: 7.8

2023-05-27

Neural Networks

Abstract:In this article, we formulate the standard mixture learning problem as a Markov Decision Process (MDP). We theoretically show that the objective value of the MDP is equivalent to the log-likelihood of the observed data with a slightly different parameter space constrained by the policy. Different from some classic mixture learning methods such as Expectation-Maximization (EM) algorithm, the proposed reinforced algorithm requires no distribution assumptions and can handle the non-convex clustered data by constructing a model-free reward to evaluate the mixture assignment based on the spectral graph theory and Linear Discriminant Analysis (LDA). Extensive experiments on both synthetic and real examples demonstrate that the proposed method is comparable with the EM algorithm when the Gaussian mixture assumption is satisfied, and significantly outperforms it and other clustering methods in most scenarios when the model is misspecified. A Python implementation of our proposed method is available at https://github.com/leyuanheart/Reinforced-Mixture-Learning .

computer science, artificial intelligence,neurosciences
Sentiment Classification Using Supervised and Semi-Supervised Conditional Maximum Entropy Modeling.

Qi Zhang,Xuanjing Huang,Lide Wu

DOI: https://doi.org/10.1142/s1793840608001937

2008-01-01

Abstract:Abstract： This paper presents our work on determining the sentiment polarity of sentences and articles. Feature combinations is the first novelty of our approaches. In order to explore better feature sets, we do some data analysis of the MPQA corpus and a few of experiments to evaluate combinations. Through those efforts, our approach achieves 77.0% accuracy in MPQA data set. Another novelty of this work is that Semi-supervised Conditional Maximum Entropy (SCME) modeling is used to combine labeled and unlabeled data. Experimental results show that our approach can significantly improve performance. In sentence-level, our approach achieves 78.2% accuracy in MPQA data set, the relative improvement given by semi-supervised technique is 5.2% over the supervised method.

Mixtures of Unsupervised Lexicon Classification

A Mixture Language Model for the Classification of Chinese Online Reviews.

Mixtures of Dirichlet-Multinomial distributions for supervised and unsupervised classification of short text data

A probability for classification based on the mixture of Dirichlet process model

Label Correlation Mixture Model: A Supervised Generative Approach to Multilabel Spoken Document Categorization

Dirichlet Mixture Allocation for Multiclass Document Collections Modeling

Unsupervised Learning of Dirichlet Process Mixture Models with Missing Data

Multi-label Text Classification Based on the Label Correlation Mixture Model.

Hierarchical mixtures of Unigram models for short text clustering: the role of Beta-Liouville priors

Nonparametric Bayesian Methods (Dirichlet Process Mixtures)

Hierarchical mixture of discriminative Generalized Dirichlet classifiers

Mixture of Mixups for Multi-label Classification of Rare Anuran Sounds

Label correlation mixture model for multi-label text categorization

MixLab: An Informative Semi-supervised Method for Multi-label Classification

Unsupervised Bilingual Lexicon Induction Via Latent Variable Models.

Cross-Lingual Mixture Model for Sentiment Classification.

Modelling the Lexicon in Unsupervised Part of Speech Induction

Topic-based mixture language modelling

Combining Supervised Learning and Reinforcement Learning for Multi-Label Classification Tasks with Partial Labels

Reinforced mixture learning

Sentiment Classification Using Supervised and Semi-Supervised Conditional Maximum Entropy Modeling.