Abstract:This paper presents an impartial and extensive benchmark for text classification involving five different text classification tasks, 20 datasets, 11 different model architectures, and 42,800 algorithm runs. The five text classification tasks are fake news classification, topic detection, emotion detection, polarity detection, and sarcasm detection. While in practice, especially in Natural Language Processing (NLP), research tends to focus on the most sophisticated models, we hypothesize that this is not always necessary. Therefore, our main objective is to investigate whether the largest state-of-the-art (SOTA) models are always preferred, or in what cases simple methods can compete with complex models, i.e. for which dataset specifications and classification tasks. We assess the performance of different methods with varying complexity, ranging from simple statistical and machine learning methods to pretrained transformers like robustly optimized BERT (Bidirectional Encoder Representations from Transformers) pretraining approach (RoBERTa). This comprehensive benchmark is lacking in existing literature, with research mainly comparing similar types of methods. Furthermore, with increasing awareness of the ecological impacts of extensive computational resource usage, this comparison is both critical and timely.We find that overall, bidirectional long short-term memory (LSTM) networks are ranked as the best-performing method albeit not statistically significantly better than logistic regression and RoBERTa. Overall, we cannot conclude that simple methods perform worse although this depends mainly on the classification task. Concretely, we find that for fake news classification and topic detection, simple techniques are the best-ranked models and consequently, it is not necessary to train complicated neural network architectures for these classification tasks. Moreover, we also find a negative correlation between F1 performance and complexity for the smallest datasets (with dataset size less than 10,000). Finally, the different models' results are analyzed in depth to explain the model decisions, which is an increasing requirement in the field of text classification.

Bayesian Performance Comparison of Text Classifiers

A Bayesian Hierarchical Model for Comparing Average F1 Scores

A Technique For Improving The Performance Of Naive Bayes Text Classification

A PERFORMANCE PREDICTION METHOD BASED ON NAIVE BAYES CLASSIFICATION

Naive Bayes Based Criminal Text Classification of Unbalanced Classes

Bayesian Naïve Bayes Classifiers to Text Classification

Performance Analysis and Comparison of Neural Networks and Support Vector Machines Classifier

Performance Analysis and Improvement of Na&#x00ef;ve Bayes in Text Classification Application

Bayes Test of Precision, Recall, and F-1 Measure for Comparison of Two Natural Language Processing Models

A Comparison Study of Bayesian Classifiers on Web Pages Classification

A Study of Classification Based on Bayes Classifiers

Bayesian Multinomial Naïve Bayes Classifier to Text Classification.

Classifier comparison using precision

Method of Feature Selection for Text Categorization with Bayesian Classifiers

Evaluating text classification: A benchmark study

Performance Comparison and Analysis of Several General Text Classification Algorithms

A Parallel Algorithm for Bayesian Text Classification Based on Noise Elimination and Dimension Reduction in Spark Computing Environment

Bagging Text Classification Algorithm Based on Classifier Performance Evaluation

Feature Selection for Text Classification with Naïve Bayes

Bayesian nonparametric cross-study validation of prediction methods

A Bayesian approach for evaluating equivalence over multiple groups, and comparison with frequentist tost