Abstract:Text classification in natural language processing (NLP) is evolving rapidly, particularly with the surge in transformer-based models, including large language models (LLM). This paper presents an in-depth survey of text classification techniques across diverse benchmarks, addressing applications from sentiment analysis to chatbot-driven question-answering. Methodologically, it utilizes NLP-facilitated approaches such as co-citation and bibliographic coupling alongside traditional research techniques. Because new use cases continue to emerge in this dynamic field, the study proposes an expanded taxonomy of text classification applications, extending the focus beyond unimodal (text-only) inputs to explore the emerging field of multimodal classification. While offering a comprehensive review of text classification with LLMs, this review highlights novel questions that arise when approaching the task with transformers: It evaluates the use of multimodal data, including text, numeric, and columnar data, and discusses the evolution of text input lengths (tokens) for long text classification; it covers the historical development of transformer-based models, emphasizing recent advancements in LLMs; it evaluates model accuracy on 358 datasets across 20 applications, with results challenging the assumption that LLMs are universally superior, revealing unexpected findings related to accuracy, cost, and safety; and it explores issues related to cost and access as models become increasingly expensive. Finally, the survey discusses new social and ethical implications raised when using LLMs for text classification, including bias and copyright. Throughout, the review emphasizes the importance of a nuanced understanding of model performance and a holistic approach to deploying transformer-based models in real-world applications.

Efficient Classification of Long Documents Using Transformers

Improving Scientific Literature Classification: A Parameter-Efficient Transformer-Based Approach

Revisiting Transformer-based Models for Long Document Classification

Can Model Fusing Help Transformers in Long Document Classification? An Empirical Study

A Survey of Text Classification With Transformers: How Wide? How Large? How Long? How Accurate? How Expensive? How Safe?

Hierarchical Transformers for Long Document Classification

A Survey on Long Text Modeling with Transformers

Transformers for Multi-label Classification of Medical Text: An Empirical Comparison

Transformers are Short Text Classifiers: A Study of Inductive Short Text Classifiers on Benchmarks and Real-world Datasets

Comparative Study of Long Document Classification

Long Text Classification with Segmentation

Efficient Transformers: A Survey

Hi-Transformer: Hierarchical Interactive Transformer for Efficient and Effective Long Document Modeling

Revisiting Bag of Words Document Representations for Efficient Ranking with Transformers

A Comprehensive Study of Vision Transformers in Image Classification Tasks

Long Document Ranking with Query-Directed Sparse Transformer

Transformers in health: a systematic review on architectures for longitudinal data analysis

Which Transformer to Favor: A Comparative Analysis of Efficiency in Vision Transformers

Simple Hack for Transformers against Heavy Long-Text Classification on a Time- and Memory-Limited GPU Service

Limitations of Transformers on Clinical Text Classification

Hierarchical Multi-modal Transformer for Cross-modal Long Document Classification