Abstract:The recent success of Large Language Models (LLMs) has gained significant attention in both academia and industry. Substantial efforts have been made to enhance the zero- and few-shot generalization capabilities of open-source LLMs through finetuning. Currently, the prevailing approach is instruction-tuning, which trains LLMs to complete real-world tasks by generating responses guided by natural language instructions. It is worth noticing that such an approach may underperform in sequence and token classification tasks. Unlike text generation tasks, classification tasks have a limited label space, where precise label prediction is more appreciated than generating diverse and human-like responses. Prior research has unveiled that instruction-tuned LLMs cannot outperform BERT, prompting us to explore the potential of leveraging latent representations from LLMs for supervised label prediction. In this paper, we introduce a label-supervised adaptation for LLMs, which aims to finetuning the model with discriminant labels. We evaluate this approach with Label Supervised LLaMA (LS-LLaMA), based on LLaMA-2-7B, a relatively small-scale LLM, and can be finetuned on a single GeForce RTX4090 GPU. We extract latent representations from the final LLaMA layer and project them into the label space to compute the cross-entropy loss. The model is finetuned by Low-Rank Adaptation (LoRA) to minimize this loss. Remarkably, without intricate prompt engineering or external knowledge, LS-LLaMA substantially outperforms LLMs ten times its size in scale and demonstrates consistent improvements compared to robust baselines like BERT-Large and RoBERTa-Large in text classification. Moreover, by removing the causal mask from decoders, LS-unLLaMA achieves the state-of-the-art performance in named entity recognition (NER). Our work will shed light on a novel approach to adapting LLMs for various downstream tasks.

Meta-LMTC - Meta-Learning for Large-Scale Multi-Label Text Classification.

Learning Label-Adaptive Representation for Large-Scale Multi-Label Text Classification

A Survey of Meta-learning for Classification Tasks

An Empirical Study on Large-Scale Multi-Label Text Classification Including Few and Zero-Shot Labels

Label-Specific Feature Augmentation for Long-Tailed Multi-Label Text Classification

Improve Meta-learning for Few-Shot Text Classification with All You Can Acquire from the Tasks

Multi-Label Text Classification Model Integrating Label Attention and Historical Attention

LMPT: Prompt Tuning with Class-Specific Embedding Loss for Long-tailed Multi-Label Visual Recognition

Learning in Imperfect Environment: Multi-Label Classification with Long-Tailed Distribution and Partial Labels

Does Head Label Help for Long-Tailed Multi-Label Text Classification

Coarse-to-Fine Pseudo-Labeling Guided Meta-Learning for Few-Shot Classification

Label Supervised LLaMA Finetuning

From Text Segmentation to Enhanced Representation Learning: A Novel Approach to Multi-Label Classification for Long Texts

Exploring Contrastive Learning for Long-Tailed Multi-Label Text Classification

Research of multi-label text classification based on label attention and correlation networks

Meta learning with language models: Challenges and opportunities in the classification of imbalanced text

Towards Better Representations for Multi-Label Text Classification with Multi-granularity Information

Enhancing Representation Learning with Label Association for Multi-Label Text Classification.

Decomposed Meta-Learning for Few-Shot Sequence Labeling

MetaMIML: Meta Multi-Instance Multi-Label Learning

Label-Related Adaptive Graph Construction Based on Attention for Multi-label Text Classification