Abstract:While Large Language Models (LLMs) have demonstrated remarkable potential in natural language generation and instruction following, a persistent challenge lies in their susceptibility to "hallucinations", which erodes trust in their outputs. Although Uncertainty Quantification (UQ) presents a promising solution, its accurate implementation within the context of LLMs remains a significant hurdle. To address this critical roadblock, our research originates from a fundamental heuristic insight: tokens within auto-regressive LLM-generated text do not equally reflect the underlying meaning. Some tokens carry greater relevance and representativeness than others, owing to the phenomenon of "linguistic redundancy", wherein a select few keywords suffice to convey the essence of lengthy sentences. Regrettably, existing methodologies treat all tokens with equal importance when estimating uncertainty, disregarding these inherent generative inequalities. Our analysis reveals a significant issue with state-of-the-art: numerous tokens (and sentences) of limited semantic significance receive equal or even excessive weighting during uncertainty estimation. To rectify this bias, we propose to jointly Shifting Attention to more Relevant (SAR) components, at both the token- and the sentence-levels for accurate uncertainty estimation. We conduct extensive experiments involving a range of popular "off-the-shelf" LLMs, including instruction-tuned LLMs such as Vicuna, WizardLM, and LLaMA-2-chat, as well as pretrained LLMs like OPT and LLaMA, with model sizes extending up to 33B parameters. We carry out evaluation across various free-form question-answering tasks, encompassing domains such as reading comprehension, science Q&A, and medical Q&A. Our experimental results demonstrate the superior performance of SAR in addressing the challenges of uncertainty estimation within the realm of LLMs.

Rethinking Sentiment Analysis under Uncertainty

Are you sure? Analysing Uncertainty Quantification Approaches for Real-world Speech Emotion Recognition

Quantifying Uncertainties in Natural Language Processing Tasks

Shifting Attention to Relevance: Towards the Uncertainty Estimation of Large Language Models

Emotion AI-Driven Sentiment Analysis: A Survey, Future Research Directions, and Open Issues

Data Uncertainty-Aware Learning for Multimodal Aspect-based Sentiment Analysis

Enhancing Sentiment Analysis Results through Outlier Detection Optimization

Hybrid Uncertainty Calibration for Multimodal Sentiment Analysis

Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Models

SAUP: Situation Awareness Uncertainty Propagation on LLM Agent

A semantic and syntactic enhanced neural model for financial sentiment analysis

Sentiment analysis is not solved! Assessing and probing sentiment classification

A Survey of Quantum-Cognitively Inspired Sentiment Analysis Models

Label-Confidence-Aware Uncertainty Estimation in Natural Language Generation

Sentiment Analysis in the Era of Large Language Models: A Reality Check

Ambiguity Meets Uncertainty: Investigating Uncertainty Estimation for Word Sense Disambiguation

UniSA: Unified Generative Framework for Sentiment Analysis

Effective Sentiment Analysis of Corporate Financial Reports.

A Survey of Uncertainty Estimation in LLMs: Theory Meets Practice

Sentiment interpretability analysis on Chinese texts employing multi-task and knowledge base