Sentiment Analysis through LLM Negotiations

Xiaofei Sun,Xiaoya Li,Shengyu Zhang,Shuhe Wang,Fei Wu,Jiwei Li,Tianwei Zhang,Guoyin Wang

2023-11-03

Abstract:A standard paradigm for sentiment analysis is to rely on a singular LLM and makes the decision in a single round under the framework of in-context learning. This framework suffers the key disadvantage that the single-turn output generated by a single LLM might not deliver the perfect decision, just as humans sometimes need multiple attempts to get things right. This is especially true for the task of sentiment analysis where deep reasoning is required to address the complex linguistic phenomenon (e.g., clause composition, irony, etc) in the input. To address this issue, this paper introduces a multi-LLM negotiation framework for sentiment analysis. The framework consists of a reasoning-infused generator to provide decision along with rationale, a explanation-deriving discriminator to evaluate the credibility of the generator. The generator and the discriminator iterate until a consensus is reached. The proposed framework naturally addressed the aforementioned challenge, as we are able to take the complementary abilities of two LLMs, have them use rationale to persuade each other for correction. Experiments on a wide range of sentiment analysis benchmarks (SST-2, Movie Review, Twitter, yelp, amazon, IMDB) demonstrate the effectiveness of proposed approach: it consistently yields better performances than the ICL baseline across all benchmarks, and even superior performances to supervised baselines on the Twitter and movie review datasets.

Computation and Language

What problem does this paper attempt to address?

The paper aims to address a key issue in sentiment analysis tasks, namely that a single large language model (LLM) may not always make the correct judgment within the existing single-turn decision framework. Specifically, sentiment analysis tasks require deep reasoning about complex linguistic phenomena (such as clause structures, sarcasm, etc.), and the output generated by a single LLM in one turn may not be sufficient to make a perfect decision. To solve this problem, the paper proposes a multi-LLM negotiation framework.

This framework includes the following core components:
1. **Reasoning-infused generator**: Responsible for generating sentiment decisions with reasoning processes.
2. **Explanation-deriving discriminator**: Evaluates the credibility of the generator's output and provides reasons to support its judgment.
3. **Negotiation mechanism**: The generator and discriminator negotiate through multiple iterations until they reach a consensus or the maximum number of negotiation rounds is reached.

This method leverages the complementary capabilities of two LLMs, allowing them to persuade each other to make corrections by providing reasoning grounds, thereby naturally solving the problem where a single LLM cannot provide the correct answer on the first attempt.

Experimental results show that this method outperforms traditional single-turn decision methods on multiple sentiment analysis benchmark datasets (including SST-2, Movie Reviews, Twitter, Yelp, Amazon, and IMDB), and even surpasses supervised learning baselines on some datasets.