Abstract:Data annotation in NLP is a costly and time-consuming task, traditionally handled by human experts who require extensive training to enhance the task-related background knowledge. Besides, labeling social media texts is particularly challenging due to their brevity, informality, creativity, and varying human perceptions regarding the sociocultural context of the world. With the emergence of GPT models and their proficiency in various NLP tasks, this study aims to establish a performance baseline for GPT-4 as a social media text annotator. To achieve this, we employ our own dataset of tweets, expertly labeled for stance detection with full inter-rater agreement among three annotators. We experiment with three techniques: Zero-shot, Few-shot, and Zero-shot with Chain-of-Thoughts to create prompts for the labeling task. We utilize four training sets constructed with different label sets, including human labels, to fine-tune transformer-based large language models and various combinations of traditional machine learning models with embeddings for stance classification. Finally, all fine-tuned models undergo evaluation using a common testing set with human-generated labels. We use the results from models trained on human labels as the benchmark to assess GPT-4's potential as an annotator across the three prompting techniques. Based on the experimental findings, GPT-4 achieves comparable results through the Few-shot and Zero-shot Chain-of-Thoughts prompting methods. However, none of these labeling techniques surpass the top three models fine-tuned on human labels. Moreover, we introduce the Zero-shot Chain-of-Thoughts as an effective strategy for aspect-based social media text labeling, which performs better than the standard Zero-shot and yields results similar to the high-performing yet expensive Few-shot approach.

Benchmarking zero-shot stance detection with FlanT5-XXL: Insights from training data, prompting, and decoding strategies into its near-SoTA performance

Enhancing Zero-Shot Stance Detection with Contrastive and Prompt Learning

Exploiting Sentiment and Common Sense for Zero-shot Stance Detection

Chain of Stance: Stance Detection with Large Language Models

Zero-Shot Stance Detection Via Contrastive Learning

Zero-shot Cross-lingual Stance Detection via Adversarial Language Adaptation

Incorporating Target-Aware Knowledge into Prompt-Tuning for Few-Shot Stance Detection

Adversarial Learning for Zero-Shot Stance Detection on Social Media

Investigating the Robustness of Modelling Decisions for Few-Shot Cross-Topic Stance Detection: A Preregistered Study

Prompting and Fine-Tuning Open-Sourced Large Language Models for Stance Classification

Zero-shot stance detection based on multi-expert collaboration

Evaluating large language models for user stance detection on X (Twitter)

A Semi-Supervised Approach to Detecting Stance in Tweets

Zero-Shot Stance Detection: A Dataset and Model using Generalized Topic Representations

EDDA: A Encoder-Decoder Data Augmentation Framework for Zero-Shot Stance Detection

Can Large Language Models Address Open-Target Stance Detection?

A Token Level Multi-target Stance Detection Dataset.

Stance Detection on Social Media with Fine-Tuned Large Language Models

GPT-4 as an X data annotator: Unraveling its performance on a stance classification task

Benchmarking Zero-Shot Recognition with Vision-Language Models: Challenges on Granularity and Specificity

Boosting Zero-Shot Crosslingual Performance using LLM-Based Augmentations with Effective Data Selection