Abstract:With the growing prevalence of large language models, it is increasingly common to annotate datasets for machine learning using pools of crowd raters. However, these raters often work in isolation as individual crowdworkers. In this work, we regard annotation not merely as inexpensive, scalable labor, but rather as a nuanced interpretative effort to discern the meaning of what is being said in a text. We describe a novel, collaborative, and iterative annotator-in-the-loop methodology for annotation, resulting in a 'Bridging Benchmark Dataset' of comments relevant to bridging divides, annotated from 11,973 textual posts in the Civil Comments dataset. The methodology differs from popular anonymous crowd-rating annotation processes due to its use of an in-depth, iterative engagement with seven US-based raters to (1) collaboratively refine the definitions of the to-be-annotated concepts and then (2) iteratively annotate complex social concepts, with check-in meetings and discussions. This approach addresses some shortcomings of current anonymous crowd-based annotation work, and we present empirical evidence of the performance of our annotation process in the form of inter-rater reliability. Our findings indicate that collaborative engagement with annotators can enhance annotation methods, as opposed to relying solely on isolated work conducted remotely. We provide an overview of the input texts, attributes, and annotation process, along with the empirical results and the resulting benchmark dataset, categorized according to the following attributes: Alienation, Compassion, Reasoning, Curiosity, Moral Outrage, and Respect.

Bureau for Rapid Annotation Tool - Collaboration Can Do More over Variety-oriented Annotations.

TeamTat: a collaborative text annotation tool

Making Flexible Use of Sub-tasks: A Multiplex Interaction Network for Unified Aspect-based Sentiment Analysis

GeoAnnotator: A Collaborative Semi-Automatic Platform for Constructing Geo-Annotated Text Corpora

Enhancing Text Annotation through Rationale-Driven Collaborative Few-Shot Prompting

ChemBrowser: a flexible framework for mining chemical documents

Annotator in the Loop: A Case Study of In-Depth Rater Engagement to Create a Bridging Benchmark Dataset

MetaTron: advancing biomedical annotation empowering relation annotation and collaboration

YEDDA: A Lightweight Collaborative Text Span Annotation Tool

GATE Teamware: a web-based, collaborative text annotation framework

A Blackboard Model for Flexible and Parallel Text Annotation

FunctionAnnotator, a versatile and efficient web tool for non-model organism annotation

Augmented Behavioral Annotation Tools, with Application to Multimodal Datasets and Models: A Systematic Review

Research on Corpus Annotation Method Based on Collective Intelligence

Faster Human-Machine Collaboration Bounding Box Annotation Framework Based on Active Learning

KCAT: A Knowledge-Constraint Typing Annotation Tool

Visualizing NLP annotations for Crowdsourcing

CAMRA: Copilot for AMR Annotation

OpenAnnotate2: Multi-Modal Auto-Annotating for Autonomous Driving

Overview of Annotation Creation: Processes & Tools

EASE: an Easily-Customized Annotation System Powered by Efficiency Enhancement Mechanisms