Abstract:Human annotations are vital to supervised learning, yet annotators often disagree on the correct label, especially as annotation tasks increase in complexity. A strategy to improve label quality is to ask multiple annotators to label the same item and aggregate their labels. Many aggregation models have been proposed for categorical or numerical annotation tasks, but far less work has considered more complex annotation tasks involving open-ended, multivariate, or structured responses. While a variety of bespoke models have been proposed for specific tasks, our work is the first to introduce aggregation methods that generalize across many diverse complex tasks, including sequence labeling, translation, syntactic parsing, ranking, bounding boxes, and keypoints. This generality is achieved by devising a task-agnostic method to model distances between labels rather than the labels themselves. This article extends our prior work with investigation of three new research questions. First, how do complex annotation properties impact aggregation accuracy? Second, how should a task owner navigate the many modeling choices to maximize aggregation accuracy? Finally, what diagnoses can verify that aggregation models are specified correctly for the given data? To understand how various factors impact accuracy and to inform model selection, we conduct simulation studies and experiments on real, complex datasets. Regarding testing, we introduce unit tests for aggregation models and present a suite of such tests to ensure that a given model is not mis-specified and exhibits expected behavior. Beyond investigating these research questions above, we discuss the foundational concept of annotation complexity, present a new aggregation model as a bridge between traditional models and our own, and contribute a new semi-supervised learning method for complex label aggregation that outperforms prior work.

Aggregation Models for People Finding in Enterprise Corpora

Incorporating similar people for expert finding in enterprise corpora

Using Kullback-Leibler Divergence Language Models to Find Experts in Enterprise Corpora

Using Hierarchical Document Evaluation And Weighted Associations To Find Experts In Enterprise Corpora

A CDD-based formal model for expert finding.

Role Centralized Modeling for Expert Search in Enterprise Corporation

A Two-Stage Model for Expert Search

Distribution-Aware Crowdsourced Entity Collection

REC: A Novel Model to Rank Experts in Communities

Enhancing Question Answering for Enterprise Knowledge Bases using Large Language Models

Features and Aggregators for Web-scale Entity Search

Entity Matching Across Heterogeneous Sources

Influence Analysis Based Expert Finding Model and Its Applications in Enterprise Social Network

Public and Expert Collaborative Evaluation Model and Algorithm for Enterprise Knowledge

A General Model for Aggregating Annotations Across Simple, Complex, and Multi-Object Annotation Tasks

Learning a Probabilistic Semantic Model from Heterogeneous Social Networks for Relationship Identification

A Probabilistic Model for Fine-Grained Expert Search

Extracting Enterprises Collaborative Network from Massive Online Documents

BeamAggR: Beam Aggregation Reasoning over Multi-source Knowledge for Multi-hop Question Answering

CareerMiner: Automatic Extraction of Professional Network from Large Chinese Resume Data

Finding Appropriate Experts for Collaboration