Abstract:Large-scale data annotation is indispensable for many applications, such as machine learning and data integration. However, existing annotation solutions either incur expensive cost for large datasets or produce noisy results. This paper introduces a cost-effective annotation approach, and focuses on the labeling rule generation problem that aims to generate high-quality rules to largely reduce the labeling cost while preserving quality. To address the problem, we first generate candidate rules, and then devise a game-based crowdsourcing approach CROWDGAME to select high-quality rules by considering coverage and precision. CROWDGAME employs two groups of crowd workers: one group answers rule validation tasks (whether a rule is valid) to play a role of rule generator, while the other group answers tuple checking tasks (whether the annotated label of a data tuple is correct) to play a role of rule refuter. We let the two groups play a two-player game: rule generator identifies high-quality rules with large coverage and precision, while rule refuter tries to refute its opponent rule generator by checking some tuples that provide enough evidence to reject rules covering the tuples. This paper studies the challenges in CROWDGAME. The first is to balance the trade-off between coverage and precision. We define the loss of a rule by considering the two factors. The second is rule precision estimation. We utilize Bayesian estimation to combine both rule validation and tuple checking tasks. The third is to select crowdsourcing tasks to fulfill the game-based framework for minimizing the loss. We introduce a minimax strategy and develop efficient task selection algorithms. We conduct experiments on entity matching and relation extraction, and the results show that our method outperforms state-of-the-art solutions.

Rule Extraction Approach to Text Categorization Based on Multi-population Collaborative Optimization

Exploiting Multi-Category Characteristics and Unified Framework to Extract Web Content

Extracting Web Content by Exploiting Multi-Category Characteristics

Rough Rule Extracting from Various Conditions: Incremental and Approximate Approaches for Inconsistent Data.

An Approach to Automatic Text Categorization Based on Combination of Rule and Bayesian Method

Decision Rule Extraction for Regularized Multiple Criteria Linear Programming Model

Fast text categorization based on collaborative work in the semantic and class spaces

Aggressive Dimensionality Reduction With Reinforcement Local Feature Selection For Text Categorization

Enabling Regional Explainability by Automatic and Model-agnostic Rule Extraction

Evolutionary Computation and Rough Set-Based Hybrid Approach to Rule Generation

A heuristic method for discovering multi-class classification rules from multi-source data in cloud–edge system

Exploiting Textual and Visual Features for Image Categorization

AN APPROACH TO RULE EXTRACTION OF NEURAL NETWORKS

A method for extracting candidate rule information of text conversion based on user behavior

A Hybrid Rule Extraction Method Using Rough Sets and Neural Networks

Automated Text Data Extraction Based on Unsupervised Small Sample Learning

Rule Generation for Classification: Scalability, Interpretability, and Fairness

Cognitive Computing and Rule Extraction in Generalized One-sided Formal Contexts

Cost-Effective Data Annotation Using Game-Based Crowdsourcing

A novel three-stage multi-population evolutionary algorithm for constrained multi-objective optimization problems

Cooperative and Distributed Multiobjective Optimization for Heterogeneous Belief Rule Base