Visualizing NLP annotations for Crowdsourcing

Hanchuan Li,Haichen Shen,Shengliang Xu,Congle Zhang

DOI: https://doi.org/10.48550/arXiv.1508.06044

2015-08-25

Abstract:Visualizing NLP annotation is useful for the collection of training data for the statistical NLP approaches. Existing toolkits either provide limited visual aid, or introduce comprehensive operators to realize sophisticated linguistic rules. Workers must be well trained to use them. Their audience thus can hardly be scaled to large amounts of non-expert crowdsourced workers. In this paper, we present CROWDANNO, a visualization toolkit to allow crowd-sourced workers to annotate two general categories of NLP problems: clustering and parsing. Workers can finish the tasks with simplified operators in an interactive interface, and fix errors conveniently. User studies show our toolkit is very friendly to NLP non-experts, and allow them to produce high quality labels for several sophisticated problems. We release our source code and toolkit to spur future research.

Computation and Language

What problem does this paper attempt to address?

The paper attempts to address the problem of how to enable non-expert crowd workers to efficiently and accurately annotate training data for natural language processing (NLP) tasks, particularly clustering and syntactic analysis tasks, by designing a visualization toolkit (CROWD ANNO).

### Specific Problem Background:
1. **Limitations of Existing Tools**:
- Existing NLP annotation tools either provide limited visualization support or introduce complex operations to implement intricate language rules.
- These tools usually require workers to undergo extensive training to use, making it difficult to scale to a large number of non-expert crowd workers.

2. **Challenges of Crowdsourced Annotation**:
- Non-expert workers are prone to errors when annotating NLP data, mainly due to:
- Complex language practices (e.g., the Penn Treebank guidelines exceed 300 pages).
- Operational complexity caused by intricate rules (e.g., identifying the part of speech for each node from hundreds of tags when generating parse trees).
- Many NLP problems are structured prediction problems, where labels are interdependent, requiring deep foresight and backtracking for each decision.

### Solution:
- **CROWD ANNO Toolkit**:
- Provides simplified operations: users only need to click and drag data, avoiding complex educational processes.
- Interactive interface: allows workers to efficiently read data dependencies and easily trial and error to correct mistakes.
- Versatility: the toolkit is designed simply to cover a broader range of NLP tasks, not just specific problems.

### Experimental Validation:
- The authors validated the effectiveness of CROWD ANNO through user studies. Experimental results show that non-expert workers using the toolkit can produce high-quality annotated data in clustering and syntactic analysis tasks, with significantly improved efficiency.

### Conclusion:
- The paper proposes a new visualization toolkit, CROWD ANNO, which significantly improves the annotation efficiency and accuracy of non-expert workers in NLP tasks through simplified operations and an interactive interface. Future work will include deploying the toolkit on actual crowdsourcing platforms to collect more real training data.