Abstract:Background: One in five U.S. adults lives with some kind of mental health condition and 4.6% of all U.S. adults have a serious mental illness. The Internet has become the first place for these people to seek online mental health information for help. However, online mental health information is not well-organized and often of low quality. There have been efforts in building evidence-based mental health knowledgebases curated with information manually extracted from the high-quality scientific literature. Manual extraction is inefficient. Crowdsourcing can potentially be a low-cost mechanism to collect labeled data from non-expert laypeople. However, there is not an existing annotation tool integrated with popular crowdsourcing platforms to perform the information extraction tasks. In our previous work, we prototyped a Semantic Text Annotation Tool (STAT) to address this gap. Objective: We aimed to refine the STAT prototype (1) to improve its usability and (2) to enhance the crowdsourcing workflow efficiency to facilitate the construction of evidence-based mental health knowledgebase, following a user-centered design (UCD) approach. Methods: Following UCD principles, we conducted four design iterations to improve the initial STAT prototype. In the first two iterations, usability testing focus groups were conducted internally with 8 participants recruited from a convenient sample, and the usability was evaluated with a modified System Usability Scale (SUS). In the following two iterations, usability testing was conducted externally using the Amazon Mechanical Turk (MTurk) platform. In each iteration, we summarized the usability testing results through thematic analysis, identified usability issues, and conducted a heuristic evaluation to map identified usability issues to Jakob Nielsen's usability heuristics. We collected suggested improvements in the usability testing sessions and enhanced STAT accordingly in the next UCD iteration. After four UCD iterations, we conducted a case study of the system on MTurk using mental health related scientific literature. We compared the performance of crowdsourcing workers with two expert annotators from two aspects: efficiency and quality. Results: The SUS score increased from 70.3 +/- 12.5 to 81.1 +/- 9.8 after the two internal UCD iterations as we improved STAT' s functionality based on the suggested improvements. We then evaluated STAT externally through MTurk in the following two iterations. The SUS score decreased to 55.7 +/- 20.1 in the third iteration, probably because of the complexity of the tasks. After further simplification of STAT and the annotation tasks with an improved annotation guideline, the SUS score increased to 73.8 +/- 13.8 in the fourth iteration of UCD. In the evaluation case study, on average, the workers spent 125.5 +/- 69.2 s on the onboarding tutorial and the crowdsourcing workers spent significantly less time on the annotation tasks compared to the two experts. In terms of annotation quality, the workers' annotation results achieved average Fl-scores ranged from 0.62 to 0.84 for the different sentences. Conclusions: We successfully developed a web-based semantic text annotation tool, STAT, to facilitate the curation of semantic web knowledgebases through four UCD iterations. The lessons learned from the UCD process could serve as a guide to further enhance STAT and the development and design of other crowdsourcing-based semantic text annotation tasks. Our study also showed that a well-organized, informative annotation guideline is as important as the annotation tool itself. Further, we learned that a crowdsourcing task should consist of multiple simple microtasks rather than a complicated task.

TeamTat: a collaborative text annotation tool

DaTo: an Atlas of Biological Databases and Tools

GATE Teamware: a web-based, collaborative text annotation framework

TACIT: An open-source text analysis, crawling, and interpretation tool

FunctionAnnotator, a versatile and efficient web tool for non-model organism annotation

MarkIt: A Collaborative Artificial Intelligence Annotation Platform Leveraging Blockchain For Medical Imaging Research

MetaTron: advancing biomedical annotation empowering relation annotation and collaboration

EXACT: a collaboration toolset for algorithm-aided annotation of images with annotation version control

Collaborative Semantic Annotation Tooling (CoAT) to Improve Efficiency and Plug-and-Play Semantic Interoperability in the Secondary Use of Medical Data: Concept, Implementation, and First Cross-Institutional Experiences

Quick Annotator: an open-source digital pathology based rapid image annotation tool

TEtrimmer: a novel tool to automate the manual curation of transposable elements

Annotating publicly-available samples and studies using interpretable modeling of unstructured metadata

User-centered Design of a Web-Based Crowdsourcing-Integrated Semantic Text Annotation Tool for Building a Mental Health Knowledge Base.

Bakta: rapid and standardized annotation of bacterial genomes via alignment-free sequence identification

A crowdsourcing semi-automatic image segmentation platform for cell biology

POTATO: The Portable Text Annotation Tool

GeoAnnotator: A Collaborative Semi-Automatic Platform for Constructing Geo-Annotated Text Corpora

DeTEXT: A Database for Evaluating Text Extraction from Biomedical Literature Figures

Collaborative Annotation of Medical Images Via Web Browser for Teleradiology

PIAT: An Evolutionarily Intelligent System for Deep Phenotyping of Chinese Electronic Health Records

GenomeTools: a comprehensive software library for efficient processing of structured genome annotations