Abstract:Background: One in five U.S. adults lives with some kind of mental health condition and 4.6% of all U.S. adults have a serious mental illness. The Internet has become the first place for these people to seek online mental health information for help. However, online mental health information is not well-organized and often of low quality. There have been efforts in building evidence-based mental health knowledgebases curated with information manually extracted from the high-quality scientific literature. Manual extraction is inefficient. Crowdsourcing can potentially be a low-cost mechanism to collect labeled data from non-expert laypeople. However, there is not an existing annotation tool integrated with popular crowdsourcing platforms to perform the information extraction tasks. In our previous work, we prototyped a Semantic Text Annotation Tool (STAT) to address this gap. Objective: We aimed to refine the STAT prototype (1) to improve its usability and (2) to enhance the crowdsourcing workflow efficiency to facilitate the construction of evidence-based mental health knowledgebase, following a user-centered design (UCD) approach. Methods: Following UCD principles, we conducted four design iterations to improve the initial STAT prototype. In the first two iterations, usability testing focus groups were conducted internally with 8 participants recruited from a convenient sample, and the usability was evaluated with a modified System Usability Scale (SUS). In the following two iterations, usability testing was conducted externally using the Amazon Mechanical Turk (MTurk) platform. In each iteration, we summarized the usability testing results through thematic analysis, identified usability issues, and conducted a heuristic evaluation to map identified usability issues to Jakob Nielsen's usability heuristics. We collected suggested improvements in the usability testing sessions and enhanced STAT accordingly in the next UCD iteration. After four UCD iterations, we conducted a case study of the system on MTurk using mental health related scientific literature. We compared the performance of crowdsourcing workers with two expert annotators from two aspects: efficiency and quality. Results: The SUS score increased from 70.3 +/- 12.5 to 81.1 +/- 9.8 after the two internal UCD iterations as we improved STAT' s functionality based on the suggested improvements. We then evaluated STAT externally through MTurk in the following two iterations. The SUS score decreased to 55.7 +/- 20.1 in the third iteration, probably because of the complexity of the tasks. After further simplification of STAT and the annotation tasks with an improved annotation guideline, the SUS score increased to 73.8 +/- 13.8 in the fourth iteration of UCD. In the evaluation case study, on average, the workers spent 125.5 +/- 69.2 s on the onboarding tutorial and the crowdsourcing workers spent significantly less time on the annotation tasks compared to the two experts. In terms of annotation quality, the workers' annotation results achieved average Fl-scores ranged from 0.62 to 0.84 for the different sentences. Conclusions: We successfully developed a web-based semantic text annotation tool, STAT, to facilitate the curation of semantic web knowledgebases through four UCD iterations. The lessons learned from the UCD process could serve as a guide to further enhance STAT and the development and design of other crowdsourcing-based semantic text annotation tasks. Our study also showed that a well-organized, informative annotation guideline is as important as the annotation tool itself. Further, we learned that a crowdsourcing task should consist of multiple simple microtasks rather than a complicated task.

TSCSet: A Crowdsourced Time-Sync Comment Dataset for Exploration of User Experience Improvement.

Trust-Aware Sensing Quality Estimation for Team Crowdsourcing in Social IoT

Crowdsourced Time-Sync Video Tagging Using Semantic Association Graph

Automating Just-In-Time Comment Updating

Video Content Classification Using Time-Sync Comments and Titles

Time-Sync Video Tag Extraction Using Semantic Association Graph

Bridging Video Content And Comments: Synchronized Video Description With Temporal Summarization Of Crowdsourced Time-Sync Comments

Reading the Videos: Temporal Labeling for Crowdsourced Time-Sync Videos Based on Semantic Embedding.

Event Detection On Online Videos Using Crowdsourced Time-Sync Comment

On Efficient and Scalable Time-Continuous Spatial Crowdsourcing -- Full Version

TSTTC: A Large-Scale Dataset for Time-to-Contact Estimation in Driving Scenarios

A Crowdsourcing Framework for Collecting Tabular Data

Automatic Article Commenting: the Task and Dataset.

TDSRC: A Task-Distributing System of Crowdsourcing Based on Social Relation Cognition.

T-Crowd: Effective Crowdsourcing for Tabular Data

Time Expression Recognition and Normalization: a Survey

Temporal Treasure Hunt: Content-based Time Series Retrieval System for Discovering Insights

A Closer Look at Temporal Sentence Grounding in Videos: Dataset and Metric

Crowdsourcing System for Multi-object Annotation in Surveillance Videos

TRRS: Temporal Recurrent Recommender System based on Time-sync Comments

User-centered Design of a Web-Based Crowdsourcing-Integrated Semantic Text Annotation Tool for Building a Mental Health Knowledge Base.