Abstract:BACKGROUND: Social media have transformed the communications landscape. People increasingly obtain news and health information online and via social media. Social media platforms also serve as novel sources of rich observational data for health research (including infodemiology, infoveillance, and digital disease detection detection). While the number of studies using social data is growing rapidly, very few of these studies transparently outline their methods for collecting, filtering, and reporting those data. Keywords and search filters applied to social data form the lens through which researchers may observe what and how people communicate about a given topic. Without a properly focused lens, research conclusions may be biased or misleading. Standards of reporting data sources and quality are needed so that data scientists and consumers of social media research can evaluate and compare methods and findings across studies.OBJECTIVE: We aimed to develop and apply a framework of social media data collection and quality assessment and to propose a reporting standard, which researchers and reviewers may use to evaluate and compare the quality of social data across studies.METHODS: We propose a conceptual framework consisting of three major steps in collecting social media data: develop, apply, and validate search filters. This framework is based on two criteria: retrieval precision (how much of retrieved data is relevant) and retrieval recall (how much of the relevant data is retrieved). We then discuss two conditions that estimation of retrieval precision and recall rely on--accurate human coding and full data collection--and how to calculate these statistics in cases that deviate from the two ideal conditions. We then apply the framework on a real-world example using approximately 4 million tobacco-related tweets collected from the Twitter firehose.RESULTS: We developed and applied a search filter to retrieve e-cigarette-related tweets from the archive based on three keyword categories: devices, brands, and behavior. The search filter retrieved 82,205 e-cigarette-related tweets from the archive and was validated. Retrieval precision was calculated above 95% in all cases. Retrieval recall was 86% assuming ideal conditions (no human coding errors and full data collection), 75% when unretrieved messages could not be archived, 86% assuming no false negative errors by coders, and 93% allowing both false negative and false positive errors by human coders.CONCLUSIONS: This paper sets forth a conceptual framework for the filtering and quality evaluation of social data that addresses several common challenges and moves toward establishing a standard of reporting social data. Researchers should clearly delineate data sources, how data were accessed and collected, and the search filter building process and how retrieval precision and recall were calculated. The proposed framework can be adapted to other public social media platforms.

A content analysis of metrics on online child sexual exploitation and abuse used by online content-sharing services

Putting the Count Back Into Accountability: An Analysis of Transparency Data About the Sexual Exploitation of Minors

Putting the Count Back Into Accountability: An Audit of Social Media Transparency Disclosures, Focusing on Sexual Exploitation of Minors

Content-Oblivious Trust and Safety Techniques: Results from a Survey of Online Service Providers

Garbage in, Garbage Out: Data Collection, Quality Assessment and Reporting Standards for Social Media Data Use in Health Research, Infodemiology and Digital Disease Detection

Thoughts on child safety on commodity platforms

Moderating online child sexual abuse material (CSAM): Does self-regulation work, or is greater state regulation needed?

The Trend of Online Child Sexual Abuse and Exploitations: A Profile of Online Sexual Offenders and Criminal Justice Response

What is Known About the Magnitude, Trend, and Risk for Child Sexual Abuse and the Commercial Sexual Exploitation of Children in the United States?

Development Of A Suite Of Attitudinal, Behavioral, And Contextual Measures To Assess Child Sexual Exploitation

Guarding the Guardians: Automated Analysis of Online Child Sexual Abuse

The prevalence of child sexual abuse with online sexual abuse added

“Our Laws Have Not Caught up with the Technology”: Understanding Challenges and Facilitators in Investigating and Prosecuting Child Sexual Abuse Materials in the United States

Investigating child sexual abuse material availability, searches, and users on the anonymous Tor network for a public health intervention strategy

Targeted and Troublesome: Tracking and Advertising on Children's Websites

Seeing without Looking: Analysis Pipeline for Child Sexual Abuse Datasets

Online Safety for Children and Youth under the 4Cs Framework—A Focus on Digital Policies in Australia, Canada, and the UK

Hiding in Plain Sight: A Measurement and Analysis of Kids' Exposure to Malicious URLs on YouTube

Objective Assessment of the Nature and Extent of Children's Internet-Based World: Protocol for the Kids Online Aotearoa Study

Online child pornography offender risk assessment using digital forensic artifacts: The need for a hybrid model

Estimating the economic burden attributable to online only child sexual abuse offenders: implications for police strategy