Abstract:The characteristics of big data, including high volume, increased variety, and velocity, pose special challenges for data analysis. As these characteristics generally preclude manual data inspection and processing, researchers must often use computational methodologies to deal with this type of data; techniques that may be unfamiliar to nonspecialists, including behavioral scientists. However, previous data analytics methodologies within the field of computer science, developed to handle the generic tasks of data collection, preprocessing, and analysis, can be appropriated for use in other disciplines. These methodologies involve a sequential pipeline of quality checks to prepare data sets for analysis and application. Building upon these methodologies, this paper describes the Big Data Quality & Statistical Assurance (BDQSA) model, applicable for researchers in the behavioral sciences. It involves a series of data preprocessing tasks, to achieve data understanding, as well as data screening, cleaning, and transformation. These are followed by a statistical quality phase, which includes extraction of the relevant data subset, type conversions, ensuring sample representativeness when appropriate, and assessing statistical assumptions. The resulting model thereby provides methodological guidance for the preprocessing of behavioral science big data, aimed at ensuring acceptable data quality before analysis is undertaken. Sample R code snippets demonstrating the application of this model are provided throughout the paper.

Research on Data Preprocessing Methods for Big Data

A Survey on Big Data Pre-processing

A Combined Data Preprocessing Method Based on K-means Clustering and Singular Spectrum Analysis

Research on the Methods of Data Reduction in Data Preprocessing

A methodology for preprocessing structured big data in the behavioral sciences

The Theory and Methods of Data Preparation: an Overview

Data Preprocessing: The Techniques for Preparing Clean and Quality Data for Data Analytics Process

Case History Data Processing Based on Data Mining

Big Data Analytics Concepts, Technologies Challenges, and Opportunities

Data Preprocessing of Spatial Data Mining

A Review on Data Preprocessing Techniques Toward Efficient and Reliable Knowledge Discovery From Building Operational Data

Research on the data preprocessing technology in spatial date mining

Big Data Processing: Big Challenges and Opportunities

Traffic Data Processing At Age Of Big Data

Big Data Challenge: a Data Management Perspective.

Pre-Processing: A Data Preparation Step

Study on Data Preprocessing Arithmetic and Its Applications in Enterprise

Research Status and Prospect of Data Extraction and Cleaning Technology in Large Environment

Big Data Processing in Cloud Computing Environments

Research and Di cussion on the Big Data

Data Prtreatment in On-Line Data Mining