Data Nuggets: A Method for Reducing Big Data While Preserving Data Structure

Traymon E. Beavers,Ge Cheng,Yajie Duan,Javier Cabrera,Mariusz Lubomirski,Dhammika Amaratunga,Jeffrey E. Teigler

2024-03-06

Abstract:Big data, with NxP dimension where N is extremely large, has created new challenges for data analysis, particularly in the realm of creating meaningful clusters of data. Clustering techniques, such as K-means or hierarchical clustering are popular methods for performing exploratory analysis on large datasets. Unfortunately, these methods are not always possible to apply to big data due to memory or time constraints generated by calculations of order PxN(N-1). To circumvent this problem, typically, the clustering technique is applied to a random sample drawn from the dataset: however, a weakness is that the structure of the dataset, particularly at the edges, is not necessarily maintained. We propose a new solution through the concept of "data nuggets", which reduce a large dataset into a small collection of nuggets of data, each containing a center, weight, and scale parameter. The data nuggets are then input into algorithms that compute methods such as principal components analysis and clustering in a more computationally efficient manner. We show the consistency of the data nuggets-based covariance estimator and apply the methodology of data nuggets to perform exploratory analysis of a flow cytometry dataset containing over one million observations using PCA and K-means clustering for weighted observations. Supplementary materials for this article are available online.

Computation,Methodology

What problem does this paper attempt to address?

This paper primarily investigates the challenges encountered in the analysis of big data (high-dimensional large sample data), particularly the computational and storage limitations in clustering analysis. Existing methods such as K-means or hierarchical clustering cannot be directly applied to large-scale data due to their high computational complexity. To address this problem, the paper proposes the concept of "Data Nuggets," which is a method of compressing big data into a small collection of datasets with central, weighting, and scaling parameters. Through Data Nuggets, principal component analysis and clustering can be performed more efficiently while maintaining the integrity of the data structure. The process of generating Data Nuggets involves dividing the large sample into equally sized small blocks (data nuggets), each containing central, weighting, and scaling information. These nuggets are then used to execute algorithms such as weighted K-means clustering to reduce computational requirements. The paper demonstrates the effectiveness of the Data Nuggets method through simulation experiments and a real flow cytometry dataset (containing over one million observations), comparing it with methods such as random sampling and support points. In addition, the paper introduces two R packages for implementing this method and discusses potential future research directions. Overall, this study proposes a novel data reduction strategy aimed at addressing the computational efficiency and resource limitations in big data analysis, particularly in preserving data structures in edge regions.

Data Nuggets: A Method for Reducing Big Data While Preserving Data Structure

Two step clustering for data reduction combining DBSCAN and k-means clustering

A Clustered Dwarf Structure to Speed Up Queries on Data Cubes

A Data-Driven Approach for Extracting Representative Information from Large Datasets with Mixed Attributes

A Novel Method for Clustering Cellular Data to Improve Classification

CDC: A Simple Framework for Complex Data Clustering

Clustering to Reduce Spatial Data Set Size

Principal component analysis and clustering on manifolds

Min max kurtosis distance based improved initial centroid selection approach of K-means clustering for big data mining on gene expression data

Uniform-in-Phase-Space Data Selection with Iterative Normalizing Flows

Big Data, Big Challenges

Hyperplane Division in Fuzzy C-Means: Clustering Big Data

Quantifying and Visualizing Hidden Preferential Aggregations Amid Heterogeneity

A new model for natural groupings in high-dimensional data

A new approach to generate diversified clusters for small data sets

Reducing Microarray Data Via Nonnegative Matrix Factorization for Visualization and Clustering Analysis

Determination of essential phenotypic elements of clusters in high-dimensional entities—DEPECHE

Clustering Large Datasets by Merging K-Means Solutions

Towards effective and efficient mining of arbitrary shaped clusters

Non-Redundant Subspace Clusterings with Nr-Kmeans and Nr-DipMeans

xk-split:A Split Clustering Algorithm Bases on k-medoids