Abstract:Several applications require counting the number of distinct items in the data, which is known as the cardinality counting problem. Example applications include health applications such as rare disease patients counting for adequate awareness and funding, and counting the number of cases of a new disease for outbreak detection, marketing applications such as counting the visibility reached for a new product, and cybersecurity applications such as tracking the number of unique views of social media posts. The data needed for the counting is however often personal and sensitive, and need to be processed using privacy-preserving techniques. The quality of data in different databases, for example typos, errors and variations, poses additional challenges for accurate cardinality estimation. While privacy-preserving cardinality counting has gained much attention in the recent times and a few privacy-preserving algorithms have been developed for cardinality estimation, no work has so far been done on privacy-preserving cardinality counting using record linkage techniques with fuzzy matching and provable privacy guarantees. We propose a novel privacy-preserving record linkage algorithm using unsupervised clustering techniques to link and count the cardinality of individuals in multiple datasets without compromising their privacy or identity. In addition, existing Elbow methods to find the optimal number of clusters as the cardinality are far from accurate as they do not take into account the purity and completeness of generated clusters. We propose a novel method to find the optimal number of clusters in unsupervised learning. Our experimental results on real and synthetic datasets are highly promising in terms of significantly smaller error rate of less than 0.1 with a privacy budget {\epsilon} = 1.0 compared to the state-of-the-art fuzzy matching and clustering method.

A Research on Mapreduce-Based Redundancy Pruning for Record Linkage

Join Query Optimization Based on MapReduce under Skewed Data

A practical approach for scalable record linkage on hadoop

A Unified Record Linkage Strategy for Web Service Data

MapDupReducer: detecting near duplicates over massive datasets.

Implementing a Hash-Based Privacy-Preserving Record Linkage Tool in the OneFlorida Clinical Research Network.

A Fuzzy Approach to Record Linkages

Efficient Techniques for Online Record Linkage

Efficient Processing of k Nearest Neighbor Joins using MapReduce

MtMR: Ensuring MapReduce Computation Integrity with Merkle Tree-Based Verifications

Privacy-Preserving Record Linkage for Cardinality Counting

MELODY-JOIN: Efficient Earth Mover's Distance Similarity Joins Using MapReduce

Resolving Object Coreference in Semantic Web Based on MapReduce

Efficient Near-Duplicate Detection for Q&A Forum.

A New Method of Redundancy Analysis in Data Reconciliation and Its Application

Distributed Affinity Propagation Clustering Based on MapReduce

On the Relation Redundancy in Fuzzy Databases

A New Method for Redundancy Analysis in Feature Selection

Research on Parallel Duplicated Webpage Deletion Based on MapReduce Model

SMERED: A Bayesian Approach to Graphical Record Linkage and De-duplication

Manual evaluation of record linkage algorithm performance in four real-world datasets