Abstract:Traditional data curation processes typically depend on human intervention. As data volume and variety grow exponentially, organizations are striving to increase efficiency of their data processes by automating manual processes and making them as unsupervised as possible. An additional challenge is to make these unsupervised processes scalable to meet the demands of increased data volume. This paper describes the parallelization of an unsupervised entity resolution (ER) process. ER is a component of many different data curation processes because it clusters records from multiple data sources that refer to the same real-world entity, such as the same customer, patient, or product. The ability to scale ER processes is particularly important because the computation effort of ER increases quadratically with data volume. The Data Washing Machine (DWM) is an already proposed unsupervised ER system which clusters references from diverse data sources. This work aims at solving the single-threaded nature of the DWM by adopting the parallelization nature of Hadoop MapReduce. However, the proposed parallelization method can be applied to both supervised systems, where matching rules are created by experts, and unsupervised systems, where expert intervention is not required. The DWM uses an entropy measure to self-evaluate the quality of record clustering. The current single-threaded implementations of the DWM in Python and Java are not scalable beyond a few 1,000 records and rely on large, shared memory. The objective of this research is to solve the major two shortcomings of the current design of the DWM which are the creation and usage of shared memory and lack of scalability by leveraging on the power of Hadoop MapReduce. We propose Hadoop Data Washing Machine (HDWM), a MapReduce implementation of the legacy DWM. The scalability of the proposed system is displayed using publicly available ER datasets. Based on results from our experiment, we conclude that HDWM can cluster from 1,000's to millions of equivalent references using multiple computational nodes with independent RAM and CPU cores.

REE-HDSC: Recognizing Extracted Entities for the Historical Database Suriname Curacao

Bench-Marking Information Extraction in Semi-Structured Historical Handwritten Records

Old Content and Modern Tools - Searching Named Entities in a Finnish OCRed Historical Newspaper Collection 1771-1910

Named Entity Recognition and Classification on Historical Documents: A Survey

Data Centric Domain Adaptation for Historical Text with OCR Errors

Unsilencing Colonial Archives via Automated Entity Recognition

Towards Population Reconstruction : Extraction of Family Relationships from Historical Documents

Dutch Named Entity Recognition and De-identification Methods for the Human Resource Domain

Contrastive Entity Coreference and Disambiguation for Historical Texts

Evaluating Dutch Named Entity Recognition and De-Identification Methods in the Human Resource Domain

End-to-End Approach for Recognition of Historical Digit Strings

RuREBus: a Case Study of Joint Named Entity Recognition and Relation Extraction from e-Government Domain

Exhaustive Entity Recognition for Coptic: Challenges and Solutions

Hacking History: Automatic Historical Event Extraction for Enriching Cultural Heritage Multimedia Collections

Transfer Learning across Several Centuries: Machine and Historian Integrated Method to Decipher Royal Secretary's Diary

x.ent: R Package for Entities and Relations Extraction based on Unsupervised Learning and Document Structure

hmBERT: Historical Multilingual Language Models for Named Entity Recognition

Harnessing Historical Corrections to build Test Collections for Named Entity Disambiguation

Advancements and Challenges in Handwritten Text Recognition: A Comprehensive Survey

Transformer-Based Approach for Joint Handwriting and Named Entity Recognition in Historical documents

A scalable MapReduce-based design of an unsupervised entity resolution system