Abstract:Data lakes are widely used to store extensive and heterogeneous datasets for advanced analytics. However, the unstructured nature of data in these repositories introduces complexities in exploiting them and extracting meaningful insights. This motivates the need of exploring efficient approaches for consolidating data lakes and deriving a common and unified schema. This paper introduces a practical data visualization and analysis approach rooted in Formal Concept Analysis (FCA) to systematically clean, organize, and design data structures within a data lake. We explore diverse data structures stored in our data lake at Infologic, including InfluxDB measurements and Elasticsearch indexes, aiming to derive conventions for a more accessible data model. Leveraging FCA, we represent data structures as objects, analyze the concept lattice, and present two strategies-top-down and bottom-up-to unify these structures and establish a common schema. Our methodology yields significant results, enabling the identification of common concepts in the data structures, such as resources along with their underlying shared fields (timestamp, type, usedRatio, etc.). Moreover, the number of distinct data structure field names is reduced by 54 percent (from 190 to 88) in the studied subset of our data lake. We achieve a complete coverage of 80 percent of data structures with only 34 distinct field names, a significant improvement from the initial 121 field names that were needed to reach such coverage. The paper provides insights into the Infologic ecosystem, problem formulation, exploration strategies, and presents both qualitative and quantitative results.

Analytic Processing in Data Lakes: A Semantic Query-Driven Discovery Approach

Semantic Data Management in Data Lakes

Data Warehouse Native Feature Based OLAP Querying with Keywords

Retrieve, Merge, Predict: Augmenting Tables with Data Lakes

Semantic Web-Based Data Service Discovery and Composition

Toward data lakes as central building blocks for data management and analysis

Manipulating Data Lakes Intelligently With Java Annotations

On data lake architectures and metadata management

Cross Modal Data Discovery over Structured and Unstructured Data Lakes

Self-supervised data lakes discovery through unsupervised metadata-driven weighted similarity

Semantic orchestration and exploitation of material data: A dataspace solution demonstrated on steel and copper applications

Exploiting Formal Concept Analysis for Data Modeling in Data Lakes

Finding Related Tables in Data Lakes for Interactive Data Science

Searching Data Lakes for Nested and Joined Data

Optimizing Federated Queries Based on the Physical Design of a Data Lake

Data Lakes for Digital Humanities

Blend: A Unified Data Discovery System

Realising Data-Centric Scientific Workflows with Provenance-Capturing on Data Lakes

Data Lakehouse: Next Generation Information System

A Zone-Based Data Lake Architecture for IoT, Small and Big Data

Semantic-based Big Data integration framework using scalable distributed ontology matching strategy