Abstract:Data lakes are widely used to store extensive and heterogeneous datasets for advanced analytics. However, the unstructured nature of data in these repositories introduces complexities in exploiting them and extracting meaningful insights. This motivates the need of exploring efficient approaches for consolidating data lakes and deriving a common and unified schema. This paper introduces a practical data visualization and analysis approach rooted in Formal Concept Analysis (FCA) to systematically clean, organize, and design data structures within a data lake. We explore diverse data structures stored in our data lake at Infologic, including InfluxDB measurements and Elasticsearch indexes, aiming to derive conventions for a more accessible data model. Leveraging FCA, we represent data structures as objects, analyze the concept lattice, and present two strategies-top-down and bottom-up-to unify these structures and establish a common schema. Our methodology yields significant results, enabling the identification of common concepts in the data structures, such as resources along with their underlying shared fields (timestamp, type, usedRatio, etc.). Moreover, the number of distinct data structure field names is reduced by 54 percent (from 190 to 88) in the studied subset of our data lake. We achieve a complete coverage of 80 percent of data structures with only 34 distinct field names, a significant improvement from the initial 121 field names that were needed to reach such coverage. The paper provides insights into the Infologic ecosystem, problem formulation, exploration strategies, and presents both qualitative and quantitative results.

Exploring software engineering data with formal concept analysis

SE-FCA: A Model of Software Evolution with Formal Concept Analysis

Using Formal Concept Analysis for Software Engineering

A Fuzzy Formal Concept Analysis Based Approach for Business Component Identification

A New Algebraic Structure for Formal Concept Analysis

Formal Plan Recognizing Technique Based on Semantic Layer Software Comprehension

Using Formal Concept Analysis to support change analysis.

A comprehensive survey on formal concept analysis, its research trends and applications

B-FES/o: an ontology-based scheme for functional modeling in design for semantic Web applications

Exploiting Formal Concept Analysis for Data Modeling in Data Lakes

An incremental algorithm for concept lattice based on structural similarity index

Management and Retrieval of Web Services Based on Formal Concept Analysis

Considering the Relationship between RST and FCA

Formal Concept Analysis on Fuzzy Sets

A Study on Information Granularity in Formal Concept Analysis Based on Concept-Bases

Formal concept analysis based schema matching

Concept Analysis Via Rough Set and AFS Algebra

Analyzing Data Flow Diagrams by Combination of Formal Methods and Visualization Techniques.

Integrating Software Engineering Data Using Semantic Web Technologies

Service System Based on Formal Concept Analysis Theory

Visualizing and Interacting with Concept Hierarchies