Abstract:The relational DBMS (RDBMS) has been widely used since it supports various high-level functionalities such as SQL, schemas, indexes, and transactions that do not exist in the O/S file system. But, a recent advent of big data technology facilitates development of new systems that sacrifice the DBMS functionality in order to efficiently manage large-scale data. Those so-called NoSQL systems use a distributed file system, which support scalability and reliability. They support scalability of the system by storing data into a large number of low-cost commodity hardware and support reliability by storing the data in replica. However, they have a drawback that they do not adequately support high-level DBMS functionality. In this paper, we propose an architecture of a DBMS that uses the DFS as storage. With this novel architecture, the DBMS is capable of supporting scalability and reliability of the DFS as well as high-level functionality of DBMS. Thus, a DBMS can utilize a virtually unlimited storage space provided by the DFS, rendering it to be suitable for big data analytics. As part of the architecture of the DBMS, we propose the notion of the meta DFS file, which allows the DBMS to use the DFS as the storage, and an efficient transaction management method including recovery and concurrency control. We implement this architecture in Odysseus/DFS, an integration of the Odysseus relational DBMS, that has been being developed at KAIST for over 24 years, with the DFS. Our experiments on transaction processing show that, due to the high-level functionality of Odysseus/DFS, it outperforms Hbase, which is a representative open-source NoSQL system. We also show that, compared with an RDBMS with local storage, the performance of Odysseus/DFS is comparable or marginally degraded, showing that the overhead of Odysseus/DFS for supporting scalability by using the DFS as the storage is not significant.

SolarDB

Architecture Design of Wide-area Distributed Real-time Database in Grid Dispatching Control System

Distributed storage and processing of real-time data in smart grid dispatching and control systems

G-Tran: A High Performance Distributed Graph Database with a Decentralized Architecture

Towards a Non-2PC Transaction Management in Distributed Database Systems

SolsDB: Solve the Ethereum's bottleneck caused by storage engine

Fast In-Memory Transaction Processing Using Rdma and Htm

Key Technologies of Distributed Transactional Database Storage Engine

InfiniteDB: a pc-cluster based parallel massive database management system.

A Tree-structured Database Machine for Large Relational Database Systems

A practical scalable distributed B-tree

Odysseus/DFS: Integration of DBMS and Distributed File System for Transaction Processing of Big Data

Blockchains and Distributed Databases: a Twin Study

Research on distributed relational database based on MySQL

Skyline-Join in Distributed Databases

Design of A More Scalable Database System

Cluster Based Parallel Database Management System for Data Intensive Computing

Eventdb: A Large-Scale Semi-Structured Scientific Data Management System

H-DB: Yet Another Big Data Hybrid System of Hadoop and DBMS

Optimistic Transaction Processing in Deterministic Database

System G Distributed Graph Database