Abstract:With the rapid development of big data and cloud computing, data management has become increasingly challenging. Over the years, a number of frameworks for data management and storage with various characteristics and features have become available. Most of these are highly efficient, but ultimately create data silos. It becomes difficult to move and work coherently with data as new requirements emerge as no single framework can efficiently fulfill the data management needs of diverse applications. A possible solution is to design smart and efficient hierarchical (multi-tier) storage solutions. A hierarchical storage system (HSS) is a meta solution that consists of different storage frameworks organized as a jointly constructed large storage pool. It brings a number of benefits including better utilization of the storage, cost-efficiency, and use of different features provided by the underlying storage frameworks. In order to maximize the gains of hierarchical storage solutions, it is important that they include intelligent and autonomous mechanisms for data management grounded in the features of the different underlying frameworks. These decisions should be made according to the characteristics of the dataset, tier status, and access patterns. These are highly dynamic parameters and defining a policy based on the mentioned parameters is a non-trivial task. This paper presents an open-source hierarchical storage framework with a dynamic migration policy based on reinforcement learning (RL). We present a mathematical model, a software architecture, and an implementation based on both simulations and a live cloud-based environment. We compare the proposed RL-based strategy to a baseline of three rule-based policies, showing that the RL-based policy achieves significantly higher efficiency and optimal data distribution in different scenarios compared to the dynamic rule-based policies.

A Tree-Based Hierarchy Data Storage Framework in A Pervasive Space

A Dynamic Host Selection Algorithm for Layered Data Storage Architecture in a Pervasive Space

A New Linkless Hierarchical Structure Based on Perfect Hashing

A Tree-Based Hierarchy Framework for RFID Data Repository

The Hierarchical Degree-of-visibility Tree

Hierarchical Storage for Massive Social Network Data Based on Improved Decision Tree

Hierarchical Storage System Based on Wireless Mesh Network

An Efficient Hierarchical Data Placement Algorithm for Massive Spatial Data Storage Systems

Load Balanced and Efficient Hierarchical Data-Centric Storage in Sensor Networks

Distributed Storage for Wireless Sensor Networks Based on Wavelet Hierarchy

Hierarchical Coding for Cloud Storage: Topology-Adaptivity, Scalability, and Flexibility

Efficient Hierarchical Storage Management Framework Empowered by Reinforcement Learning

Hids: A Multifunctional Generator Of Hierarchical Data Streams

A Data-Centric Storage Approach for Efficient Query of Large-Scale Smart Grid.

D2-Tree: A Distributed Double-Layer Namespace Tree Partition Scheme for Metadata Management in Large-Scale Storage Systems

A data-centric storage approach for monitoring system of large-scale smart grid

A Universal Storage Architecture for Big Data in Cloud Environment

Research on data storage structure and management model in hierarchical storage

Design and Implementation of Hierarchical- Tree Assembly Data Model Based on Object Oriented

Research on Dynamic Storage Hierarchy of Data Grid Virtual Machine

NSSSD: A New Semantic Hierarchical Storage for Sensor Data