Abstract:A major challenge confronting today’s distributed metadata management schemes is how to meet the dynamic requirements of various applications through effectively mapping and migrating metadata nodes to different metadata servers (MDS’s). Most of the existing works dynamically reallocate nodes to different servers adopting history-based coarse-grained methods, failing to make a timely and efficient update on the distribution of nodes. In this paper, we present the first deep reinforcement learning-leveraged distributed metadata management scheme, AdaM, to address the aforementioned dilemma. AdaM is an adaptive fine-grained metadata management scheme that trains an actor-critic network to migrate “hot” metadata nodes to different MDS’s based on its observations of the current “states” (i.e., access pattern, the structure of namespace tree and current distribution of nodes on MDS’s). Adaptive to varying access patterns, AdaM can automatically migrate hot metadata nodes among servers to keep load balancing while maintaining metadata locality. Besides, we propose a self-adaptive metadata cache policy, which dynamically combines the two strategies of managing caches on the server side and the client side to gain better query performance. Last but not least, we design a distributed metadata processing 2PC Protocol called MST-based 2PC to ensure data consistency. Experiments on a real-world dataset demonstrate the superiority of our proposed method over other schemes.

Distributed Metadata Management Based on Hierarchical Bloom Filters in Data Grid

Architecture Design of Wide-area Distributed Real-time Database in Grid Dispatching Control System

An Adaptive Metadata Management Scheme Based on Deep Reinforcement Learning for Large-Scale Distributed File Systems.

A Metadata Framework For Distributed Geo-Spatial Databases In Grid Environment

AdaM: An Adaptive Fine-Grained Scheme for Distributed Metadata Management

Distributed storage and retrieval of massive small files metadata

MetaFlow: A Scalable Metadata Lookup Service for Distributed File Systems in Data Centers

CHMasters: A Scalable and Speed-Efficient Metadata Service in Distributed File System

A P2P Metadata Management Mechanism Based on Interest Aggregation

Clover: A Distributed File System of Expandable Metadata Service Derived from HDFS

Research and Design of MDS in Distributed Storage System

A Framework for Distributed Metadata Directory Service Based on Globe

Efficient Dynamic Management of Distributed Metadata

Research on Distributed Data Mining Technology Based on Grid.

A Hierarchical Grid Architecture Based on Computation/Application Metadata

D2-Tree: A Distributed Double-Layer Namespace Tree Partition Scheme for Metadata Management in Large-Scale Storage Systems

Metadata Management Scheme Based on Semantic Links for Grid Monitoring

Mds-based Geo-spatial Metadata Framework*

Counting Evolving Data Stream Based on Hierarchical Counting Bloom Filter

The Distributed Bloom Filter

HSM2 - A Hybrid and Scalable Metadata Management Method in Distributed File Systems.