Abstract:Recently using machine learning (ML) based techniques to optimize modern database management systems has attracted intensive interest from both industry and academia. With an objective to tune a specific component of a DBMS (e.g., index selection, knobs tuning), the ML-based tuning agents have shown to be able to find better configurations than experienced database administrators. However, one critical yet challenging question remains unexplored -- how to make those ML-based tuning agents work collaboratively. Existing methods do not consider the dependencies among the multiple agents, and the model used by each agent only studies the effect of changing the configurations in a single component. To tune different components for DBMS, a coordinating mechanism is needed to make the multiple agents cognizant of each other. Also, we need to decide how to allocate the limited tuning budget among the agents to maximize the performance. Such a decision is difficult to make since the distribution of the reward for each agent is unknown and non-stationary. In this paper, we study the above question and present a unified coordinating framework to efficiently utilize existing ML-based agents. First, we propose a message propagation protocol that specifies the collaboration behaviors for agents and encapsulates the global tuning messages in each agent's model. Second, we combine Thompson Sampling, a well-studied reinforcement learning algorithm with a memory buffer so that our framework can allocate budget judiciously in a non-stationary environment. Our framework defines the interfaces adapted to a broad class of ML-based tuning agents, yet simple enough for integration with existing implementations and future extensions. We show that it can effectively utilize different ML-based agents and find better configurations with 1.4~14.1X speedups on the workload execution time compared with baselines.

Tempo: Robust and Self-Tuning Resource Management in Multi-tenant Parallel Databases

Schema-Driven Performance Evaluation for Highly Concurrent Scenarios.

Time-sharing Parallel Applications Through Performance-Targeted Feedback-Controlled Real-Time Scheduling.

ResTune: Resource Oriented Tuning Boosted by Meta-Learning for Cloud Databases

Supporting Cost-Efficient Multi-Tenant Database Services With Service Level Objectives (Slos)

Autonomic Architecture for Big Data Performance Optimization

TempoScale: A Cloud Workloads Prediction Approach Integrating Short-Term and Long-Term Information

Metis: Robustly Optimizing Tail Latencies Of Cloud Systems

A Unified and Efficient Coordinating Framework for Autonomous DBMS Tuning

Tromino: Demand and DRF Aware Multi-Tenant Queue Manager for Apache Mesos Cluster

Optimistic Transaction Processing in Deterministic Database

Improving resource utilization by timely fine-grained scheduling

DMR API: Improving cluster productivity by turning applications into malleable

Design and Implementation of Distributed Resource Management for Time Sensitive Applications

A demonstration of the ottertune automatic database management system tuning service

Endure: A Robust Tuning Paradigm for LSM Trees Under Workload Uncertainty

Intelligent Management of Virtualized Resources for Database Systems in Cloud Environment

SHA: QoS-Aware Software and Hardware Auto-Tuning for Database Systems

Fine-Grained Modeling and Optimization for Intelligent Resource Management in Big Data Processing

Fast In-Memory Transaction Processing Using Rdma and Htm

Towards Optimal Transaction Scheduling