Abstract:Recently, researches on dynamic decision-making based on streaming data are in full swing. As an indispensable technology for data management and analysis, indexing methods are also evolving. The indexing paradigm named learned index has been proposed to replace the traditional B-tree family in some scenarios. It has been proved that learned indexes can provide higher lookup efficiency and lower storage cost overhead than traditional indexes. Usually, learned indexes assume that the data is static or at least the data distribution is unchanged. However, the streaming scenarios break the strong assumption. This paper presents a learned index scheme for streaming scenarios (LIFOSS for short), where the workloads insert, delete, and query data arbitrarily. Precisely, LIFOSS consists of three parts: a) an adaptive packed-memory array which stores data and handles updates with lower bound of performance guaranteed; b) a middle-layer model group, used to fit the cumulative distribution function of data; c) a feedback mechanism designed to update parameters of the model group above in real-time locally. Extensive experiments on two public datasets show that LIFOSS performs better than the state-of-the-art dynamic learned index method. LIFOSS reduces the lookup latency by at least 6%\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$6\%$$\end{document}, and its dynamic performance is more stable, requiring only a tiny amount of extra space.

Distributed scheduling and storage scheme based on LSM-OCTree for spatiotemporal stream

A Workload-Controllable Dynamic Spatio-Temporal Index Scheme for Streaming Processing

A Study of Performance Optimization Method for Massive Spaito-temporal Data Based on Spatio-temporal Partition Clustering

Mining Spread Patterns of Spatio-temporal Co-occurrences over Zones

Study of Seamless Organization and Storage Structure for Massive Spatio-Temporal Data

<i>SA-LSM</i>: Optimize Data Layout for LSM-tree Based Storage using Survival Analysis

HBaseSpatial: A Scalable Spatial Data Storage Based on HBase

SA-LSM

Time-tired compaction: An elastic compaction scheme for LSM-tree based time-series database

Lc‐Stream: An elastic scheduling strategy with latency constraints in geo‐distributed stream computing environments

QBS-Tree: A Spatial Index with High Update Efficiency for Real-Time Processing System

An On-the-Fly Scheduling Strategy for Distributed Stream Processing Platform.

LIFOSS: a learned index scheme for streaming scenarios

An Efficient Graph-Based Spatio-Temporal Indexing Method for Task-Oriented Multi-Modal Scene Data Organization

Distributed and Parallel Processing for Real-Time and Dynamic Spatio-Temporal Graph

Real-Time Spatial Queries for Moving Objects Using Storm Topology.

A Flexible Spatio-Temporal Indexing Scheme for Large-Scale GPS Track Retrieval

Adaptive Scheduling Strategy For Data Stream Management System

A Real-Time Scheduling Strategy Based on Processing Framework of Hadoop

Plsm: A Highly Efficient LSM-Tree Index Supporting Real-Time Big Data Analysis

A spatiotemporal data model and an index structure for computational time geography