Abstract:The unprecedented development and popularization of the Internet, combined with the emergence of a variety of modern applications, such as search engines, online transactions, climate warning systems and so on, enables the worldwide storage of data to grow unprecedented. Efficient storage, management and processing of such huge amounts of data has become an important academic research topic. The detection and removal of duplicate and redundant data from such multi-trillion data, while ensuring resource and computational efficiency, has constituted a challenging area of research.Because of the fact that all the data of potentially unbounded data streams can not be stored, and the need to delete duplicated data as accurately as possible, intelligent approximate duplicate data detection algorithms are urgently required. Many well-known methods based on the bitmap structure, Bloom Filter and its variants are listed in the literature. In this paper, we propose a new data structure, Improved Streaming Quotient Filter (ISQF), to efficiently detect and remove duplicate data in a data stream. ISQF intelligently stores the signatures of elements in a data stream, while using an eviction strategy to provide near zero error rates. We show that ISQF achieves near optimal performance with fairly low memory requirements, making it an ideal and efficient method for repeated data detection. It has a very low error rate. Empirically, we compared ISQF with some existing methods (especially Steaming Quotient Filter (SQF)). The results show that our proposed method outperforms theexisting methods in terms of memory usage and accuracy.We also discuss the parallel implementation of ISQF

Longest Increasing Subsequence Computation over Streaming Sequences

Efficient Algorithms for Finding a Longest Common Increasing Subsequence

LIL for the Length of the Longest Increasing Subsequences

The Longest Wave Subsequence Problem: Generalizations of the Longest Increasing Subsequence Problem

Space-Efficient Subgraph Search Over Streaming Graph With Timing Order Constraint

Novel structures for counting frequent items in time decayed streams

A simple linear space algorithm for computing a longest common increasing subsequence

Distributed scheduling and storage scheme based on LSM-OCTree for spatiotemporal stream

Finding Frequent Items in Time Decayed Data Streams.

LIFOSS: a learned index scheme for streaming scenarios

A Space-Bounded Anytime Algorithm for the Multiple Longest Common Subsequence Problem

The Longest Almost Increasing Subsequence Problem with Sliding Windows

SeqStream: Mining Closed Sequential Patterns over Stream Sliding Windows

Time Constrained Continuous Subgraph Search over Streaming Graphs

Web Technologies and Applications

Fast continuous streaming sort in big streaming data environment under fixed-size single storage

Matching Consecutive Subpatterns over Streaming Time Series.

Improved Streaming Quotient Filter: A Duplicate Detection Approach for Data Streams

Dolha - an Efficient and Exact Data Structure for Streaming Graphs

Lc‐Stream: An elastic scheduling strategy with latency constraints in geo‐distributed stream computing environments

LM-SRPQ: Efficiently Answering Regular Path Query in Streaming Graphs.