Efficient document analytics on compressed data

ZhangFeng,ZhaiJidong,ShenXipeng,MutluOnur,ChenWenguang

2018-01-01

Abstract:Today\u0027s rapidly growing document volumes pose pressing challenges to modern document analytics, in both space usage and processing time. In this work, we propose the concept of compression-based di...

What problem does this paper attempt to address?

Efficient Document Analytics on Compressed Data

Feng Zhang,Jidong Zhai,Xipeng Shen,Onur Mutlu,Wenguang Chen

DOI: https://doi.org/10.14778/3236187.3236203

IF: 2.5

2018-01-01

Proceedings of the VLDB Endowment

Abstract:Today's rapidly growing document volumes pose pressing challenges to modern document analytics, in both space usage and processing time. In this work, we propose the concept of compression-based direct processing to alleviate issues in both dimensions. The main idea is to enable direct document analytics on compressed data. We present how the concept can be materialized on Sequitur, a compression algorithm that produces hierarchical grammar-like representations. We discuss the major challenges in applying the idea to various document analytics tasks, and reveal a set of guidelines and also assistant software modules for developers to effectively apply compression-based direct processing . Experiments show that our proposed techniques save 90.8% storage space and 77.5% memory usage, while speeding up data processing significantly, i.e., by 1.6X on sequential systems, and 2.2X on distributed clusters, on average.
TADOC: Text Analytics Directly on Compression

Zhang Feng,Zhai Jidong,Shen Xipeng,Wang Dalin,Chen Zheng,Mutlu Onur,Chen Wenguang,Du Xiaoyong

DOI: https://doi.org/10.1007/s00778-020-00636-3

2020-01-01

Abstract:This article provides a comprehensive description of text analytics directly on compression (TADOC), which enables direct document analytics on compressed textual data. The article explains the concept of TADOC and the challenges to its effective realizations. Additionally, a series of guidelines and technical solutions that effectively address those challenges, including the adoption of a hierarchical compression method and a set of novel algorithms and data structure designs, are presented. Experiments on six data analytics tasks of various complexities show that TADOC can save 90.8% storage space and 87.9% memory usage, while halving data processing times.
CompressDB: Enabling Efficient Compressed Data Direct Processing for Various Databases

Feng Zhang,Weitao Wan,Chenyang Zhang,Jidong Zhai,Yunpeng Chai,Haixiang Li,Xiaoyong Du

DOI: https://doi.org/10.1145/3514221.3526130

2022-01-01

Abstract:In modern data management systems, directly performing operations on compressed data has been proven to be a big success facing big data problems. These systems have demonstrated significant compression benefits and performance improvement for data analytics applications. However, current systems only focus on data queries, while a complete big data system must support both data query and data manipulation. We develop a new storage engine, called CompressDB, which can support data processing for databases without decompression. CompressDB has the following advantages. First, CompressDB utilizes context-free grammar to compress data, and supports both data query and data manipulation. Second, for adaptability, we integrate CompressDB to file systems so that a wide range of databases can directly use CompressDB without any change. Third, we enable operation pushdown to storage so that we can perform data query and manipulation in storage systems without bringing large data to memory for high efficiency. We validate the efficacy of CompressDB supporting various kinds of database systems, including SQLite, LevelDB, MongoDB, and ClickHouse. We evaluate our method using six real-world datasets with various lengths, structures, and content in both single node and cluster environments. Experiments show that CompressDB achieves 40% throughput improvement and 44% latency reduction, along with 1.81 compression ratio on average.
Content-Aware Partial Compression for Textual Big Data Analysis in Hadoop

Dapeng Dong,John Herbert

DOI: https://doi.org/10.1109/tbdata.2017.2721431

2018-12-01

IEEE Transactions on Big Data

Abstract:A substantial amount of information in companies and on the Internet is present in the form of text. The value of this semi-structured and unstructured data has been widely acknowledged, with consequent scientific and commercial exploitation. The ever-increasing data production, however, pushes data analytic platforms to their limit. Compression as an effective means to reduce data size has been employed by many emerging data analytic platforms, whom the main purpose of data compression is to save storage space and reduce data transmission cost over the network. Since general purpose compression methods endeavour to achieve higher compression ratios by leveraging data transformation techniques and contextual data, this context-dependency forces the access to the compressed data to be sequential. Processing such compressed data in parallel, such as desirable in a distributed environment, is extremely challenging. This work proposes techniques for more efficient textual big data analysis with an emphasis on content-aware compression schemes suitable for the Hadoop analytic platform. The compression schemes have been evaluated for a number of standard MapReduce analysis tasks using a collection of public and private real-world datasets. In comparison with existing solutions, they have shown substantial improvement in performance and significant reduction in system resource requirements.

computer science, information systems, theory & methods
Compressed Data Direct Computing for Databases

Weitao Wan,Feng Zhang,Chenyang Zhang,Mingde Zhang,Jidong Zhai,Yunpeng Chai,Huanchen Zhang,Wei Lu,Yuxing Chen,Haixiang Li,Anqun Pan,Xiaoyong Du

DOI: https://doi.org/10.1109/tkde.2023.3316274

IF: 9.235

2023-01-01

IEEE Transactions on Knowledge and Data Engineering

Abstract:Directly performing operations on compressed data has been proven to be a big success facing big data problems in modern data management systems. These systems have demonstrated significant compression benefits and performance improvement for data analytics applications. However, current systems only focus on data queries, while a complete big data system must support both data query and data manipulation. To solve this problem, we develop CompressDB, which is a new storage engine that can support data processing for databases without decompression. CompressDB has the following advantages. First, CompressDB utilizes context-free grammar to compress data, and supports both data query and data manipulation. Second, for adaptability, we integrate CompressDB to file systems so that a wide range of databases can directly use CompressDB without any change. Third, we enable operation pushdown to storage so that we can perform data query and manipulation in storage systems without bringing large data to memory for high efficiency. We validate the efficacy of CompressDB supporting various kinds of database systems, including SQLite, MySQL, LevelDB, MongoDB, ClickHouse, and Neo4j. We evaluate our method using seven real-world datasets with various lengths, structures, and content in both single node and cluster environments. Experiments show that CompressDB achieves 40% throughput improvement and 44% latency reduction, along with 1.75 compression ratio on average.

computer science, information systems, artificial intelligence,engineering, electrical & electronic
Enabling Efficient Random Access to Hierarchically-Compressed Data

Feng Zhang,Jidong Zhai,Xipeng Shen,Onur Mutlu,Xiaoyong Du

DOI: https://doi.org/10.1109/icde48307.2020.00097

2020-01-01

Abstract:Recent studies have shown the promise of direct data processing on hierarchically-compressed text documents. By removing the need for decompressing data, the direct data processing technique brings large savings in both time and space. However, its benefits have been limited to data traversal operations; for random accesses, direct data processing is several times slower than the state-of-the-art baselines. This paper presents a set of techniques that successfully eliminate the limitation, and for the first time, establishes the feasibility of effectively handling both data traversal operations and random data accesses on hierarchically-compressed data. The work yields a new library, which achieves 3.1 × speedup over the state-of-the-art on random data accesses to compressed data, while preserving the capability of supporting traversal operations efficiently and providing large (3.9 ×) space savings.
IC-Data: Improving Compressed Data Processing in Hadoop.

Adnan Haider,Xi Yang,Ning Liu,Xian-He Sun,Shuibing He

DOI: https://doi.org/10.1109/HiPC.2015.28

2015-01-01

Abstract:As dataset sizes for data analytic applications and scientific applications running on Hadoop increases, data compression has become essential to store this data within a reasonable storage cost. Although data is often stored compressed, currently Hadoop takes 49% longer to process compressed data compared to uncompressed data. Processing compressed data reduces the amount of task parallelism and creates uneven workload distribution both of which are fundamental issues the MapReduce parallel programming paradigm should alleviate. In this paper, we propose the design and implementation of a Network Overlapped Compression scheme, NOC, and Compression Aware Storage scheme, CAS. NOC reduces data load time and hides compression overhead by interleaving network I/O with compression. CAS increases parallelism by dynamically changing a file's block size based on compression ratio. Additionally, we develop a MapReduce Module which recognizes the characteristics of compressed data to improve resource allocation and load balance. Collectively, NOC, CAS, and the MapReduce Module decrease job execution time on average by 66% and data load time by 31%.
Data Compression for Analytics over Large-scale In-memory Column Databases

Chunbin Lin,Jianguo Wang,Yannis Papakonstantinou

DOI: https://doi.org/10.48550/arXiv.1606.09315

2016-07-06

Abstract:Data compression schemes have exhibited their importance in column databases by contributing to the high-performance OLAP (Online Analytical Processing) query processing. Existing works mainly concentrate on evaluating compression schemes for disk-resident databases as data is mostly stored on disks. With the continuously decreasing of the price/capacity ratio of main memory, it is the tendencies of the times to reside data in main memory. But the discussion of data compression on in-memory databases is very vague in the literature. In this work, we present an updated discussion about whether it is valuable to use data compression techniques in memory databases. If yes, how should memory databases apply data compression schemes to maximize performance?

Databases
Direct Processing of Document Images in Compressed Domain

Mohammed Javed,P. Nagabhushan,B.B. Chaudhuri

DOI: https://doi.org/10.13140/2.1.2523.9042

2014-10-14

Abstract:With the rapid increase in the volume of Big data of this digital era, fax documents, invoices, receipts, etc are traditionally subjected to compression for the efficiency of data storage and transfer. However, in order to process these documents, they need to undergo the stage of decompression which indents additional computing resources. This limitation induces the motivation to research on the possibility of directly processing of compressed images. In this research paper, we summarize the research work carried out to perform different operations straight from run-length compressed documents without going through the stage of decompression. The different operations demonstrated are feature extraction; text-line, word and character segmentation; document block segmentation; and font size detection, all carried out in the compressed version of the document. Feature extraction methods demonstrate how to extract the conventionally defined features such as projection profile, run-histogram and entropy, directly from the compressed document data. Document segmentation involves the extraction of compressed segments of text-lines, words and characters using the vertical and horizontal projection profile features. Further an attempt is made to segment randomly a block of interest from the compressed document and subsequently facilitate absolute and relative characterization of the segmented block which finds real time applications in automatic processing of Bank Cheques, Challans, etc, in compressed domain. Finally an application to detect font size at text line level is also investigated. All the proposed algorithms are validated experimentally with sufficient data set of compressed documents.

Computer Vision and Pattern Recognition
Regression Cubes with Lossless Compression and Aggregation

Yixin Chen,Guozhu Dong,Jiawei Han,Jian Pei,Benjamin W. Wah,Jianyong Wang

DOI: https://doi.org/10.1109/tkde.2006.196

IF: 9.235

2006-01-01

IEEE Transactions on Knowledge and Data Engineering

Abstract:As OLAP engines are widely used to support multidimensional data analysis, it is desirable to support in data cubes advanced statistical measures, such as regression and filtering, in addition to the traditional simple measures such as count and average. Such new measures will allow users to model, smooth, and predict the trends and patterns of data. Existing algorithms for simple distributive and algebraic measures are inadequate for efficient computation of statistical measures in a multidimensional space. In this paper, we propose a fundamentally new class of measures, compressible measures, in order to support efficient computation of the statistical models. For compressible measures, we compress each cell into an auxiliary matrix with a size independent of the number of tuples. We can then compute the statistical measures for any data cell from the compressed data of the lower-level cells without accessing the raw data. Time- and space-efficient lossless aggregation formulae are derived for regression and filtering measures. Our analytical and experimental studies show that the resulting system, regression cube, substantially reduces the memory usage and the overall response time for statistical analysis of multidimensional data.
Impression Store: Compressive Sensing-based Storage for Big Data Analytics.

Jiaxing Zhang,Ying Yan,Liang Jeff Chen,Minjie Wang,Thomas Moscibroda,Zheng Zhang

2014-01-01

Abstract:For many big data analytics workloads, approximate results suffice. This begs the question, whether and how the underlying system architecture can take advantage of such relaxations, thereby lifting constraints inherent in today's architectures. This position paper explores one of the possible directions. Impression Store is a distributed storage system with the abstraction of big data vectors. It aggregates updates internally and responds to the retrieval of top- K high-value entries. With proper extension, Impression Store supports various aggregations, top- K queries, outlier and major mode detection. While restricted in scope, such queries represent a substantial and important portion of many production workloads. In return, the system has unparalleled scalability; any node in the system can process any query, both reads and updates. The key technique we leverage is compressive sensing, a technique that substantially reduces the amount of active memory state, IO, and traffic volume needed to achieve such scalability.
Enabling Efficient NVM-Based Text Analytics Without Decompression

Xiaokun Fang,Feng Zhang,Junxiang Nong,Mingxing Zhang,Puyun Hu,Yunpeng Chai,Xiaoyong Du

DOI: https://doi.org/10.1109/icde60146.2024.00286

2024-01-01

Abstract:Text analytics directly on compression (TADOC) is a promising technology designed for handling big data analytics. However, a substantial amount of DRAM is required for high performance, which limits its usage in many important scenarios where the capacity of DRAM is limited, such as memory-constrained systems. Non-volatile memory (NVM) is a novel storage technology that combines the advantage of reading per-formance and byte addressability of DRAM with the durability of traditional storage devices like SSD and HDD. Unfortunately, no research demonstrates how to use NVM to reduce DRAM utilization in compressed data analytics. In this paper, we propose N-TADOC, which substitutes DRAM with NVM while maintaining TADOC's analytics performance and space savings. Utilizing an NVM block device to reduce DRAM utilization presents two challenges, including poor data locality in traversing datasets and auxiliary data structure reconstruction on NVM. We develop novel designs to solve these challenges, including a pruning method with NVM pool management, bottom-up upper bound estimation, correspondent data structures, and persistence strategy at different levels of cost. Experimental results show that on four real-world datasets, N-TADOC achieves 2.04× performance speedup compared to the processing directly on the uncompressed data and 70.7% DRAM space saving compared to the original TADOC.
Data-Aware Adaptive Compression for Stream Processing

Yu Zhang,Feng Zhang,Hourun Li,Shuhao Zhang,Xiaoguang Guo,Yuxing Chen,Anqun Pan,Xiaoyong Du

DOI: https://doi.org/10.1109/tkde.2024.3377710

IF: 9.235

2024-01-01

IEEE Transactions on Knowledge and Data Engineering

Abstract:Stream processing has been in widespread use, and one of the most common application scenarios is SQL query on streams. By 2021, the global deployment of IoT endpoints reached 12.3 billion, indicating a surge in data generation. However, the escalating demands for high throughput and low latency in stream processing systems have posed significant challenges due to the increasing data volume and evolving user requirements. We present a compression-based stream processing engine, called CompressStreamDB, which enables adaptive fine-grained stream processing directly on compressed streams, to significantly enhance the performance of existing stream processing solutions. CompressStreamDB utilizes nine diverse compression methods tailored for different stream data types and integrates a cost model to automatically select the most efficient compression schemes. CompressStreamDB provides high throughput with low latency in stream SQL processing by identifying and eliminating redundant data among streams. Our evaluation demonstrates that CompressStreamDB improves average performance by 3.84× and reduces average delay by 68.0% compared to the state-of-the-art stream processing solution for uncompressed streams, along with 68.7% space savings. Besides, our edge trials show an average throughput/price ratio of 9.95× and a throughput/power ratio of 7.32× compared to the cloud design.

computer science, information systems, artificial intelligence,engineering, electrical & electronic
Zwift: A Programming Framework for High Performance Text Analytics on Compressed Data.

Feng Zhang,Jidong Zhai,Xipeng Shen,Onur Mutlu,Wenguang Chen

DOI: https://doi.org/10.1145/3205289.3205325

2018-01-01

Abstract:Today's rapidly growing document volumes pose pressing challenges to modern document analytics frameworks, in both space usage and processing time. Recently, a promising method, called text analytics directly on compressed data (TADOC), was proposed for improving both the time and space efficiency of text analytics. The main idea of the technique is to enable direct document analytics on compressed data. This paper focuses on the programming challenges for developing efficient TADOC programs. It presents Zwift, the first programming framework for TADOC, which consists of a Domain Specific Language, a compiler and runtime, and a utility library. Experiments show that Zwift significantly improves programming productivity, while effectively unleashing the power of TADOC, producing code that reduces storage usage by 90.8% and execution time by 41.0% on six text analytics problems.
Logzip: Extracting Hidden Structures via Iterative Clustering for Log Compression

Jinyang Liu,Jieming Zhu,Shilin He,Pinjia He,Zibin Zheng,Michael R. Lyu

DOI: https://doi.org/10.48550/arXiv.1910.00409

2019-09-24

Abstract:System logs record detailed runtime information of software systems and are used as the main data source for many tasks around software engineering. As modern software systems are evolving into large scale and complex structures, logs have become one type of fast-growing big data in industry. In particular, such logs often need to be stored for a long time in practice (e.g., a year), in order to analyze recurrent problems or track security issues. However, archiving logs consumes a large amount of storage space and computing resources, which in turn incurs high operational cost. Data compression is essential to reduce the cost of log storage. Traditional compression tools (e.g., gzip) work well for general texts, but are not tailed for system logs. In this paper, we propose a novel and effective log compression method, namely logzip. Logzip is capable of extracting hidden structures from raw logs via fast iterative clustering and further generating coherent intermediate representations that allow for more effective compression. We evaluate logzip on five large log datasets of different system types, with a total of 63.6 GB in size. The results show that logzip can save about half of the storage space on average over traditional compression tools. Meanwhile, the design of logzip is highly parallel and only incurs negligible overhead. In addition, we share our industrial experience of applying logzip to Huawei's real products.

Databases,Software Engineering
CompAct: Compressing Retrieved Documents Actively for Question Answering

Chanwoong Yoon,Taewhoo Lee,Hyeon Hwang,Minbyul Jeong,Jaewoo Kang

2024-10-14

Abstract:Retrieval-augmented generation supports language models to strengthen their factual groundings by providing external contexts. However, language models often face challenges when given extensive information, diminishing their effectiveness in solving questions. Context compression tackles this issue by filtering out irrelevant information, but current methods still struggle in realistic scenarios where crucial information cannot be captured with a single-step approach. To overcome this limitation, we introduce CompAct, a novel framework that employs an active strategy to condense extensive documents without losing key information. Our experiments demonstrate that CompAct brings significant improvements in both performance and compression rate on multi-hop question-answering benchmarks. CompAct flexibly operates as a cost-efficient plug-in module with various off-the-shelf retrievers or readers, achieving exceptionally high compression rates (47x).

Computation and Language
Spatiotemporally adaptive compression for scientific dataset with feature preservation -- a case study on simulation data with extreme climate events analysis

Qian Gong,Chengzhu Zhang,Xin Liang,Viktor Reshniak,Jieyang Chen,Anand Rangarajan,Sanjay Ranka,Nicolas Vidal,Lipeng Wan,Paul Ullrich,Norbert Podhorszki,Robert Jacob,Scott Klasky

DOI: https://doi.org/10.1109/e-Science58273.2023.10254796

2024-01-07

Abstract:Scientific discoveries are increasingly constrained by limited storage space and I/O capacities. For time-series simulations and experiments, their data often need to be decimated over timesteps to accommodate storage and I/O limitations. In this paper, we propose a technique that addresses storage costs while improving post-analysis accuracy through spatiotemporal adaptive, error-controlled lossy compression. We investigate the trade-off between data precision and temporal output rates, revealing that reducing data precision and increasing timestep frequency lead to more accurate analysis outcomes. Additionally, we integrate spatiotemporal feature detection with data compression and demonstrate that performing adaptive error-bounded compression in higher dimensional space enables greater compression ratios, leveraging the error propagation theory of a transformation-based compressor.

Computer Vision and Pattern Recognition,Numerical Analysis
Czip: A Fast Lossless Compression Algorithm for Climate Data

Xiaomeng Huang,Yufang Ni,Dexun Chen,Songbin Liu,Haohuan Fu,Guangwen Yang

DOI: https://doi.org/10.1007/s10766-016-0403-z

2016-01-01

International Journal of Parallel Programming

Abstract:Climate data have been dramatically increasing in volume in recent years. This huge volume of climate data poses considerable challenges for data storage, archiving and sharing. In this paper, we propose a lossless compression algorithm for climate data, named czip. We efficiently eliminate data redundancy through several new methods, including adaptive prediction, eXclusive OR differencing, multiway compression and static regions. To utilize the multiple cores available on modern computers, czip is implemented in parallel. Experimental results show that czip can achieve outstanding compression ratios as well as deflating and inflating throughputs; czip can achieve 800 MB/s deflating throughputs and over 2600 MB/s inflating throughputs on a server with 16 cores.
Efficient Filtering and Fragment Dissemination of Compressed XML Documents

WU Ling,YANG Dong-Qing,TANG Shi-Wei,WANG Teng-Jiao,BAO Xiao-Yuan

2005-01-01

Computer Science

Abstract:The filtering and dissemination of XML documents is one of the most popular research areas of XML applica- tions. In order to improve the transfer efficiency of XML documents, compression should be used. But few of the filte- ring and dissemination algorithms of compressed XML documents have been proposed. We introduce a new method of filtering and dissemination of compressed XML document, which can efficiently filter and accurately disseminate the fragments to end users. Experiments show that our method is time and space efficient.
POCLib: A High-Performance Framework for Enabling Near Orthogonal Processing on Compression

Feng Zhang,Jidong Zhai,Xipeng Shen,Onur Mutlu,Xiaoyong Du

DOI: https://doi.org/10.1109/tpds.2021.3093234

IF: 5.3

2022-01-01

IEEE Transactions on Parallel and Distributed Systems

Abstract:Parallel technology boosts data processing in recent years, and parallel direct data processing on hierarchically compressed documents exhibits great promise. The high-performance direct data processing technique brings large savings in both time and space by removing the need for decompressing data. However, its benefits have been limited to data traversal operations; for random accesses, direct data processing is several times slower than the state-of-the-art baselines. This article proposes a novel concept, orthogonal processing on compression (orthogonal POC), which means that text analytics can be efficiently supported directly on compressed data, regardless of the type of the data processing – that is, the type of data processing is orthogonal to its capability of conducting POC. Previous proposals, such as TADOC, are not orthogonal POC. This article presents a set of techniques that successfully eliminate the limitation, and for the first time, establishes the near orthogonal POC feasibility of effectively handling both data traversal operations and random data accesses on hierarchically-compressed data. The work focuses on text data and yields a unified high-performance library, called POCLib. In a ten-node distributed Spark cluster on Amazon EC2, POCLib achieves 3.1× speedup over the state-of-the-art on random data accesses to compressed data, while preserving the capability of supporting traversal operations efficiently and providing large (3.9×) space savings.

Efficient document analytics on compressed data

Efficient Document Analytics on Compressed Data

TADOC: Text Analytics Directly on Compression

CompressDB: Enabling Efficient Compressed Data Direct Processing for Various Databases

Content-Aware Partial Compression for Textual Big Data Analysis in Hadoop

Compressed Data Direct Computing for Databases

Enabling Efficient Random Access to Hierarchically-Compressed Data

IC-Data: Improving Compressed Data Processing in Hadoop.

Data Compression for Analytics over Large-scale In-memory Column Databases

Direct Processing of Document Images in Compressed Domain

Regression Cubes with Lossless Compression and Aggregation

Impression Store: Compressive Sensing-based Storage for Big Data Analytics.

Enabling Efficient NVM-Based Text Analytics Without Decompression

Data-Aware Adaptive Compression for Stream Processing

Zwift: A Programming Framework for High Performance Text Analytics on Compressed Data.

Logzip: Extracting Hidden Structures via Iterative Clustering for Log Compression

CompAct: Compressing Retrieved Documents Actively for Question Answering

Spatiotemporally adaptive compression for scientific dataset with feature preservation -- a case study on simulation data with extreme climate events analysis

Czip: A Fast Lossless Compression Algorithm for Climate Data

Efficient Filtering and Fragment Dissemination of Compressed XML Documents

POCLib: A High-Performance Framework for Enabling Near Orthogonal Processing on Compression