Abstract:MOTIVATION: The Burrows-Wheeler transform (BWT) is the foundation of many algorithms for compression and indexing of text data, but the cost of computing the BWT of very large string collections has prevented these techniques from being widely applied to the large sets of sequences often encountered as the outcome of DNA sequencing experiments. In previous work, we presented a novel algorithm that allows the BWT of human genome scale data to be computed on very moderate hardware, thus enabling us to investigate the BWT as a tool for the compression of such datasets.RESULTS: We first used simulated reads to explore the relationship between the level of compression and the error rate, the length of the reads and the level of sampling of the underlying genome and compare choices of second-stage compression algorithm. We demonstrate that compression may be greatly improved by a particular reordering of the sequences in the collection and give a novel 'implicit sorting' strategy that enables these benefits to be realized without the overhead of sorting the reads. With these techniques, a 45× coverage of real human genome sequence data compresses losslessly to under 0.5 bits per base, allowing the 135.3 Gb of sequence to fit into only 8.2 GB of space (trimming a small proportion of low-quality bases from the reads improves the compression still further). This is >4 times smaller than the size achieved by a standard BWT-based compressor (bzip2) on the untrimmed reads, but an important further advantage of our approach is that it facilitates the building of compressed full text indexes such as the FM-index on large-scale DNA sequence collections.AVAILABILITY: Code to construct the BWT and SAP-array on large genomic datasets is part of the BEETL library, available as a github repository at https://github.com/BEETL/BEETL.

Secure Compression and Pattern Matching Based on Burrows-Wheeler Transform

A Privacy-Preserving Multi-Pattern Matching Scheme for Searching Strings in Cloud Database

Privacy-Preserving Multi-pattern Matching.

Secure Hashing-Based Verifiable Pattern Matching

Large-scale compression of genomic sequence databases with the Burrows-Wheeler transform

Secure and Privacy-Preserving Pattern Matching in Outsourced Computing

Secure and Practical Wildcard Searchable Encryption System Based on Inner Product

A New Burrows Wheeler Transform Markov Distance

A Privacy-Preserving Image Retrieval Scheme Using Secure Local Binary Pattern in Cloud Computing

Privacy protection based on binary fingerprint compression

Secure pattern matching using somewhat homomorphic encryption

Efficient and Private Edit Distance Based Range Query over Encrypted String Sequences

A Novel Hardware Architecture to Accelerate Burrows-Wheeler Transform

Rphx: Result Pattern Hiding Conjunctive Query Over Private Compressed Index Using Intel SGX

When Geo-Text Meets Security: Privacy-Preserving Boolean Spatial Keyword Queries

Ppopm: More Efficient Privacy Preserving Outsourced Pattern Matching

Performance Enhanced Secure Spatial Keyword Similarity Query With Arbitrary Spatial Ranges

Search Me in the Dark: Privacy-preserving Boolean Range Query over Encrypted Spatial Data

Debwt: Parallel Construction of Burrows-Wheeler Transform for Large Collection of Genomes with De Bruijn-branch Encoding.

Toward Complex Search for Encrypted Cloud Data via Blind Index Storage

Practical wildcard searchable encryption with tree‐based index