'Google for DNA' indexes 10% of world's known sequence data

Catherine Offord

DOI: https://doi.org/10.1126/science.adq8853

IF: 56.9

2024-06-07

Science

Abstract:Achievement demonstrates feasibility of making all of life's code easily searchable, researchers say

multidisciplinary sciences

What problem does this paper attempt to address?

Indexing All Life's Known Biological Sequences

Mikhail Karasikov,Harun Mustafa,Daniel Danciu,Marc Zimmermann,Christopher Barber,Gunnar Ratsch,Andre Kahles

DOI: https://doi.org/10.1101/2020.10.01.322164

2024-06-07

Abstract:The amount of biological sequencing data available in public repositories is growing exponentially, forming an invaluable biomedical research resource. Yet, making it full-text searchable and easily accessible to researchers in life and data science is an unsolved problem. In this work, we take advantage of recently developed, very efficient data structures and algorithms for representing sequence sets. We make Petabases of DNA sequences across all clades of life, including viruses, bacteria, fungi, plants, animals, and humans, fully searchable. Our indexes are freely available to the research community. This highly compressed representation of the input sequences (up to 5800 fold) fits on a single consumer hard drive (≈100 USD), making this valuable resource cost-effective to use and easily transportable. We present the underlying methodological framework, called MetaGraph, that allows us to scalably index very large sets of DNA or protein sequences using annotated De Bruijn graphs. We demonstrate the feasibility of indexing the full extent of existing sequencing data and present new approaches for efficient and cost-effective full-text search at an on-demand cost of $0.10 per queried Mpb. We explore several practical use cases to mine existing archives for interesting associations and demonstrate the utility of our indexes for integrative analyses.

Bioinformatics
Molecular-level similarity search brings computing to DNA data storage

Callista Bee,Yuan-Jyue Chen,Melissa Queen,David Ward,Xiaomeng Liu,Lee Organick,Georg Seelig,Karin Strauss,Luis Ceze

DOI: https://doi.org/10.1038/s41467-021-24991-z

IF: 16.6

2021-08-06

Nature Communications

Abstract:Abstract As global demand for digital storage capacity grows, storage technologies based on synthetic DNA have emerged as a dense and durable alternative to traditional media. Existing approaches leverage robust error correcting codes and precise molecular mechanisms to reliably retrieve specific files from large databases. Typically, files are retrieved using a pre-specified key, analogous to a filename. However, these approaches lack the ability to perform more complex computations over the stored data, such as similarity search: e.g., finding images that look similar to an image of interest without prior knowledge of their file names. Here we demonstrate a technique for executing similarity search over a DNA-based database of 1.6 million images. Queries are implemented as hybridization probes, and a key step in our approach was to learn an image-to-sequence encoding ensuring that queries preferentially bind to targets representing visually similar images. Experimental results show that our molecular implementation performs comparably to state-of-the-art in silico algorithms for similarity search.

multidisciplinary sciences
Scaling up DNA data storage and random access retrieval

Lee Organick,Siena Dumas Ang,Yuan-Jyue Chen,Randolph Lopez,Sergey Yekhanin,Konstantin Makarychev,Miklos Z. Racz,Govinda Kamath,Parikshit Gopalan,Bichlien Nguyen,Christopher Takahashi,Sharon Newman,Hsing-Yeh Parker,Cyrus Rashtchian,Kendall Stewart,Gagan Gupta,Robert Carlson,John Mulligan,Douglas Carmean,Georg Seelig,Luis Ceze,Karin Strauss

DOI: https://doi.org/10.1101/114553

2017-03-07

Abstract:Current storage technologies can no longer keep pace with exponentially growing amounts of data. 1 Synthetic DNA offers an attractive alternative due to its potential information density of ~ 10 18 B/mm 3 , 10 7 times denser than magnetic tape, and potential durability of thousands of years. 2 Recent advances in DNA data storage have highlighted technical challenges, in particular, coding and random access, but have stored only modest amounts of data in synthetic DNA. 3,4,5 This paper demonstrates an end-to-end approach toward the viability of DNA data storage with large-scale random access. We encoded and stored 35 distinct files, totaling 200MB of data, in more than 13 million DNA oligonucleotides (about 2 billion nucleotides in total) and fully recovered the data with no bit errors, representing an advance of almost an order of magnitude compared to prior work. 6 Our data curation focused on technologically advanced data types and historical relevance, including the Universal Declaration of Human Rights in over 100 languages, 7 a high-definition music video of the band OK Go, 8 and a CropTrust database of the seeds stored in the Svalbard Global Seed Vault. 9 We developed a random access methodology based on selective amplification, for which we designed and validated a large library of primers, and successfully retrieved arbitrarily chosen items from a subset of our pool containing 10.3 million DNA sequences. Moreover, we developed a novel coding scheme that dramatically reduces the physical redundancy (sequencing read coverage) required for error-free decoding to a median of 5x, while maintaining levels of logical redundancy comparable to the best prior codes. We further stress-tested our coding approach by successfully decoding a file using the more error-prone nanopore-based sequencing. We provide a detailed analysis of errors in the process of writing, storing, and reading data from synthetic DNA at a large scale, which helps characterize DNA as a storage medium and justify our coding approach. Thus, we have demonstrated a significant improvement in data volume, random access, and encoding/decoding schemes that contribute to a whole-system vision for DNA data storage.
How to build a DNA search engine like Google?

Wang Liang,Fang Bo

DOI: https://doi.org/10.48550/arXiv.1006.4114

2011-10-10

Abstract:This paper proposed a new method to build the large scale DNA sequences search system based on web search engine technology. We give a very brief introduction for the methods used in search engine first. Then how to build a DNA search system like Google is illustrated in detail. Since there is no local alignment process, this system is able to provide the ms level search services for billions of DNA sequences in a typical server.

Genomics,Emerging Technologies,Information Retrieval
From barcodes to genomes: extending the concept of DNA barcoding

Eric Coissac,Peter M. Hollingsworth,Sébastien Lavergne,Pierre Taberlet

DOI: https://doi.org/10.1111/mec.13549

IF: 6.622

2016-03-14

Molecular Ecology

Abstract:DNA barcoding has had a major impact on biodiversity science. The elegant simplicity of establishing massive scale databases for a few barcode loci is continuing to change our understanding of species diversity patterns, and continues to enhance human abilities to distinguish among species. Capitalizing on the developments of next generation sequencing technologies and decreasing costs of genome sequencing, there is now the opportunity for the DNA barcoding concept to be extended to new kinds of genomic data. We illustrate the benefits and capacity to do this, and also note the constraints and barriers to overcome before it is truly scalable. We advocate a twin track approach: (i) continuation and acceleration of global efforts to build the DNA barcode reference library of life on earth using standard DNA barcodes and (ii) active development and application of extended DNA barcodes using genome skimming to augment the standard barcoding approach.

biochemistry & molecular biology,ecology,evolutionary biology
Beyond DNA: ML-Empowered Nanopore Base-Calling of 12-Letter Genetic Alphabets

Sneha Mittal,Milan Jena,Biswarup Pathak

DOI: https://doi.org/10.26434/chemrxiv-2024-xv61j

2024-12-02

Abstract:The standard 4-letter genetic code (A, T, G, C) is the blueprint of life on earth. However, beyond this foundational framework lies a realm of artificial genetics, which has now expanded the genetic code up to 12-letter (A, T, G, C, B, S, P, Z, X, K, J, V). Strikingly, at a time when detection methods for genomics and transcriptomics have progressed to their “fourth generation” with successful commercialization, the field of artificial genetics is still in its nascent stage, the “zeroth generation”. Herein, in the framework of DFT and machine learning (ML), we report a next-generation solid-state nanopore sequencing to both assess and decode the DNA code with expanded alphabets. For assessing, we leverage the ML regression tools, which predict the transmission signatures of each natural and xenonucleobase with low mean squared error as validated through DFT. Parameterizing SMILES (simplified molecular input line entry system) strings of expanded alphabets, including isomers, allows the structural, molecular, and bonding configuration of nucleobases to be meticulously incorporated during predictions. Further, custom ML classification tools are developed, and each standard, isoG/isoC, hachimoji, and supernumerary code is decoded with SHAP (Shapley Additive exPlanations) explainability. By introducing ML accelerated nanopore sequencing of supernumerary DNA, we pave the way for rapid analysis of expanded alphabets, offering insights into life’s possibilities across the cosmos.

Chemistry
CRISPR-powered quantitative keyword search engine in DNA data storage

Jiongyu Zhang,Chengyu Hou,Changchun Liu

DOI: https://doi.org/10.1038/s41467-024-46767-x

IF: 16.6

2024-03-15

Nature Communications

Abstract:Abstract Despite the growing interest of archiving information in synthetic DNA to confront data explosion, quantitatively querying the data stored in DNA is still a challenge. Herein, we present Search Enabled by Enzymatic Keyword Recognition (SEEKER), which utilizes CRISPR-Cas12a to rapidly generate visible fluorescence when a DNA target corresponding to the keyword of interest is present. SEEKER achieves quantitative text searching since the growth rate of fluorescence intensity is proportional to keyword frequency. Compatible with SEEKER, we develop non-collision grouping coding, which reduces the size of dictionary and enables lossless compression without disrupting the original order of texts. Using four queries, we correctly identify keywords in 40 files with a background of ~8000 irrelevant terms. Parallel searching with SEEKER can be performed on a 3D-printed microfluidic chip. Overall, SEEKER provides a quantitative approach to conducting parallel searching over the complete content stored in DNA with simple implementation and rapid result generation.

multidisciplinary sciences
Content-Based Similarity Search in Large-Scale DNA Data Storage Systems

Callista Bee,Yuan-Jyue Chen,David Ward,Xiaomeng Liu,Georg Seelig,Karin Strauss,Luis Ceze

DOI: https://doi.org/10.1101/2020.05.25.115477

2020-05-27

Abstract:Abstract Synthetic DNA has the potential to store the world’s continuously growing amount of data in an extremely dense and durable medium. Current proposals for DNA-based digital storage systems include the ability to retrieve individual files by their unique identifier, but not by their content. Here, we demonstrate content-based retrieval from a DNA database by learning a mapping from images to DNA sequences such that an encoded query image will retrieve visually similar images from the database via DNA hybridization. We encoded and synthesized a database of 1.6 million images and queried it with a variety of images, showing that each query retrieves a sample of the database containing visually similar images are retrieved at a rate much greater than chance. We compare our results with several algorithms for similarity search in electronic systems, and demonstrate that our molecular approach is competitive with state-of-the-art electronics. One Sentence Summary Learned encodings enable content-based image similarity search from a database of 1.6 million images encoded in synthetic DNA.
LISA: Towards Learned DNA Sequence Search

Darryl Ho,Jialin Ding,Sanchit Misra,Nesime Tatbul,Vikram Nathan,Vasimuddin Md,Tim Kraska

DOI: https://doi.org/10.48550/arXiv.1910.04728

2019-10-11

Abstract:Next-generation sequencing (NGS) technologies have enabled affordable sequencing of billions of short DNA fragments at high throughput, paving the way for population-scale genomics. Genomics data analytics at this scale requires overcoming performance bottlenecks, such as searching for short DNA sequences over long reference sequences. In this paper, we introduce LISA (Learned Indexes for Sequence Analysis), a novel learning-based approach to DNA sequence search. As a first proof of concept, we focus on accelerating one of the most essential flavors of the problem, called exact search. LISA builds on and extends FM-index, which is the state-of-the-art technique widely deployed in genomics tool-chains. Initial experiments with human genome datasets indicate that LISA achieves up to a factor of 4X performance speedup against its traditional counterpart.

Databases,Data Structures and Algorithms,Machine Learning
The status of the human gene catalogue

Paulo Amaral,Silvia Carbonell-Sala,Francisco M. De La Vega,Tiago Faial,Adam Frankish,Thomas Gingeras,Roderic Guigo,Jennifer L Harrow,Artemis G. Hatzigeorgiou,Rory Johnson,Terence D. Murphy,Mihaela Pertea,Kim D. Pruitt,Shashikant Pujar,Hazuki Takahashi,Igor Ulitsky,Ales Varabyou,Christine A. Wells,Mark Yandell,Piero Carninci,Steven L. Salzberg

2023-03-24

Abstract:Scientists have been trying to identify all of the genes in the human genome since the initial draft of the genome was published in 2001. Over the intervening years, much progress has been made in identifying protein-coding genes, and the estimated number has shrunk to fewer than 20,000, although the number of distinct protein-coding isoforms has expanded dramatically. The invention of high-throughput RNA sequencing and other technological breakthroughs have led to an explosion in the number of reported non-coding RNA genes, although most of them do not yet have any known function. A combination of recent advances offers a path forward to identifying these functions and towards eventually completing the human gene catalogue. However, much work remains to be done before we have a universal annotation standard that includes all medically significant genes, maintains their relationships with different reference genomes, and describes clinically relevant genetic variants.

Genomics,Quantitative Methods
Explorer: efficient DNA coding by De Bruijn graph toward arbitrary local and global biochemical constraints

Chang Dou,Yijie Yang,Fei Zhu,BingZhi Li,Yuping Duan

DOI: https://doi.org/10.1093/bib/bbae363

2024-07-25

Abstract:With the exponential growth of digital data, there is a pressing need for innovative storage media and techniques. DNA molecules, due to their stability, storage capacity, and density, offer a promising solution for information storage. However, DNA storage also faces numerous challenges, such as complex biochemical constraints and encoding efficiency. This paper presents Explorer, a high-efficiency DNA coding algorithm based on the De Bruijn graph, which leverages its capability to characterize local sequences. Explorer enables coding under various biochemical constraints, such as homopolymers, GC content, and undesired motifs. This paper also introduces Codeformer, a fast decoding algorithm based on the transformer architecture, to further enhance decoding efficiency. Numerical experiments indicate that, compared with other advanced algorithms, Explorer not only achieves stable encoding and decoding under various biochemical constraints but also increases the encoding efficiency and bit rate by ¿10%. Additionally, Codeformer demonstrates the ability to efficiently decode large quantities of DNA sequences. Under different parameter settings, its decoding efficiency exceeds that of traditional algorithms by more than two-fold. When Codeformer is combined with Reed-Solomon code, its decoding accuracy exceeds 99%, making it a good choice for high-speed decoding applications. These advancements are expected to contribute to the development of DNA-based data storage systems and the broader exploration of DNA as a novel information storage medium.
Covering All Bases: The Next Inning in DNA Sequencing Efficiency

Hadas Abraham,Rayn Gabrys,Eitan Yaakobi

2024-01-31

Abstract:DNA emerges as a promising medium for the exponential growth of digital data due to its density and durability. This study extends recent research by addressing the \emph{coverage depth problem} in practical scenarios, exploring optimal error-correcting code pairings with DNA storage systems to minimize coverage depth. Conducted within random access settings, the study provides theoretical analyses and experimental simulations to examine the expectation and probability distribution of samples needed for files recovery. Structured into sections covering definitions, analyses, lower bounds, and comparative evaluations of coding schemes, the paper unveils insights into effective coding schemes for optimizing DNA storage systems.

Information Theory
Fundamental significance of DNA mass-sequencing factory for biological sciences in future

A Wada

DOI: https://doi.org/10.1016/0065-227x(94)90011-6

Abstract:We are now entering an era of the mass-analysis of genetic information, which will signal the beginning of the study of living organisms on the basis of their most detailed plan: the DNA base-sequence. The researcher's essential requirement in reading and deciphering the DNA base-sequence is the precision, speed, reliability, and low cost of such operation. I have long stressed that these requirements can only be satisfied by establishing a centralized organization, a DNA sequencing factory, where a large-scale and international operation of the sequencing can be carried out with the assistance of high technologies and under an administration which equitably represents the research interests of researchers worldwide. In this context, the basic concept and aims of the Japanese project which was begun in 1981 are described, detailing the advantages of such a factory with a mega-base daily output. Scientists and engineers are devising a high speed and automated DNA sequencing system by coordinating available DNA-processing technologies, particularly in the field of computer/robotic engineering. Typical of the achievements of this enterprise is the recent progress made by RIKEN in the development of a DNA mass-sequencing line which aims the production of raw base-sequence data of 0.1 mega-base per day (machine throughput). Finally, the elemental units in the current DNA sequencing procedures and the means of DNA processing are displayed on a family tree network to illustrate their functional interrelation in the current scheme of DNA analysis. It is hoped that this will provide a clue to finding new dimensions in DNA processing.
DNAscan: a fast, computationally and memory efficient bioinformatics pipeline for the analysis of DNA next-generation-sequencing data

A Iacoangeli,A Al Khleifat,W Sproviero,A Shatunov,AR Jones,R Dobson,SJ Newhouse,A Al-Chalabi

DOI: https://doi.org/10.1101/267195

2018-02-18

Abstract:Abstract The generation of DNA Next Generation Sequencing (NGS) data is a commonly applied approach for studying the genetic basis of biological processes, including diseases, and underpins the aspirations of precision medicine. However, there are significant challenges when dealing with NGS data. A huge number of bioinformatics tools exist and it is therefore challenging to design an analysis pipeline; NGS analysis is computationally intensive, requiring expensive infrastructure which can be problematic given that many medical and research centres do not have adequate high performance computing facilities and the use of cloud computing facilities is not always possible due to privacy and ownership issues. We have therefore developed a fast and efficient bioinformatics pipeline that allows for the analysis of DNA sequencing data, while requiring little computational effort and memory usage. We achieved this by exploiting state-of-the-art bioinformatics tools. DNAscan can analyse raw, 40x whole genome NGS data in 8 hours, using as little as 8 threads and 16 Gbs of RAM, while guaranteeing a high performance. DNAscan can look for SNVs, small indels, SVs, repeat expansions and viral genetic material (or any other organism). Its results are annotated using a customisable variety of databases including ClinVar, Exac and dbSNP, and a local deployment of the gene.iobio platform is available for an on-the-fly result visualisation.
The evolution of next-generation sequencing technologies

Olaitan Akintunde,Trichina Tucker,Valerie J. Carabetta

DOI: https://doi.org/10.48550/arXiv.2305.08724

2023-05-15

Abstract:The genetic information that dictates the structure and function of all life forms is encoded in the DNA. In 1953, Watson and Crick first presented the double helical structure of a DNA molecule. Their findings unearthed the desire to elucidate the exact composition and sequence of DNA molecules. Discoveries and the subsequent development and optimization of techniques that allowed for deciphering the DNA sequence has opened new doors in research, biotech, and healthcare. The application of high-throughput sequencing technologies in these industries has positively impacted and will continue to contribute to the betterment of humanity and the global economy. Improvements, such as the use of radioactive molecules for DNA sequencing to the use of florescent dyes and the implementation of polymerase chain reaction (PCR) for amplification, led to sequencing a few hundred base pairs in days, to automation, where sequencing of thousands of base pairs in hours became possible. Significant advances have been made, but there is still room for improvement. Here, we look at the history and the technology of the currently available next-generation sequencing platforms and the possible applications of such technologies to biomedical research and beyond.

Genomics
Accurate and General DNA Representations Emerge from Genome Foundation Models at Scale

Caleb N Ellington,Ning Sun,Nicholas Ho,Tianhua Tao,Sazan Mahbub,Dian Li,Yonghao Zhuang,Hongyi Wang,Le Song,Eric P. Xing

DOI: https://doi.org/10.1101/2024.12.01.625444

2024-12-05

Abstract:Language models applied to protein sequences have become a panacea, enabling therapeutics development, materials engineering, and core biology research. Despite the successes of protein language models, genome language models remain nascent. Recent studies suggest the bottleneck is data volume or modeling context size, since long-range interactions are widely acknowledged but sparsely annotated. However, it may be the case that even short DNA sequences are modeled poorly by existing approaches, and current models are unable to represent the wide array of functions encoded by DNA. To study this, we develop AIDO.DNA, a pretrained module for DNA representation in an AI-driven Digital Organism. AIDO.DNA is a seven billion parameter encoder-only transformer trained on 10.6 billion nucleotides from a dataset of 796 species. By scaling model size while maintaining a short context length of 4k nucleotides, AIDO.DNA shows substantial improvements across a breadth of supervised, generative, and zero-shot tasks relevant to functional genomics, synthetic biology, and drug development. Notably, AIDO.DNA outperforms prior encoder-only architectures without new data, suggesting that new scaling laws are needed to achieve compute-optimal DNA language models. Models and code are available through ModelGenerator in https://github.com/genbio-ai/AIDO and on Hugging Face.

Biology
Survey of Information Encoding Techniques for DNA

Thomas Heinis,Roman Sokolovskii,Jamie J. Alnasir

DOI: https://doi.org/10.1145/3626233

2023-10-03

Abstract:The yearly global production of data is growing exponentially, outpacing the capacity of existing storage media, such as tape and disk, and surpassing our ability to store it. DNA storage - the representation of arbitrary information as sequences of nucleotides - offers a promising storage medium. DNA is nature's information-storage molecule of choice and has a number of key properties: it is extremely dense, offering the theoretical possibility of storing 455 EB/g; it is durable, with a half-life of approximately 520 years that can be increased to thousands of years when DNA is chilled and stored dry; and it is amenable to automated synthesis and sequencing. Furthermore, biochemical processes that act on DNA potentially enable highly parallel data manipulation. Whilst biological information is encoded in DNA via a specific mapping from triplet sequences of nucleotides to amino acids, DNA storage is not limited to a single encoding scheme, and there are many possible ways to map data to chemical sequences of nucleotides for synthesis, storage, retrieval and data manipulation. However, there are several biological, error-tolerance and information-retrieval considerations that an encoding scheme needs to address to be viable. This comprehensive review focuses on comparing existing work done in encoding arbitrary data within DNA in terms of their encoding schemes, methods to address biological constraints and measures to provide error correction. We compare encoding approaches on the overall information density and coverage they achieve, as well as the data-retrieval method they use (i.e., sequential or random access). We also discuss the background and evolution of the encoding schemes.

Quantitative Methods,Databases,Data Structures and Algorithms,Information Theory
Deoxyribonucleic Acid as a Tool for Digital Information Storage: An Overview

Ashish C Patel,C G Joshi

DOI: https://doi.org/10.21887/ijvsbt.15.1.1

2019-07-25

THE INDIAN JOURNAL OF VETERINARY SCIENCES AND BIOTECHNOLOGY

Abstract:Current data storage technologies cannot keep pace longer with exponentially growing amounts of data through the extensive use of social networking photos and media, etc. The "digital world” with 4.4 zettabytes in 2013 has predicted it to reach 44 zettabytes by 2020. From the past 30 years, scientists and researchers have been trying to develop a robust way of storing data on a medium which is dense and ever-lasting and found DNA as the most promising storage medium. Unlike existing storage devices, DNA requires no maintenance, except the need to store at a cool and dark place. DNA has a small size with high density; just 1 gram of dry DNA can store about 455 exabytes of data. DNA stores the informations using four bases, viz., A, T, G, and C, while CDs, hard disks and other devices stores the information using 0’s and 1’s on the spiral tracks. In the DNA based storage, after binarization of digital file into the binary codes, encoding and decoding are important steps in DNA based storage system. Once the digital file is encoded, the next step is to synthesize arbitrary single-strand DNA sequences and that can be stored in the deep freeze until use.When there is a need for information to be recovered, it can be done using DNA sequencing. New generation sequencing (NGS) capable of producing sequences with very high throughput at a much lower cost about less than 0.1 USD for one MB of data than the first sequencing technologies. Post-sequencing processing includes alignment of all reads using multiple sequence alignment (MSA) algorithms to obtain different consensus sequences. The consensus sequence is decoded as the reversal of the encoding process. Most prior DNA data storage efforts sequenced and decoded the entire amount of stored digital information with no random access, but nowadays it has become possible to extract selective files (e.g., retrieving only required image from a collection) from a DNA pool using PCR-based random access. Various scientists successfully stored up to 110 zettabytes data in one gram of DNA. In the future, with an efficient encoding, error corrections, cheaper DNA synthesis,and sequencing, DNA based storage will become a practical solution for storage of exponentially growing digital data.
High-scale random access on DNA storage systems

Alex El-Shaikh,Marius Welzel,Dominik Heider,Bernhard Seeger

DOI: https://doi.org/10.1093/nargab/lqab126

2022-01-14

NAR Genomics and Bioinformatics

Abstract:ABSTRACT Due to the rapid cost decline of synthesizing and sequencing deoxyribonucleic acid (DNA), high information density, and its durability of up to centuries, utilizing DNA as an information storage medium has received the attention of many scientists. State-of-the-art DNA storage systems exploit the high capacity of DNA and enable random access (predominantly random reads) by primers, which serve as unique identifiers for directly accessing data. However, primers come with a significant limitation regarding the maximum available number per DNA library. The number of different primers within a library is typically very small (e.g. ≈10). We propose a method to overcome this deficiency and present a general-purpose technique for addressing and directly accessing thousands to potentially millions of different data objects within the same DNA pool. Our approach utilizes a fountain code, sophisticated probe design, and microarray technologies. A key component is locality-sensitive hashing, making checks for dissimilarity among such a large number of probes and data objects feasible.
SPIDER-WEB generates coding algorithms with superior error tolerance and real-time information retrieval capacity

Haoling Zhang,Zhaojun Lan,Wenwei Zhang,Xun Xu,Zhi Ping,Yiwei Zhang,Yue Shen

DOI: https://doi.org/10.48550/arXiv.2204.02855

2023-03-30

Abstract:DNA has been considered a promising medium for storing digital information. As an essential step in the DNA-based data storage workflow, coding algorithms are responsible to implement functions including bit-to-base transcoding, error correction, etc. In previous studies, these functions are normally realized by introducing multiple algorithms. Here, we report a graph-based architecture, named SPIDER-WEB, providing an all-in-one coding solution by generating customized algorithms automatically. SPIDERWEB is able to correct a maximum of 4% edit errors in the DNA sequences including substitution and insertion/deletion (indel), with only 5.5% redundant symbols. Since no DNA sequence pretreatment is required for the correcting and decoding processes, SPIDER-WEB offers the function of real-time information retrieval, which is 305.08 times faster than the speed of single-molecule sequencing techniques. Our retrieval process can improve 2 orders of magnitude faster compared to the conventional one under megabyte-level data and can be scalable to fit exabyte-level data. Therefore, SPIDER-WEB holds the potential to improve the practicability in large-scale data storage applications.

Emerging Technologies,Information Theory,Combinatorics,Genomics

'Google for DNA' indexes 10% of world's known sequence data

Indexing All Life's Known Biological Sequences

Molecular-level similarity search brings computing to DNA data storage

Scaling up DNA data storage and random access retrieval

How to build a DNA search engine like Google?

From barcodes to genomes: extending the concept of DNA barcoding

Beyond DNA: ML-Empowered Nanopore Base-Calling of 12-Letter Genetic Alphabets

CRISPR-powered quantitative keyword search engine in DNA data storage

Content-Based Similarity Search in Large-Scale DNA Data Storage Systems

LISA: Towards Learned DNA Sequence Search

The status of the human gene catalogue

Explorer: efficient DNA coding by De Bruijn graph toward arbitrary local and global biochemical constraints

Covering All Bases: The Next Inning in DNA Sequencing Efficiency

Fundamental significance of DNA mass-sequencing factory for biological sciences in future

DNAscan: a fast, computationally and memory efficient bioinformatics pipeline for the analysis of DNA next-generation-sequencing data

The evolution of next-generation sequencing technologies

Accurate and General DNA Representations Emerge from Genome Foundation Models at Scale

Survey of Information Encoding Techniques for DNA

Deoxyribonucleic Acid as a Tool for Digital Information Storage: An Overview

High-scale random access on DNA storage systems

SPIDER-WEB generates coding algorithms with superior error tolerance and real-time information retrieval capacity