Abstract:I present efficient data mining algorithms for knowledge discovery on two types of emerging large-scale sequence-based scientific datasets: (1) static sequence data generated from SNP diversity arrays for genomic studies, and (2) dynamic sequence data collected in streaming and sensor network systems for environmental studies. The massive, noisy nature of the SNP arrays and the distributive, online nature of sensor network data pose challenging issues for knowledge discovery such as scalability, robustness, and efficiency. Despite the different characteristics of the SNP arrays and streaming sensor data, when viewed as sequences of ordered observations, both can be efficiently mined using algorithms based on block-wise decomposition methods. I present models and mining algorithms for inferring the genetic variation structure in genome-wide Single-Nucleotide Polymorphism (SNP) arrays. Genome-wide SNP arrays provide a comprehensive view of genome variation and serve as powerful resources for genetic and biomedical studies. Understanding the patterns of genetic variation in a population of individuals plays an important role in solving many genetics problems such as genealogy reconstruction and gene association studies. In this thesis, I propose data mining models and algorithms to efficiently infer genetic variation structure from the massive SNP panels of recombinant sequences resulting from meiotic recombination. I introduced the Minimum Segmentation Problem (MSP) to infer the segmentation structure of a single recombinant strain, as well as the Minimum Mosaic Problem (MMP) to infer the mosaic structure on a panel of recombinant strains. Both MSP and MMP estimate the ancestral polymorphism patterns exhibited in recombinant strains which provides important inputs for the subsequent association analysis. Efficient dynamic programming and graph algorithms based on block-wise decomposition are proposed which can solve MSP and MMP on genome-wide large-scale panels. I present efficient algorithms for mining massive streaming and sensor network data for observational sciences such as ecology and environmental studies. I proposed efficient algoirithms using block-wise synopsis construction to capture the data distribution online for the dynamic sequence data collected in the sensor network and streaming systems including clustering analysis and order-statistics computation, which is critical for real-time monitoring, anomaly detection, and other domain specific analysis.

Kssd: Sequence Dimensionality Reduction by K-Mer Substring Space Sampling Enables Real-Time Large-Scale Datasets Analysis

Fractional Hitting Sets for Efficient and Lightweight Genomic Data Sketching

MetaKSSD: Boosting the Scalability of Reference Taxonomic Marker Database and the Performance of Metagenomic Profiling Using Sketch Operations

Analyzing big datasets of genomic sequences: fast and scalable collection of k-mer statistics

k-nonical space: sketching with reverse complements

SSCC: A Novel Computational Framework for Rapid and Accurate Clustering Large-scale Single Cell RNA-seq Data.

Interpolative Multidimensional Scaling Techniques for the Identification of Clusters in Very Large Sequence Sets

KCOSS: an ultra-fast k-mer counter for assembled genome analysis

Sequence-based Multiscale Model (SeqMM) for High-throughput chromosome conformation capture (Hi-C) data analysis

KSI：a DNA sequence matching library for terabyte scale bio-data

CRAFT: Compact genome Representation toward large-scale Alignment-Free daTabase

CHTKC: a robust and efficient k-mer counting algorithm based on a lock-free chaining hash table

These are not the k-mers you are looking for: efficient online k-mer counting using a probabilistic data structure

Estimation of genomic characteristics by analyzing k-mer frequency in de novo genome projects

Kmer2vec: A Novel Method for Comparing DNA Sequences by Word2vec Embedding

Memory Efficient De Bruijn Graph Construction

Mining emerging massive scientific sequence data using block-wise decomposition methods

SKSV: Ultrafast Structural Variation Detection from Circular Consensus Sequencing Reads.

KmerGO: A Tool to Identify Group-Specific Sequences With k-mers

Dace: A Scalable Dp-Means Algorithm for Clustering Extremely Large Sequence Data

SSCC: a novel computational framework for rapid and accurate clustering large single cell RNA-seq data