Abstract:String kernels are typically used to compare genome-scale sequences whose length makes alignment impractical, yet their computation is based on data structures that are either space-inefficient, or incur large slowdowns. We show that a number of exact kernels on pairs of strings of total length n, like the k-mer kernel, the substrings kernels, a number of length-weighted kernels, the minimal absent words kernel, and kernels with Markovian corrections, can all be computed in O(nd) time and in o(n) bits of space in addition to the input, using just a rangeDistinct\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\mathtt {rangeDistinct}$$\end{document} data structure on the Burrows–Wheeler transform of the input strings that takes O(d) time per element in its output. The same bounds hold for a number of measures of compositional complexity based on multiple values of k, like the k-mer profile and the k-th order empirical entropy, and for calibrating the value of k using the data. All such algorithms become O(n) using a suitable implementation of the rangeDistinct\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\mathtt {rangeDistinct}$$\end{document} data structure, and by concatenating them to a suitable BWT construction algorithm, we can compute all the mentioned kernels and complexity measures, directly from the input strings, in O(n) time and in O(nlogσ)\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$O(n\log {\sigma })$$\end{document} bits of space in addition to the input, where σ\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\sigma $$\end{document} is the size of the alphabet. Using similar data structures, we also show how to build a compact representation of the variable-length Markov chain of a string T of length n, that takes just 3nlogσ+o(nlogσ)\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$3n\log {\sigma }+o(n\log {\sigma })$$\end{document} bits of space, and that can be learnt in randomized O(n) time using O(nlogσ)\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$O(n\log {\sigma })$$\end{document} bits of space in addition to the input. Such model can then be used to assign a probability to a query string S of length m in O(m) time and in 2m+o(m)\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$2m+o(m)$$\end{document} bits of additional space, thus providing an alternative, compositional measure of the similarity between S and T that does not require alignment.

The Bulk and The Tail of Minimal Absent Words in Genome Sequences

Computing Minimal Absent Words and Extended Bispecial Factors with CDAWG Space

Quasireplicas and universal lengths of microbial genomes

Long-Tail Feature of DNA Words Over- and Under-Representation in Coding Sequences

Range-Limited Heaps' Law for Functional DNA Words in the Human Genome

Mining Unique-m Substrings from Genomes

On Missing Mass Variance

The mysterious orphans of Mycoplasmataceae

Combinatorics of minimal absent words for a sliding window

Reconsidering the significance of genomic word frequency

Microbial genome as a fluctuating system: Distribution and correlation of coding sequence lengths

Inference of Markovian Properties of Molecular Sequences from NGS Data and Applications to Comparative Genomics

Minimal model for genome evolution and growth

Zimin patterns in genomes

Universal power law behaviors in genomic sequences and evolutionary models

Maximal Sequence Length of Exact Match Between Members from a Gene Family During Early Evolution.

A Framework for Space-Efficient String Kernels

Statistical estimation of sparsity and efficiency for molecular codes

Comparative statistical analysis of bacteria genomes in "word" context

Short Read Alignment Based On Maximal Approximate Match Seeds

Codon Usage Heterogeneity in the Multipartite Prokaryote Genome: Selection-Based Coding Bias Associated with Gene Location, Expression Level, and Ancestry