Abstract:String kernels are typically used to compare genome-scale sequences whose length makes alignment impractical, yet their computation is based on data structures that are either space-inefficient, or incur large slowdowns. We show that a number of exact kernels on pairs of strings of total length n, like the k-mer kernel, the substrings kernels, a number of length-weighted kernels, the minimal absent words kernel, and kernels with Markovian corrections, can all be computed in O(nd) time and in o(n) bits of space in addition to the input, using just a rangeDistinct\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\mathtt {rangeDistinct}$$\end{document} data structure on the Burrows–Wheeler transform of the input strings that takes O(d) time per element in its output. The same bounds hold for a number of measures of compositional complexity based on multiple values of k, like the k-mer profile and the k-th order empirical entropy, and for calibrating the value of k using the data. All such algorithms become O(n) using a suitable implementation of the rangeDistinct\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\mathtt {rangeDistinct}$$\end{document} data structure, and by concatenating them to a suitable BWT construction algorithm, we can compute all the mentioned kernels and complexity measures, directly from the input strings, in O(n) time and in O(nlogσ)\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$O(n\log {\sigma })$$\end{document} bits of space in addition to the input, where σ\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\sigma $$\end{document} is the size of the alphabet. Using similar data structures, we also show how to build a compact representation of the variable-length Markov chain of a string T of length n, that takes just 3nlogσ+o(nlogσ)\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$3n\log {\sigma }+o(n\log {\sigma })$$\end{document} bits of space, and that can be learnt in randomized O(n) time using O(nlogσ)\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$O(n\log {\sigma })$$\end{document} bits of space in addition to the input. Such model can then be used to assign a probability to a query string S of length m in O(m) time and in 2m+o(m)\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$2m+o(m)$$\end{document} bits of additional space, thus providing an alternative, compositional measure of the similarity between S and T that does not require alignment.

Conway–Bromage–Lyndon (CBL): an exact, dynamic representation of k -mer sets

Conway-Bromage-Lyndon (CBL): an exact, dynamic representation of -mer sets

Brisk: Exact resource-efficient dictionary for k-mers

Hyper-k-mers: efficient streaming k-mers representation

Binary Collaborative Filtering Ensemble

Advances in practical k-mer sets: essentials for the curious

Disk compression of k-mer sets

Representation of k-Mer Sets Using Spectrum-Preserving String Sets

COBS: a Compact Bit-Sliced Signature Index

Eulertigs: minimum plain text representation of k-mer sets without repetitions in linear time

Lossless Indexing with Counting de Bruijn Graphs

Where the patterns are: repetition-aware compression for colored de Bruijn graphs

Advances in colored k-mer sets: essentials for the curious

CHTKC: a robust and efficient k-mer counting algorithm based on a lock-free chaining hash table

Cdbgtricks: strategies to update a compacted de bruijn graph

A Framework for Space-Efficient String Kernels

KMC 2: Fast and resource-frugal $k$-mer counting

A fast, lock-free approach for efficient parallel counting of occurrences of k-mers

These are not the k-mers you are looking for: efficient online k-mer counting using a probabilistic data structure

Prokrustean Graph: A substring index for rapid k-mer size analysis

A fast and simple approach to k-mer decomposition