Abstract:Large models and enormous data are essential driving forces of the unprecedented successes achieved by modern algorithms, especially in scientific computing and machine learning. Nevertheless, the growing dimensionality and model complexity, as well as the non-negligible workload of data pre-processing, also bring formidable costs to such successes in both computation and data aggregation. As the deceleration of Moore's Law slackens the cost reduction of computation from the hardware level, fast heuristics for expensive classical routines and efficient algorithms for exploiting limited data are increasingly indispensable for pushing the limit of algorithm potency. This thesis explores some of such algorithms for fast execution and efficient data utilization. From the computational efficiency perspective, we design and analyze fast randomized low-rank decomposition algorithms for large matrices based on "matrix sketching", which can be regarded as a dimension reduction strategy in the data space. These include the randomized pivoting-based interpolative and CUR decomposition discussed in Chapter 2 and the randomized subspace approximations discussed in Chapter 3. From the sample efficiency perspective, we focus on learning algorithms with various incorporations of data augmentation that improve generalization and distributional robustness provably. Specifically, Chapter 4 presents a sample complexity analysis for data augmentation consistency regularization where we view sample efficiency from the lens of dimension reduction in the function space. Then in Chapter 5, we introduce an adaptively weighted data augmentation consistency regularization algorithm for distributionally robust optimization with applications in medical image segmentation.

Fast Dimension Reduction Through Random Permutation.

FAST AND EFFICIENT DIMENSIONALITY REDUCTION USING STRUCTURALLY RANDOMMATRICES

Randomized Dimension Reduction with Statistical Guarantees

Fast Fisher discriminant analysis with randomized algorithms.

On Probabilistic Embeddings in Optimal Dimension Reduction

Linearly-scalable learning of smooth low-dimensional patterns with permutation-aided entropic dimension reduction

A survey of dimensionality reduction techniques based on random projection

Permutation invariant functions: statistical tests, density estimation, and computationally efficient embedding

Fast and Orthogonal Locality Preserving Projections for Dimensionality Reduction.

Fast Fisher Sparsity Preserving Projections

Permutation methods for factor analysis and PCA

Incremental dimension reduction of tensors with random index

Tensor Random Projection for Low Memory Dimension Reduction

A Permutation Network for Configurable and Scalable FFT Processors.

Fréchet sufficient dimension reduction for random objects

Fast Computation of the Multidimensional Discrete Fourier Transform and Discrete Backward Fourier Transform on Sparse Grids

Orthogonal Component Analysis: A Fast Dimensionality Reduction Algorithm

Random Angular Projection for Fast Nearest Subspace Search

Randomized Algorithms for Computation of Tucker Decomposition and Higher Order SVD (HOSVD)

Randomized Dynamic Mode Decomposition

Parallel Randomized Tucker Decomposition Algorithms