Abstract:Large models and enormous data are essential driving forces of the unprecedented successes achieved by modern algorithms, especially in scientific computing and machine learning. Nevertheless, the growing dimensionality and model complexity, as well as the non-negligible workload of data pre-processing, also bring formidable costs to such successes in both computation and data aggregation. As the deceleration of Moore's Law slackens the cost reduction of computation from the hardware level, fast heuristics for expensive classical routines and efficient algorithms for exploiting limited data are increasingly indispensable for pushing the limit of algorithm potency. This thesis explores some of such algorithms for fast execution and efficient data utilization. From the computational efficiency perspective, we design and analyze fast randomized low-rank decomposition algorithms for large matrices based on "matrix sketching", which can be regarded as a dimension reduction strategy in the data space. These include the randomized pivoting-based interpolative and CUR decomposition discussed in Chapter 2 and the randomized subspace approximations discussed in Chapter 3. From the sample efficiency perspective, we focus on learning algorithms with various incorporations of data augmentation that improve generalization and distributional robustness provably. Specifically, Chapter 4 presents a sample complexity analysis for data augmentation consistency regularization where we view sample efficiency from the lens of dimension reduction in the function space. Then in Chapter 5, we introduce an adaptively weighted data augmentation consistency regularization algorithm for distributionally robust optimization with applications in medical image segmentation.

On random embeddings and their application to optimisation

Optimization Can Learn Johnson Lindenstrauss Embeddings

On Probabilistic Embeddings in Optimal Dimension Reduction

Derivative-Free Optimization of High-Dimensional Non-Convex Functions by Sequential Random Embeddings

Optimizations on Unknown Low-Dimensional Structures Given by High-Dimensional Data

Adaptive and Oblivious Randomized Subspace Methods for High-Dimensional Optimization: Sharp Analysis and Lower Bounds

Bayesian Optimization in a Billion Dimensions via Random Embeddings

Random mappings designed for commercial search engines

Randomised subspace methods for non-convex optimization, with applications to nonlinear least-squares

Random projections of random manifolds

Optimisation via encodings: a renormalisation group perspective

Linear shrinkage for optimization in high dimensions

On The Expected Distortion Bound Of Direct Random Projection And Its Applications

High-Dimensional Bayesian Optimization Using Both Random and Supervised Embeddings

Optimizing Probabilistic Box Embeddings with Distance Measures

Normed Spaces for Graph Embedding

Optimal Oblivious Subspace Embeddings with Near-optimal Sparsity

Subspace Embeddings Under Nonlinear Transformations

Embedded Bandits for Large-Scale Black-Box Optimization

Random Projections for Classification: A Recovery Approach

Randomized Dimension Reduction with Statistical Guarantees