Abstract:Minimizing a convex function of matrices regularized by the nuclear norm arises in many applications such as collaborative filtering and multi-task learning. In this paper, we study the general setting where the convex function could be non-smooth. When the size of the data matrix, denoted by m x n, is very large, existing optimization methods are inefficient because in each iteration, they need to perform a singular value decomposition (SVD) which takes O(m^2 n) time. To reduce the computation cost, we exploit the dual characterization of the nuclear norm to introduce a convex-concave optimization problem and design a subgradient-based algorithm without performing SVD. In each iteration, the proposed algorithm only computes the largest singular vector, reducing the time complexity from O(m^2 n) to O(mn). To the best of our knowledge, this is the first SVD-free convex optimization approach for nuclear-norm regularized problems that does not rely on the smoothness assumption. Theoretical analysis shows that the proposed algorithm converges at an optimal O(1/\sqrt{T}) rate where T is the number of iterations. We also extend our algorithm to the stochastic case where only stochastic subgradients of the convex function are available and a special case that contains an additional non-smooth regularizer (e.g., L1 norm regularizer). We conduct experiments on robust low-rank matrix approximation and link prediction to demonstrate the efficiency of our algorithms.

Nuclear norm regularized loop optimization for tensor network

Loop Optimization for Tensor Network Renormalization

Nuclear Norm Regularization for Deep Learning

SVD-free Convex-Concave Approaches for Nuclear Norm Regularization.

Low-rank Tensor Learning with Nonconvex Overlapped Nuclear Norm Regularization

Stochastic Proximal Gradient Descent for Nuclear Norm Regularization

Tensor network renormalization: application to dynamic correlation functions and non-hermitian systems

Matrix completion by Truncated Nuclear Norm Regularization

A lagrange programming neural network approach for nuclear norm optimization

Nonconvex Nonsmooth Low-Rank Minimization Via Iteratively Reweighted Nuclear Norm

Tensor recovery using the tensor nuclear norm based on nonconvex and nonlinear transformations

Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernel

Nesting-structured Nuclear Norm Minimization for Spatially Correlated Matrix Variate.

Low-Rank Tensor Completion Via Tensor Nuclear Norm with Hybrid Smooth Regularization.

Online Optimization for Large-Scale Max-Norm Regularization

Linearized augmented Lagrangian and alternating direction methods for nuclear norm minimization.

Accelerating nuclear-norm regularized low-rank matrix optimization through Burer-Monteiro decomposition

Distributed Design for Nuclear Norm Minimization of Linear Matrix Equations with Constraints

Truncated nuclear norm regularization for low-rank tensor completion

Large Scale Multi-Class Classification with Truncated Nuclear Norm Regularization

A Frobenius Norm Regularization Method for Convolutional Kernel Tensors in Neural Networks