Abstract:The inference performance of the pseudolikelihood method is discussed in the framework of the inverse Ising problem when the l (2)-regularized (ridge) linear regression is adopted. This setup is introduced for theoretically investigating the situation where the data generation model is different from the inference one, namely the model mismatch situation. In the teacher-student scenario under the assumption that the teacher couplings are sparse, the analysis is conducted using the replica and cavity methods, with a special focus on whether the presence/absence of teacher couplings is correctly inferred or not. The result indicates that despite the model mismatch, one can perfectly identify the network structure using naive linear regression without regularization when the number of spins N is smaller than the dataset size M, in the thermodynamic limit N -> infinity. Further, to access the underdetermined region M < N, we examine the effect of the l (2) regularization, and find that biases appear in all the coupling estimates, preventing the perfect identification of the network structure. We, however, find that the biases are shown to decay exponentially fast as the distance from the center spin chosen in the pseudolikelihood method grows. Based on this finding, we propose a two-stage estimator: in the first stage, the ridge regression is used and the estimates are pruned by a relatively small threshold; in the second stage the naive linear regression is conducted only on the remaining couplings, and the resultant estimates are again pruned by another relatively large threshold. This estimator with the appropriate regularization coefficient and thresholds is shown to achieve the perfect identification of the network structure even in 0 < M/N < 1. Results of extensive numerical experiments support these findings.

Concentration Estimates for Learning with ℓ1-Regularizer and Data Dependent Hypothesis Spaces

Learning Theory Estimates for Coefficient-Based Regularized Regression

Structure Learning in Inverse Ising Problems Using ℓ_2-Regularized Linear Estimator

Concentration Estimates for Learning with Unbounded Sampling

Learning with coefficient-based regularization and ℓ 1 −penalty

Generalization error bound of semi-supervised learning with ℓ1 regularization in sum space.

Linear Convergence of Inexact Descent Method and Inexact Proximal Gradient Algorithms for Lower-Order Regularization Problems

Quantile Regression With L(1)-Regularization And Gaussian Kernels

High-dimensional Inference Via Lipschitz Sparsity-Yielding Regularizers.

Tikhonov Regularization with L^0-Term Complementing a Convex Penalty: L^1 Convergence under Sparsity Constraints

Analysis of Regularized Learning for Linear-functional Data in Banach Spaces

On the Performance of Regularized Regression Learning in Hilbert Space

A Duality Approach to Regularized Learning Problems in Banach Spaces

Analysis of Regularized Least-Squares in Reproducing Kernel Kreĭn Spaces

Learning rates for regularized least squares ranking algorithm

Optimal rate of the regularized regression learning algorithm

Learning Rates of Regularized Regression on the Unit Sphere

Constrained Analysis Dictionary Learning with the L(1/2)-Norm Regularizer

Learning with Convex Loss and Indefinite Kernels

Regularized Regression Problem in hyper-RKHS for Learning Kernels

Error Analysis on Regularized Regression Based on the Maximum Correntropy Criterion