Abstract:In this paper we propose a variant of the linear least squares model allowing practitioners to partition the input features into groups of variables that they require to contribute similarly to the final result. The output allows practitioners to assess the importance of each group and of each variable in the group. We formally show that the new formulation is not convex and provide two alternative methods to deal with the problem: one non-exact method based on an alternating least squares approach; and one exact method based on a reformulation of the problem using an exponential number of sub-problems whose minimum is guaranteed to be the optimal solution. We formally show the correctness of the exact method and also compare the two solutions showing that the exact solution provides better results in a fraction of the time required by the alternating least squares solution (assuming that the number of partitions is small). For the sake of completeness, we also provide an alternative branch and bound algorithm that can be used in place of the exact method when the number of partitions is too large, and a proof of NP-completeness of the optimization problem introduced in this paper.

What problem does this paper attempt to address?

The paper primarily focuses on addressing the issue of how to better utilize feature grouping in linear regression analysis to enhance the interpretability and practicality of the model. Specifically, the researchers propose a new variant of the linear least squares model—the Partitioned Least Squares (PartitionedLS) model, which allows users to divide input features into different groups and requires the variables within these groups to make similar contributions to the final result. ### Research Questions The main issues the paper attempts to address can be summarized as follows: 1. **Feature Grouping**: In certain application areas, such as chemistry and educational assessment, grouping variables is highly beneficial. This not only helps incorporate domain expert knowledge but also improves the interpretability of the model. 2. **Consistency in Contribution Direction**: To ensure that the features within the same group contribute in a consistent direction to the prediction result (i.e., either all positive or all negative contributions), the paper proposes a new model to constrain the signs of the feature parameters, ensuring that features within a group have the same contribution direction. 3. **Non-convex Optimization Problem**: The paper proves that the proposed Partitioned Least Squares problem is non-convex and NP-complete, which means that directly solving for the optimal solution is challenging. ### Solutions To address the above issues, the paper proposes two methods: 1. **Alternating Least Squares Method**: This is an approximate solution method that iteratively solves the problem by alternately optimizing parameters α and β. Although this method may get trapped in local optima, it usually provides good solutions in practical applications. 2. **Exact Solution Method**: Based on a reformulation of the problem, this method solves it by converting it into a series of subproblems. Each subproblem is convex and can find the global optimal solution by enumerating all possible cases. Additionally, a branch-and-bound algorithm is provided to handle situations where the number of groups is large. ### Experimental Validation The paper validates the effectiveness of the proposed methods through experiments on multiple datasets, including applications such as chemical compound analysis and house price prediction. The experimental results show that the exact solution method usually provides better and faster results when the number of feature groups is small; whereas, in cases with many feature groups or where very high precision is not required, the alternating least squares method is a better choice. In summary, by introducing the concept of feature grouping and providing corresponding solution algorithms, this paper aims to improve the practicality and interpretability of existing least squares regression models.

Partitioned Least Squares

lspartition: Partitioning-Based Least Squares Regression

An efficient variable projection formulation for separable nonlinear least squares problems.

Partitioning group correction Cholesky techniques for large scale sparse unconstrained optimization

Subsampling for Partial Least-Squares Regression Via an Influence Function

An Efficient Algorithm for Ill-Conditioned Separable Nonlinear Least Squares

Author Response for "partial Least Median of Squares Regression"

Efficient Parameters Estimation Method for the Separable Nonlinear Least Squares Problem

Partition-wise Linear Models

Partial least trimmed squares regression

Parallel Inexact Levenberg-Marquardt Method for Nearly-Separable Nonlinear Least Squares

[HDDA] sparse subspace constrained partial least squares

On Some Separated Algorithms for Separable Nonlinear Least Squares Problems

New algorithms for computing the least trimmed squares estimator

Variance constrained partial least squares

Variations on least squares

On Relations Between Weighted Least-Squares Estimators of Parametric Functions under a General Partitioned Linear Model and Its Small Models

A Proximal Peaceman-Rachford Splitting Method for Solving the Multi-Block Separable Convex Minimization Problems.

A Levenberg-Marquardt Method for Nonsmooth Regularized Least Squares

A Strictly Contractive Peaceman-Rachford Splitting Method with Logarithmic-Quadratic Proximal Regularization for Convex Programming

Orthogonal least squares based on singular value decomposition for spare basis selection