4.1 Settings
In this section, we will conduct a monte carol study on our proposed best subset ridge regression method and make a comparison with other commonly used variable selection methods in three aspects. The first aspect is the predictive performance on a held-out testing data of the same size of training data. For linear regression, this is defined as \(\frac{\Vert X \beta^\dagger - X \hat{\beta}\Vert _2}{\Vert X \beta^\dagger\Vert_2}\), where \(\beta^\dagger\) is the true coefficient and \(\hat{\beta}\) is an estimator. For logistic regression, we calculate the classification accuracy by the area under the ROC curve (AUC). The second aspect is the coefficient estimation performance defined as \(\Vert \beta^\dagger - \hat{\beta}\Vert\), where \(\beta^\dagger\) denote the underlying true coefficients and \(\beta^\dagger\) is out estimation. The third aspect is the selection performance in terms of true positive rate (TPR), false positive rate (FPR), the Matthews correlation coefficient (MCC) score, and the model size.
We generate a multivariate Gaussian data matrix \(X_{n\times p} \sim MVN(0, \Sigma)\) and consider the variables to have exponential correlation. \(n\) denotes the sample size and \(p\) the number of total variables, and we set \(n = 200\), \(p = 2000\). We consider the following instance of \(\Sigma := ((\sigma_{ij}))\), setting each entry \(\sigma_{ij} = 0.5 ^{|i-j|}\). We use a sparse coefficient vector \(\beta^\dagger\) with \(20\) equi-spaced nonzero entries, each set to 1. Then the response vector y is generated with gaussian noise added. For linear regression, \(y = X\beta^\dagger+\epsilon\). For logistic regression, \(y = Bernoulli(Pr(Y=1))\), where \(Pr(Y=1) = \exp (x^T \beta^\dagger + \epsilon)/(1+\exp (x^T \beta^\dagger + \epsilon))\). \(\epsilon\sim N(0, \sigma^2)\) is independent of \(X\). We define the signal-to-noise ratio (SNR) by SNR = \(\frac{Var(X\beta^\dagger)}{Var(\epsilon)} = \frac{\beta^{\dagger T} \Sigma \beta^\dagger}{\sigma^2}\).
we compare the performance of the following methods:
(BSRR methods): Tuning parameters selected through a sequential path and the Powell method. Denoted as Grid-BSRR and Powell-BSRR.
(BSS method): The best subset selection method. We consider the primal-dual active subset selection method and use our own implementation in bestridge package.
(\(L_1\) method): Lasso estimator. We use the implementation of glmnet.
(\(L_1L_2\) method): Elastic net estimator. This uses a combination of the \(L_1\) and \(L_2\) regularization. We use the implementation of glmnet.
For BSRR estimators, the 2D grid of tuning parameters has \(\lambda\) taking 100 values between \(\lambda_{max}\) and \(\lambda_{min}\) on a log scale. The \(\lambda_{max}\) and \(\lambda_{min}\) will be specified. For the BSS method, the model size parameter takes value in \(\{1, \dots, 30\}\). The parameter combined the \(L_1\) penalty and the \(L_2\) penalty in the Elastic net is fixed at \(0.5\). And for \(L_1\) and \(L_1L_2\) methods, we use the default settings. The regularization parameters are chosen by 5-fold cross-validation.
4.2 Linear regression
Results for linear regression are reported in Figure 1. When SNR is low, BSS has the largest estimation error and selects the smallest model size compared with other methods, while the extra \(L_2\) penalty in BSRR has effectively lowered the estimation error of BSS. For high SNR, the performance of BSS and BSRR is similar to each other. In terms of prediction error, all methods are good in the high SNR scenario, but it costs the Lasso and the Elastic Net a very dense supports while still cannot catch up with BSS and BSRR. Notice that for SNR 100, the selected model size of the Elastic Net is almost 7 times the true model size. This trend is no better for the Lasso. Of all measures across the whole SNR range, BSRR generally exhibits excellent performance—accurate in variables selection and prediction.
Figure 1: Performance measures as the signal-to-noise ratio (SNR) is varied between 0.01 and 100 for linear regression.
4.3 Logistic regression
Results for linear regression are reported in Figure 2. The result shows that as the SNR rises, the extra shrinkage in BSRR methods helps the best subset ridge regression make impressive improvement in terms of prediction accuracy, estimation ability, and variable selection, where it outperforms the state-of-the-art variable selection methods Lasso and Elastic Net.
Figure 2: Performance measures as the signal-to-noise ratio(SNR)is varied between 0.01 and 100 for logistic regression.
