Home

Performance measure

The prediction error we define here is $$\textrm{Pred.Err}(\hat\beta;y_{\rm test}, X_{\rm test}) = \frac{\textrm{RMSE}(\hat\beta;y_{\rm test} - X_{\rm test})}{\sigma} = \frac{\|y_{\rm test} - X_{\rm test} \hat\beta \|}{\sqrt{n}\sigma}$$ where $y_{\rm test}$ and $X_{\rm test}$ are test data sample in the same way. If $\hat\beta$ is fairly accurate, then we expect that $\rm RMSE$ is similar to $\sigma$. Therefore in average $\textrm{Pred.Err} \geq 1$ and the smaller the better.

Here RMSE is the root-mean-squared error, and

Default setting

Throughout the studies, the default setting is

1. The number of samples $n = 500$ and the number of coefficients $p = 2000$.
2. The number of nonzero coefficients $s = 20$ (sparsity)
3. Design: IndepGauss, i.e. $X_{ij} \sim N(0,1)$

4. Signal shape: PointNormal, i.e. $$\beta_j = \left\{ \begin{array}{ll} \textrm{a normal random sample }\sim N(0,1), & j \in S \\ 0, & j \notin S \end{array} \right.$$ where $S$ is an index set of nonzero coefficients, sampled uniformly at random from $\{1,\cdots,p\}$. We define $s = |S|$ is a sparsity level. If $s$ is smaller, then we say the signal $\beta$ is sparser.

5. PVE (Proportion of Variance Explained): 0.5 Given $X$ and $\beta$, we sample $y = X\beta + \epsilon$, where $\epsilon \sim N(0,\sigma^2 I_n)$. In this case, PVE is defined by $${\rm PVE} = \frac{\textrm{Var}(X\beta)}{\textrm{Var}(X\beta) + \sigma^2},$$ where $\textrm{Var}(a)$ denotes the sample variance of $a$ calculated using R function var. To this end, we set $\sigma^2 = \textrm{Var}(X\beta)$.
6. Update order: increasing, i.e. (1,...,p) = 1:p in this order.

7. Initialization: null i.e. a vector of zeros.

Some special cases

1. Ridge case.

The ridge case is when the default setting applies to all but $s = p$. We observe that the relative performance is dependent on the value of $p$ (compared to $n$). Thus we consider $p = 50,100,200,500,1000,2000$ while $n = 500$ being fixed.

The message here is MR.ASH performs reasonably well, and the only penalized linear regression method that outperforms Ridge with cross-validation. However, let us mention that Blasso (Bayesian Lasso) performs the best, which uses a non-sparse Laplace prior on $\beta$. Also, BayesB performs better than MR.ASH, which uses a spike-and-slab prior on $\beta$.

2. Nonconvex case.

This experiment is to compare the flexibility of convex/non-convex shrinkage operators (E-NET, SCAD, MCP, L0Learn) vs MR.ASH shrinkage operator. The default setting applies to all but the signal shape and PVE = 0.99.

The reason why we set PVE = 0.99, the cross-validated solution of the method (E-NET, SCAD or MCP) acheives a nearly minimum prediction error on the solution path, i.e. the cross-validated solution is nearly the best the method can achieve. Thus we conclude that the cross-validation is fairly accurate.

Important parameters for high-dimensional regression settings

3. High-dimensionality $p$.

This experiment is to compare performance of the methods when $p$ varies while the rest being fixed.

(3-1) Result:

(3-2) Discussion: since the theory says (e.g. Slope meets Lasso: improved oracle bounds and optimality ) that the prediction error scales as $\sqrt{s\log(p/s)/n}$

4. Sparsity $s$.

5. PVE.

Performance on different design settings

6. LowdimIndepGauss

7. RealGenotype

8. Changepoint

9. EquicorrGauss