Bayesian Model Averaging for Elastic Net Using Regularization Path and its Applications

Researchers often build models in multiple regressions and then select the best one using certain variable selection methods. However, none of the existing approaches seem to deal with the uncertainty related to selected models and to balance overfitting and biased prediction in the sense of considering shrinkage and multicollinearity in variable selections as well as incorporating model uncertainty. In contrast, Bayesian model averaging (BMA) is proposed as a Bayesian approach to quantify uncertainty. BMA provides a way to tackle model uncertainty and becomes popular as a data analysis tool along with model selection process. In the meantime, regression regularization methods become more and more popular among statisticians for a more frequent appearance in high-dimensional problems. Regularized regression achieves simultaneous parameter estimation and variable selection by penalizing the model parameters and shrinking them towards zero.

In this dissertation, we propose and investigate the elastic net shrinkage method under Bayesian model averaging for regression problems. This method combines the strength of the elastic net shrinkage and the Bayesian model averaging. It handles collinearity and model uncertainty simultaneously. We extend this method to variable selection by credible interval criteria. We develop the computational algorithm for the proposed method and compare its performance with lasso, elastic net, Bayesian lasso, Bayesian elastic net and Bayesian model averaging methods in simulation studies as well as in applications to real datasets. Results show that the proposed method works better than the existing ones in many situations. We also extend the proposed method to logistic regression, and the simulation studies show smaller classification error comparing to simple logistic regression. In addition, we incorporate proposed Bayesian elastic net averaging method with quantile regression in order to deal with skewed distribution.