Abstract
L0 penalized likelihood procedures like Mallows' Cp, AIC, and BIC directly penalize for the number of variables included in a regression model. This is a straightforward approach to the problem of overfitting, and these methods are now part of every statistician's repertoire. However, these procedures have been shown to sometimes result in unstable parameter estimates as a result on the L0 penalty's discontinuity at zero. One proposed alternative, seamless-L0 (SELO), utilizes a continuous penalty function that mimics L0 and allows for stable estimates. Like other similar methods (e.g. LASSO and SCAD), SELO produces sparse solutions because the penalty function is non-differentiable at the origin. Because these penalized likelihoods are singular (non-differentiable) at zero, there is no closed-form solution for the extremum of the objective function. We propose a continuous and everywhere-differentiable penalty function that can have arbitrarily steep slope in a neighborhood near zero, thus mimicking the L0 penalty, but allowing for a nearly closed-form solution for the beta-hat vector. Because our function is not singular at zero, beta-hat will have no zero-valued components, although some will have been shrunk arbitrarily close thereto. We employ a BIC-selected tuning parameter used in the shrinkage step to perform zero-thresholding as well. We call the resulting vector of coefficients the ShrinkSet estimator. It is comparable to SELO in terms of model performance (selecting the truly nonzero coefficients, overall MSE, etc.), but we believe it to be more intuitive and simpler to compute. We provide strong evidence that the estimator enjoys favorable asymptotic properties, including the oracle property.
Degree
MS
College and Department
Physical and Mathematical Sciences; Statistics
Rights
http://lib.byu.edu/about/copyright/
BYU ScholarsArchive Citation
VanDerwerken, Douglas Nielsen, "Variable Selection and Parameter Estimation Using a Continuous and Differentiable Approximation to the L0 Penalty Function" (2011). Theses and Dissertations. 2486.
https://scholarsarchive.byu.edu/etd/2486
Date Submitted
2011-03-10
Document Type
Selected Project
Handle
http://hdl.lib.byu.edu/1877/etd4248
Keywords
Penalized likelihood, variable selection, oracle property, large p, small n
Language
English