Theoretical Guarantees for Bayesian Generalized Linear Regression And Variational Boosting
Abstract
I provide statistical guarantees for Bayesian variational boosting by
proposing a novel small bandwidth Gaussian mixture variational family. We employ a functional
version of Frank-Wolfe optimization as our variational algorithm and study frequentist properties of the iterative boosting updates. Comparisons are drawn to the recent literature on boosting, describing
how the choice of the variational family and the discrepancy measure affect both convergence and
finite-sample statistical properties of the optimization routine. Specifically, we first demonstrate
stochastic boundedness of the boosting iterates with respect to the data generating distribution. We
next integrate this within our algorithm to provide an explicit convergence rate, ending with a result
on the required number of boosting updates. Next, I develop a framework to study posterior contraction rates in sparse high dimensional generalized linear models (GLM). We introduce a new family of GLMs, denoted by clipped GLM, which subsumes many standard GLMs and makes minor modification of the rest. With a sparsity inducing prior on the regression coefficients, we delineate sufficient conditions
on true data generating density that leads to minimax optimal rates of posterior contraction of the coefficients in l_1 norm. Our key contribution is to develop sufficient conditions commensurate with the geometry of the clipped GLM family, propose prior distributions which do not require any knowledge of the true parameters and avoid any assumption on the growth rate of the true coefficient vector
Subject
Approximate Bayesian InferenceVariational Boosting
Frank--Wolfe Algorithm
Convergence Rate
Kullback-Leibler Divergence
Gaussian Mixtures
High-dimension
Sparse Regression
Generalized Linear Models
Posterior Convergence
Model Selection
Adaptive Estimation
Spike-and-slab Prior
Minimax Rate
Citation
Guha, Biraj Subhra (2021). Theoretical Guarantees for Bayesian Generalized Linear Regression And Variational Boosting. Doctoral dissertation, Texas A&M University. Available electronically from https : / /hdl .handle .net /1969 .1 /195138.