On variational Bayes estimation and variational information criteria for linear regression models

Chong You*, John T. Ormerod, Samuel Müller

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

18 Citations (Scopus)

Abstract

Variational Bayes (VB) estimation is a fast alternative to Markov Chain Monte Carlo for performing approximate Baesian inference. This procedure can be an efficient and effective means of analyzing large datasets. However, VB estimation is often criticised, typically on empirical grounds, for being unable to produce valid statistical inferences. In this article we refute this criticism for one of the simplest models where Bayesian inference is not analytically tractable, that is, the Bayesian linear model (for a particular choice of priors). We prove that under mild regularity conditions, VB based estimators enjoy some desirable frequentist properties such as consistency and can be used to obtain asymptotically valid standard errors. In addition to these results we introduce two VB information criteria: the variational Akaike information criterion and the variational Bayesian information criterion. We show that variational Akaike information criterion is asymptotically equivalent to the frequentist Akaike information criterion and that the variational Bayesian information criterion is first order equivalent to the Bayesian information criterion in linear regression. These results motivate the potential use of the variational information criteria for more complex models. We support our theoretical results with numerical examples.

Original languageEnglish
Pages (from-to)73-87
Number of pages15
JournalAustralian and New Zealand Journal of Statistics
Volume56
Issue number1
DOIs
Publication statusPublished - Mar 2014
Externally publishedYes

Keywords

  • Akaike information criterion
  • Bayesian information criterion
  • consistency
  • deviance information criterion
  • Markov Chain Monte Carlo

Fingerprint Dive into the research topics of 'On variational Bayes estimation and variational information criteria for linear regression models'. Together they form a unique fingerprint.

Cite this