Statistical Problem Classes and Their Links to Information Theory

Bertrand Clarke, Jennifer Clarke, Chi Wai Yu

Research output: Contribution to journalArticle

3 Scopus citations

Abstract

We begin by recalling the tripartite division of statistical problems into three classes, M-closed, M-complete, and M-open and then reviewing the key ideas of introductory Shannon theory. Focusing on the related but distinct goals of model selection and prediction, we argue that different techniques for these two goals are appropriate for the three different problem classes. For M-closed problems we give relative entropy justification that the Bayes information criterion (BIC) is appropriate for model selection and that the Bayes model average is information optimal for prediction. For M-complete problems, we discuss the principle of maximum entropy and a way to use the rate distortion function to bypass the inaccessibility of the true distribution. For prediction in the M-complete class, there is little work done on information based model averaging so we discuss the Akaike information criterion (AIC) and its properties and variants. For the M-open class, we argue that essentially only predictive criteria are suitable. Thus, as an analog to model selection, we present the key ideas of prediction along a string under a codelength criterion and propose a general form of this criterion. Since little work appears to have been done on information methods for general prediction in the M-open class of problems, we mention the field of information theoretic learning in certain general function spaces.

Original languageEnglish (US)
Pages (from-to)337-371
Number of pages35
JournalEconometric Reviews
Volume33
Issue number1-4
DOIs
Publication statusPublished - Feb 1 2014

    Fingerprint

Keywords

  • Bayesian
  • Codelength
  • Entropy
  • Information theory
  • M-closed
  • M-complete
  • M-open
  • Model selection
  • Mutual information
  • Prediction
  • Rate distortion
  • Relative entropy

ASJC Scopus subject areas

  • Economics and Econometrics

Cite this