Jeffreys' prior is asymptotically least favorable under entropy risk

Bertrand S. Clarke, Andrew R. Barron

Research output: Contribution to journalArticle

163 Scopus citations

Abstract

We provide a rigorous proof that Jeffreys' prior asymptotically maximizes Shannon's mutual information between a sample of size n and the parameter. This was conjectured by Bernardo (1979) and, despite the absence of a proof, forms the basis of the reference prior method in Bayesian statistical analysis. Our proof rests on an examination of large sample decision theoretic properties associated with the relative entropy or the Kullback-Leibler distance between probability density functions for independent and identically distributed random variables. For smooth finite-dimensional parametric families we derive an asymptotic expression for the minimax risk and for the related maximin risk. As a result, we show that, among continuous positive priors, Jeffreys' prior uniquely achieves the asymptotic maximin value. In the discrete parameter case we show that, asymptotically, the Bayes risk reduces to the entropy of the prior so that the reference prior is seen to be the maximum entropy prior. We identify the physical significance of the risks by giving two information-theoretic interpretations in terms of probabilistic coding.

Original languageEnglish (US)
Pages (from-to)37-60
Number of pages24
JournalJournal of Statistical Planning and Inference
Volume41
Issue number1
DOIs
Publication statusPublished - Aug 1994

    Fingerprint

Keywords

  • Bayes risk
  • Fisher information
  • Jeffreys' prior
  • Kullback-Leibler information
  • Shannon's mutual information
  • data compression
  • minimax risk
  • parametric density estimation
  • reference priors, least favorable priors

ASJC Scopus subject areas

  • Statistics and Probability
  • Statistics, Probability and Uncertainty
  • Applied Mathematics

Cite this