Information bounds for the risk of Bayesian predictions and the redundancy of universal codes

Andrew Barron, Bertrand S Clarke, David Raussler

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Citations (Scopus)

Abstract

Several diverse problems have solutions in terms of an information-theoretic quantity for which we examine the asymptotics. Let Y1, Y2,..., YN be a sample of random variables with distribution depending on a (possibly infinite-dimensional) parameter θ. The maximum of the mutual information IN = I(θ; Y1, Y2,..., YN) over choices of the prior distribution of θ provides a bound on the cumulative Bayes risk of prediction of the sequence of random variables for several choices of loss function. This same quantity is the minimax redundancy of universal data compression and the capacity of certain channels. General bounds for this mutual information are given. A special case concerns the estimation of binary-valued functions with Vapnik-Chervonenkis dimension dvc, for which the information is bounded by dvc log N. For smooth families of probability densities with a Euclidean parameter of dimension d, the information bound is (d/2) log N plus a constant. The prior density proportional to the square root of the Fisher information determinant is the unique continuous density that achieves a mutual information within o(1) of the capacity for large N. The Bayesian procedure with this prior is asymptotically minimax for the cumulative relative entropy risk.

Original languageEnglish (US)
Title of host publicationProceedings of the 1993 IEEE International Symposium on Information Theory
PublisherPubl by IEEE
Number of pages1
ISBN (Print)0780308786
StatePublished - Jan 1 1993
EventProceedings of the 1993 IEEE International Symposium on Information Theory - San Antonio, TX, USA
Duration: Jan 17 1993Jan 22 1993

Publication series

NameProceedings of the 1993 IEEE International Symposium on Information Theory

Other

OtherProceedings of the 1993 IEEE International Symposium on Information Theory
CitySan Antonio, TX, USA
Period1/17/931/22/93

Fingerprint

Random variables
Redundancy
Data compression
Entropy

ASJC Scopus subject areas

  • Engineering(all)

Cite this

Barron, A., Clarke, B. S., & Raussler, D. (1993). Information bounds for the risk of Bayesian predictions and the redundancy of universal codes. In Proceedings of the 1993 IEEE International Symposium on Information Theory (Proceedings of the 1993 IEEE International Symposium on Information Theory). Publ by IEEE.

Information bounds for the risk of Bayesian predictions and the redundancy of universal codes. / Barron, Andrew; Clarke, Bertrand S; Raussler, David.

Proceedings of the 1993 IEEE International Symposium on Information Theory. Publ by IEEE, 1993. (Proceedings of the 1993 IEEE International Symposium on Information Theory).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Barron, A, Clarke, BS & Raussler, D 1993, Information bounds for the risk of Bayesian predictions and the redundancy of universal codes. in Proceedings of the 1993 IEEE International Symposium on Information Theory. Proceedings of the 1993 IEEE International Symposium on Information Theory, Publ by IEEE, Proceedings of the 1993 IEEE International Symposium on Information Theory, San Antonio, TX, USA, 1/17/93.
Barron A, Clarke BS, Raussler D. Information bounds for the risk of Bayesian predictions and the redundancy of universal codes. In Proceedings of the 1993 IEEE International Symposium on Information Theory. Publ by IEEE. 1993. (Proceedings of the 1993 IEEE International Symposium on Information Theory).
Barron, Andrew ; Clarke, Bertrand S ; Raussler, David. / Information bounds for the risk of Bayesian predictions and the redundancy of universal codes. Proceedings of the 1993 IEEE International Symposium on Information Theory. Publ by IEEE, 1993. (Proceedings of the 1993 IEEE International Symposium on Information Theory).
@inproceedings{050c526790f44ec49742ce5e756d0436,
title = "Information bounds for the risk of Bayesian predictions and the redundancy of universal codes",
abstract = "Several diverse problems have solutions in terms of an information-theoretic quantity for which we examine the asymptotics. Let Y1, Y2,..., YN be a sample of random variables with distribution depending on a (possibly infinite-dimensional) parameter θ. The maximum of the mutual information IN = I(θ; Y1, Y2,..., YN) over choices of the prior distribution of θ provides a bound on the cumulative Bayes risk of prediction of the sequence of random variables for several choices of loss function. This same quantity is the minimax redundancy of universal data compression and the capacity of certain channels. General bounds for this mutual information are given. A special case concerns the estimation of binary-valued functions with Vapnik-Chervonenkis dimension dvc, for which the information is bounded by dvc log N. For smooth families of probability densities with a Euclidean parameter of dimension d, the information bound is (d/2) log N plus a constant. The prior density proportional to the square root of the Fisher information determinant is the unique continuous density that achieves a mutual information within o(1) of the capacity for large N. The Bayesian procedure with this prior is asymptotically minimax for the cumulative relative entropy risk.",
author = "Andrew Barron and Clarke, {Bertrand S} and David Raussler",
year = "1993",
month = "1",
day = "1",
language = "English (US)",
isbn = "0780308786",
series = "Proceedings of the 1993 IEEE International Symposium on Information Theory",
publisher = "Publ by IEEE",
booktitle = "Proceedings of the 1993 IEEE International Symposium on Information Theory",

}

TY - GEN

T1 - Information bounds for the risk of Bayesian predictions and the redundancy of universal codes

AU - Barron, Andrew

AU - Clarke, Bertrand S

AU - Raussler, David

PY - 1993/1/1

Y1 - 1993/1/1

N2 - Several diverse problems have solutions in terms of an information-theoretic quantity for which we examine the asymptotics. Let Y1, Y2,..., YN be a sample of random variables with distribution depending on a (possibly infinite-dimensional) parameter θ. The maximum of the mutual information IN = I(θ; Y1, Y2,..., YN) over choices of the prior distribution of θ provides a bound on the cumulative Bayes risk of prediction of the sequence of random variables for several choices of loss function. This same quantity is the minimax redundancy of universal data compression and the capacity of certain channels. General bounds for this mutual information are given. A special case concerns the estimation of binary-valued functions with Vapnik-Chervonenkis dimension dvc, for which the information is bounded by dvc log N. For smooth families of probability densities with a Euclidean parameter of dimension d, the information bound is (d/2) log N plus a constant. The prior density proportional to the square root of the Fisher information determinant is the unique continuous density that achieves a mutual information within o(1) of the capacity for large N. The Bayesian procedure with this prior is asymptotically minimax for the cumulative relative entropy risk.

AB - Several diverse problems have solutions in terms of an information-theoretic quantity for which we examine the asymptotics. Let Y1, Y2,..., YN be a sample of random variables with distribution depending on a (possibly infinite-dimensional) parameter θ. The maximum of the mutual information IN = I(θ; Y1, Y2,..., YN) over choices of the prior distribution of θ provides a bound on the cumulative Bayes risk of prediction of the sequence of random variables for several choices of loss function. This same quantity is the minimax redundancy of universal data compression and the capacity of certain channels. General bounds for this mutual information are given. A special case concerns the estimation of binary-valued functions with Vapnik-Chervonenkis dimension dvc, for which the information is bounded by dvc log N. For smooth families of probability densities with a Euclidean parameter of dimension d, the information bound is (d/2) log N plus a constant. The prior density proportional to the square root of the Fisher information determinant is the unique continuous density that achieves a mutual information within o(1) of the capacity for large N. The Bayesian procedure with this prior is asymptotically minimax for the cumulative relative entropy risk.

UR - http://www.scopus.com/inward/record.url?scp=0027307823&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=0027307823&partnerID=8YFLogxK

M3 - Conference contribution

SN - 0780308786

T3 - Proceedings of the 1993 IEEE International Symposium on Information Theory

BT - Proceedings of the 1993 IEEE International Symposium on Information Theory

PB - Publ by IEEE

ER -