### Abstract

Several diverse problems have solutions in terms of an information-theoretic quantity for which we examine the asymptotics. Let Y_{1}, Y_{2},..., Y_{N} be a sample of random variables with distribution depending on a (possibly infinite-dimensional) parameter θ. The maximum of the mutual information I_{N} = I(θ; Y_{1}, Y_{2},..., Y_{N}) over choices of the prior distribution of θ provides a bound on the cumulative Bayes risk of prediction of the sequence of random variables for several choices of loss function. This same quantity is the minimax redundancy of universal data compression and the capacity of certain channels. General bounds for this mutual information are given. A special case concerns the estimation of binary-valued functions with Vapnik-Chervonenkis dimension d_{vc}, for which the information is bounded by d_{vc} log N. For smooth families of probability densities with a Euclidean parameter of dimension d, the information bound is (d/2) log N plus a constant. The prior density proportional to the square root of the Fisher information determinant is the unique continuous density that achieves a mutual information within o(1) of the capacity for large N. The Bayesian procedure with this prior is asymptotically minimax for the cumulative relative entropy risk.

Original language | English (US) |
---|---|

Title of host publication | Proceedings of the 1993 IEEE International Symposium on Information Theory |

Publisher | Publ by IEEE |

Number of pages | 1 |

ISBN (Print) | 0780308786 |

Publication status | Published - Jan 1 1993 |

Event | Proceedings of the 1993 IEEE International Symposium on Information Theory - San Antonio, TX, USA Duration: Jan 17 1993 → Jan 22 1993 |

### Publication series

Name | Proceedings of the 1993 IEEE International Symposium on Information Theory |
---|

### Other

Other | Proceedings of the 1993 IEEE International Symposium on Information Theory |
---|---|

City | San Antonio, TX, USA |

Period | 1/17/93 → 1/22/93 |

### Fingerprint

### ASJC Scopus subject areas

- Engineering(all)

### Cite this

*Proceedings of the 1993 IEEE International Symposium on Information Theory*(Proceedings of the 1993 IEEE International Symposium on Information Theory). Publ by IEEE.