Information conversion, effective samples, and parameter size

Xiaodong Lin, Jennifer L Clarke, Bertrand S Clarke

Research output: Contribution to journalArticle

5 Citations (Scopus)

Abstract

Consider the relative entropy between a posterior density for a parameter given a sample and a second posterior density for the same parameter, based on a different model and a different data set. Then the relative entropy can be minimized over the second sample to get a virtual sample that would make the second posterior as close as possible to the first in an informational sense. If the first posterior is based on a dependent dataset and the second posterior uses an independence model, the effective inferential power of the dependent sample is transferred into the independent sample by the optimization. Examples of this optimization are presented for models with nuisance parameters, finite mixture models, and models for correlated data. Our approach is also used to choose the effective parameter size in a Bayesian hierarchical model.

Original languageEnglish (US)
Pages (from-to)4438-4456
Number of pages19
JournalIEEE Transactions on Information Theory
Volume53
Issue number12
DOIs
StatePublished - Dec 1 2007

Fingerprint

entropy
Entropy

Keywords

  • Asymptotic relative efficiency
  • Number of parameters
  • Relative entropy
  • Sample size

ASJC Scopus subject areas

  • Information Systems
  • Computer Science Applications
  • Library and Information Sciences

Cite this

Information conversion, effective samples, and parameter size. / Lin, Xiaodong; Clarke, Jennifer L; Clarke, Bertrand S.

In: IEEE Transactions on Information Theory, Vol. 53, No. 12, 01.12.2007, p. 4438-4456.

Research output: Contribution to journalArticle

Lin, Xiaodong ; Clarke, Jennifer L ; Clarke, Bertrand S. / Information conversion, effective samples, and parameter size. In: IEEE Transactions on Information Theory. 2007 ; Vol. 53, No. 12. pp. 4438-4456.
@article{d8549711296447e89eda7f869901394c,
title = "Information conversion, effective samples, and parameter size",
abstract = "Consider the relative entropy between a posterior density for a parameter given a sample and a second posterior density for the same parameter, based on a different model and a different data set. Then the relative entropy can be minimized over the second sample to get a virtual sample that would make the second posterior as close as possible to the first in an informational sense. If the first posterior is based on a dependent dataset and the second posterior uses an independence model, the effective inferential power of the dependent sample is transferred into the independent sample by the optimization. Examples of this optimization are presented for models with nuisance parameters, finite mixture models, and models for correlated data. Our approach is also used to choose the effective parameter size in a Bayesian hierarchical model.",
keywords = "Asymptotic relative efficiency, Number of parameters, Relative entropy, Sample size",
author = "Xiaodong Lin and Clarke, {Jennifer L} and Clarke, {Bertrand S}",
year = "2007",
month = "12",
day = "1",
doi = "10.1109/TIT.2007.909168",
language = "English (US)",
volume = "53",
pages = "4438--4456",
journal = "IEEE Transactions on Information Theory",
issn = "0018-9448",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
number = "12",

}

TY - JOUR

T1 - Information conversion, effective samples, and parameter size

AU - Lin, Xiaodong

AU - Clarke, Jennifer L

AU - Clarke, Bertrand S

PY - 2007/12/1

Y1 - 2007/12/1

N2 - Consider the relative entropy between a posterior density for a parameter given a sample and a second posterior density for the same parameter, based on a different model and a different data set. Then the relative entropy can be minimized over the second sample to get a virtual sample that would make the second posterior as close as possible to the first in an informational sense. If the first posterior is based on a dependent dataset and the second posterior uses an independence model, the effective inferential power of the dependent sample is transferred into the independent sample by the optimization. Examples of this optimization are presented for models with nuisance parameters, finite mixture models, and models for correlated data. Our approach is also used to choose the effective parameter size in a Bayesian hierarchical model.

AB - Consider the relative entropy between a posterior density for a parameter given a sample and a second posterior density for the same parameter, based on a different model and a different data set. Then the relative entropy can be minimized over the second sample to get a virtual sample that would make the second posterior as close as possible to the first in an informational sense. If the first posterior is based on a dependent dataset and the second posterior uses an independence model, the effective inferential power of the dependent sample is transferred into the independent sample by the optimization. Examples of this optimization are presented for models with nuisance parameters, finite mixture models, and models for correlated data. Our approach is also used to choose the effective parameter size in a Bayesian hierarchical model.

KW - Asymptotic relative efficiency

KW - Number of parameters

KW - Relative entropy

KW - Sample size

UR - http://www.scopus.com/inward/record.url?scp=64649091408&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=64649091408&partnerID=8YFLogxK

U2 - 10.1109/TIT.2007.909168

DO - 10.1109/TIT.2007.909168

M3 - Article

VL - 53

SP - 4438

EP - 4456

JO - IEEE Transactions on Information Theory

JF - IEEE Transactions on Information Theory

SN - 0018-9448

IS - 12

ER -