Accounting for parametric uncertainty in Markov decision processes

Adam W. Schapaugh, Richard AJ Tyre

Research output: Contribution to journalArticle

6 Citations (Scopus)

Abstract

Markov decision processes have become the standard tool for modeling sequential decision-making problems in conservation. In many real-world applications, however, it is practically infeasible to accurately parameterize the state transition function. In this study, we introduce a new way of dealing with ambiguity in the state transition function. In contrast to existing methods, we explore the effects of uncertainty at the level of the policy, rather than at the level of decisions within states. We use information-gap decision theory to ask the question of how much uncertainty in the state transition function can be tolerated while still delivering a specified expected value given by the objective function. Accordingly, the goal of the optimization problem is no longer to maximize expected value, but to maximize local robustness to uncertainty (while still meeting the desired level of performance). We analyze a simple land acquisition problem, using info-gap decision theory to propagate uncertainties and rank alternative policies. Rather than requiring information about the extent of parameter uncertainty at the outset, info-gap addresses the question of how much uncertainty is permissible in the state transition function before the optimal policy would change.

Original languageEnglish (US)
Pages (from-to)15-21
Number of pages7
JournalEcological Modelling
Volume254
DOIs
StatePublished - Apr 10 2013

Fingerprint

decision making
modeling
decision
decision process
policy
land
effect
method
parameter

Keywords

  • Information-gap
  • Markov decision process
  • Reserve selection
  • Stochastic dynamic programming
  • Uncertainty

ASJC Scopus subject areas

  • Ecological Modeling

Cite this

Accounting for parametric uncertainty in Markov decision processes. / Schapaugh, Adam W.; Tyre, Richard AJ.

In: Ecological Modelling, Vol. 254, 10.04.2013, p. 15-21.

Research output: Contribution to journalArticle

@article{dec644ac8c0f417fa8cacbd8d1d77ac1,
title = "Accounting for parametric uncertainty in Markov decision processes",
abstract = "Markov decision processes have become the standard tool for modeling sequential decision-making problems in conservation. In many real-world applications, however, it is practically infeasible to accurately parameterize the state transition function. In this study, we introduce a new way of dealing with ambiguity in the state transition function. In contrast to existing methods, we explore the effects of uncertainty at the level of the policy, rather than at the level of decisions within states. We use information-gap decision theory to ask the question of how much uncertainty in the state transition function can be tolerated while still delivering a specified expected value given by the objective function. Accordingly, the goal of the optimization problem is no longer to maximize expected value, but to maximize local robustness to uncertainty (while still meeting the desired level of performance). We analyze a simple land acquisition problem, using info-gap decision theory to propagate uncertainties and rank alternative policies. Rather than requiring information about the extent of parameter uncertainty at the outset, info-gap addresses the question of how much uncertainty is permissible in the state transition function before the optimal policy would change.",
keywords = "Information-gap, Markov decision process, Reserve selection, Stochastic dynamic programming, Uncertainty",
author = "Schapaugh, {Adam W.} and Tyre, {Richard AJ}",
year = "2013",
month = "4",
day = "10",
doi = "10.1016/j.ecolmodel.2013.01.003",
language = "English (US)",
volume = "254",
pages = "15--21",
journal = "Ecological Modelling",
issn = "0304-3800",
publisher = "Elsevier",

}

TY - JOUR

T1 - Accounting for parametric uncertainty in Markov decision processes

AU - Schapaugh, Adam W.

AU - Tyre, Richard AJ

PY - 2013/4/10

Y1 - 2013/4/10

N2 - Markov decision processes have become the standard tool for modeling sequential decision-making problems in conservation. In many real-world applications, however, it is practically infeasible to accurately parameterize the state transition function. In this study, we introduce a new way of dealing with ambiguity in the state transition function. In contrast to existing methods, we explore the effects of uncertainty at the level of the policy, rather than at the level of decisions within states. We use information-gap decision theory to ask the question of how much uncertainty in the state transition function can be tolerated while still delivering a specified expected value given by the objective function. Accordingly, the goal of the optimization problem is no longer to maximize expected value, but to maximize local robustness to uncertainty (while still meeting the desired level of performance). We analyze a simple land acquisition problem, using info-gap decision theory to propagate uncertainties and rank alternative policies. Rather than requiring information about the extent of parameter uncertainty at the outset, info-gap addresses the question of how much uncertainty is permissible in the state transition function before the optimal policy would change.

AB - Markov decision processes have become the standard tool for modeling sequential decision-making problems in conservation. In many real-world applications, however, it is practically infeasible to accurately parameterize the state transition function. In this study, we introduce a new way of dealing with ambiguity in the state transition function. In contrast to existing methods, we explore the effects of uncertainty at the level of the policy, rather than at the level of decisions within states. We use information-gap decision theory to ask the question of how much uncertainty in the state transition function can be tolerated while still delivering a specified expected value given by the objective function. Accordingly, the goal of the optimization problem is no longer to maximize expected value, but to maximize local robustness to uncertainty (while still meeting the desired level of performance). We analyze a simple land acquisition problem, using info-gap decision theory to propagate uncertainties and rank alternative policies. Rather than requiring information about the extent of parameter uncertainty at the outset, info-gap addresses the question of how much uncertainty is permissible in the state transition function before the optimal policy would change.

KW - Information-gap

KW - Markov decision process

KW - Reserve selection

KW - Stochastic dynamic programming

KW - Uncertainty

UR - http://www.scopus.com/inward/record.url?scp=84873957215&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84873957215&partnerID=8YFLogxK

U2 - 10.1016/j.ecolmodel.2013.01.003

DO - 10.1016/j.ecolmodel.2013.01.003

M3 - Article

VL - 254

SP - 15

EP - 21

JO - Ecological Modelling

JF - Ecological Modelling

SN - 0304-3800

ER -