A simple method for dealing with large state spaces

Adam W. Schapaugh, Andrew J. Tyre

Research output: Contribution to journalArticle

4 Citations (Scopus)

Abstract

1. Most sequential decision-making problems in conservation can be viewed conceptually and modelled as a Markov decision process. The goal in this context is to construct a policy that associates each state of the system with a particular action. This policy should offer optimal performance in the sense of maximizing or minimizing a specified conservation objective 2. Dynamic programming algorithms rely on explicit enumeration to derive the optimal policy. This is problematic from a computational perspective as the size of the state space grows exponentially with the number of state variables. 3. We present a state aggregation method where the idea is to capture the most important aspects of the original Markov decision process, find an optimal policy over this reduced space and use this as an approximate solution to the original problem. 4. Applying the aggregation method to a species reintroduction problem, we demonstrate how we were able to reduce the number of states by 75% and reduce the size of the transition matrices by almost 94% (324 vs. 5184), and the abstract action matched the optimal action more than 86% of the time. 5. We conclude that the aggregation method is not a panacea for the curse of dimensionality, but it does advance our ability to construct approximately optimal policies in systems with large state spaces.

Original languageEnglish (US)
Pages (from-to)949-957
Number of pages9
JournalMethods in Ecology and Evolution
Volume3
Issue number6
DOIs
StatePublished - Dec 1 2012

Fingerprint

species reintroduction
methodology
dynamic programming
decision making
method
policy
matrix
decision process

Keywords

  • Abstraction
  • Curse of dimensionality
  • Markov decision process
  • State space
  • Stochastic dynamic programming

ASJC Scopus subject areas

  • Ecology, Evolution, Behavior and Systematics
  • Ecological Modeling

Cite this

A simple method for dealing with large state spaces. / Schapaugh, Adam W.; Tyre, Andrew J.

In: Methods in Ecology and Evolution, Vol. 3, No. 6, 01.12.2012, p. 949-957.

Research output: Contribution to journalArticle

@article{dbf991abc8244038afdba3c0cf050888,
title = "A simple method for dealing with large state spaces",
abstract = "1. Most sequential decision-making problems in conservation can be viewed conceptually and modelled as a Markov decision process. The goal in this context is to construct a policy that associates each state of the system with a particular action. This policy should offer optimal performance in the sense of maximizing or minimizing a specified conservation objective 2. Dynamic programming algorithms rely on explicit enumeration to derive the optimal policy. This is problematic from a computational perspective as the size of the state space grows exponentially with the number of state variables. 3. We present a state aggregation method where the idea is to capture the most important aspects of the original Markov decision process, find an optimal policy over this reduced space and use this as an approximate solution to the original problem. 4. Applying the aggregation method to a species reintroduction problem, we demonstrate how we were able to reduce the number of states by 75{\%} and reduce the size of the transition matrices by almost 94{\%} (324 vs. 5184), and the abstract action matched the optimal action more than 86{\%} of the time. 5. We conclude that the aggregation method is not a panacea for the curse of dimensionality, but it does advance our ability to construct approximately optimal policies in systems with large state spaces.",
keywords = "Abstraction, Curse of dimensionality, Markov decision process, State space, Stochastic dynamic programming",
author = "Schapaugh, {Adam W.} and Tyre, {Andrew J.}",
year = "2012",
month = "12",
day = "1",
doi = "10.1111/j.2041-210X.2012.00242.x",
language = "English (US)",
volume = "3",
pages = "949--957",
journal = "Methods in Ecology and Evolution",
issn = "2041-210X",
publisher = "John Wiley and Sons Inc.",
number = "6",

}

TY - JOUR

T1 - A simple method for dealing with large state spaces

AU - Schapaugh, Adam W.

AU - Tyre, Andrew J.

PY - 2012/12/1

Y1 - 2012/12/1

N2 - 1. Most sequential decision-making problems in conservation can be viewed conceptually and modelled as a Markov decision process. The goal in this context is to construct a policy that associates each state of the system with a particular action. This policy should offer optimal performance in the sense of maximizing or minimizing a specified conservation objective 2. Dynamic programming algorithms rely on explicit enumeration to derive the optimal policy. This is problematic from a computational perspective as the size of the state space grows exponentially with the number of state variables. 3. We present a state aggregation method where the idea is to capture the most important aspects of the original Markov decision process, find an optimal policy over this reduced space and use this as an approximate solution to the original problem. 4. Applying the aggregation method to a species reintroduction problem, we demonstrate how we were able to reduce the number of states by 75% and reduce the size of the transition matrices by almost 94% (324 vs. 5184), and the abstract action matched the optimal action more than 86% of the time. 5. We conclude that the aggregation method is not a panacea for the curse of dimensionality, but it does advance our ability to construct approximately optimal policies in systems with large state spaces.

AB - 1. Most sequential decision-making problems in conservation can be viewed conceptually and modelled as a Markov decision process. The goal in this context is to construct a policy that associates each state of the system with a particular action. This policy should offer optimal performance in the sense of maximizing or minimizing a specified conservation objective 2. Dynamic programming algorithms rely on explicit enumeration to derive the optimal policy. This is problematic from a computational perspective as the size of the state space grows exponentially with the number of state variables. 3. We present a state aggregation method where the idea is to capture the most important aspects of the original Markov decision process, find an optimal policy over this reduced space and use this as an approximate solution to the original problem. 4. Applying the aggregation method to a species reintroduction problem, we demonstrate how we were able to reduce the number of states by 75% and reduce the size of the transition matrices by almost 94% (324 vs. 5184), and the abstract action matched the optimal action more than 86% of the time. 5. We conclude that the aggregation method is not a panacea for the curse of dimensionality, but it does advance our ability to construct approximately optimal policies in systems with large state spaces.

KW - Abstraction

KW - Curse of dimensionality

KW - Markov decision process

KW - State space

KW - Stochastic dynamic programming

UR - http://www.scopus.com/inward/record.url?scp=84871014380&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84871014380&partnerID=8YFLogxK

U2 - 10.1111/j.2041-210X.2012.00242.x

DO - 10.1111/j.2041-210X.2012.00242.x

M3 - Article

AN - SCOPUS:84871014380

VL - 3

SP - 949

EP - 957

JO - Methods in Ecology and Evolution

JF - Methods in Ecology and Evolution

SN - 2041-210X

IS - 6

ER -