Semi-Markov decision problems and performance sensitivity analysis

Xi Ren Cao

doi:10.3182/20020721-6-es-1901.00511

Semi-Markov decision problems and performance sensitivity analysis

Xi Ren Cao^*

^*Corresponding author for this work

Department of Electronic and Computer Engineering

Research output: Chapter in Book/Conference Proceeding/Report › Conference Paper published in a book › peer-review

2 Citations (Scopus)

Abstract

We extend the results about performance potentials, perturbation realization matrices, policy iteration of Markov decision processes, etc., to semi-Markov processes (SMPs). Starting with the concept of perturbation realization, we define a realization matrix and prove that it satisfies the Lyapunov equation. From the realization matrix we define a performance potential and prove that it satisfies the Poisson equation. Sensitivity formulas and policy iteration algorithms of Semi-Markov decision process (SMDPs) can be derived. The performance sensitivities can be obtained and policy iteration of SMDPs can be implemeted on a single sample path of the SMPs.

Original language	English
Title of host publication	IFAC Proceedings Volumes (IFAC-PapersOnline)
Editors	Gabriel Ferrate, Eduardo F. Camacho, Luis Basanez, Juan. A. de la Puente
Publisher	IFAC Secretariat
Pages	139-143
Number of pages	5
Edition	1
ISBN (Print)	9783902661746
DOIs	https://doi.org/10.3182/20020721-6-es-1901.00511
Publication status	Published - 2002
Event	15th World Congress of the International Federation of Automatic Control, 2002 - Barcelona, Spain Duration: 21 Jul 2002 → 26 Jul 2002

Publication series

Name	IFAC Proceedings Volumes (IFAC-PapersOnline)
Number	1
Volume	15
ISSN (Print)	1474-6670

Conference

Conference	15th World Congress of the International Federation of Automatic Control, 2002
Country/Territory	Spain
City	Barcelona
Period	21/07/02 → 26/07/02

Bibliographical note

Publisher Copyright:
Copyright © 2002 IFAC.

Keywords

Lyapunov equations
Perturbation analysis
Poisson equations
Policy iteration
Potentials

Access to Document

10.3182/20020721-6-es-1901.00511

Cite this

@inproceedings{79a259b3c33d427083bb85ceccbcdcdf,

title = "Semi-Markov decision problems and performance sensitivity analysis",

abstract = "We extend the results about performance potentials, perturbation realization matrices, policy iteration of Markov decision processes, etc., to semi-Markov processes (SMPs). Starting with the concept of perturbation realization, we define a realization matrix and prove that it satisfies the Lyapunov equation. From the realization matrix we define a performance potential and prove that it satisfies the Poisson equation. Sensitivity formulas and policy iteration algorithms of Semi-Markov decision process (SMDPs) can be derived. The performance sensitivities can be obtained and policy iteration of SMDPs can be implemeted on a single sample path of the SMPs.",

keywords = "Lyapunov equations, Perturbation analysis, Poisson equations, Policy iteration, Potentials",

author = "Cao, \{Xi Ren\}",

note = "Publisher Copyright: Copyright {\textcopyright} 2002 IFAC.; 15th World Congress of the International Federation of Automatic Control, 2002 ; Conference date: 21-07-2002 Through 26-07-2002",

year = "2002",

doi = "10.3182/20020721-6-es-1901.00511",

language = "English",

isbn = "9783902661746",

series = "IFAC Proceedings Volumes (IFAC-PapersOnline)",

publisher = "IFAC Secretariat",

number = "1",

pages = "139--143",

editor = "Gabriel Ferrate and Camacho, \{Eduardo F.\} and Luis Basanez and \{de la Puente\}, \{Juan. A.\}",

booktitle = "IFAC Proceedings Volumes (IFAC-PapersOnline)",

edition = "1",

}

Cao, XR 2002, Semi-Markov decision problems and performance sensitivity analysis. in G Ferrate, EF Camacho, L Basanez & JA de la Puente (eds), IFAC Proceedings Volumes (IFAC-PapersOnline). 1 edn, IFAC Proceedings Volumes (IFAC-PapersOnline), no. 1, vol. 15, IFAC Secretariat, pp. 139-143, 15th World Congress of the International Federation of Automatic Control, 2002, Barcelona, Spain, 21/07/02. https://doi.org/10.3182/20020721-6-es-1901.00511

Semi-Markov decision problems and performance sensitivity analysis. / Cao, Xi Ren.
IFAC Proceedings Volumes (IFAC-PapersOnline). ed. / Gabriel Ferrate; Eduardo F. Camacho; Luis Basanez; Juan. A. de la Puente. 1. ed. IFAC Secretariat, 2002. p. 139-143 (IFAC Proceedings Volumes (IFAC-PapersOnline); Vol. 15, No. 1).

Research output: Chapter in Book/Conference Proceeding/Report › Conference Paper published in a book › peer-review

TY - GEN

T1 - Semi-Markov decision problems and performance sensitivity analysis

AU - Cao, Xi Ren

PY - 2002

Y1 - 2002

N2 - We extend the results about performance potentials, perturbation realization matrices, policy iteration of Markov decision processes, etc., to semi-Markov processes (SMPs). Starting with the concept of perturbation realization, we define a realization matrix and prove that it satisfies the Lyapunov equation. From the realization matrix we define a performance potential and prove that it satisfies the Poisson equation. Sensitivity formulas and policy iteration algorithms of Semi-Markov decision process (SMDPs) can be derived. The performance sensitivities can be obtained and policy iteration of SMDPs can be implemeted on a single sample path of the SMPs.

AB - We extend the results about performance potentials, perturbation realization matrices, policy iteration of Markov decision processes, etc., to semi-Markov processes (SMPs). Starting with the concept of perturbation realization, we define a realization matrix and prove that it satisfies the Lyapunov equation. From the realization matrix we define a performance potential and prove that it satisfies the Poisson equation. Sensitivity formulas and policy iteration algorithms of Semi-Markov decision process (SMDPs) can be derived. The performance sensitivities can be obtained and policy iteration of SMDPs can be implemeted on a single sample path of the SMPs.

KW - Lyapunov equations

KW - Perturbation analysis

KW - Poisson equations

KW - Policy iteration

KW - Potentials

UR - https://www.scopus.com/pages/publications/84945547755

U2 - 10.3182/20020721-6-es-1901.00511

DO - 10.3182/20020721-6-es-1901.00511

M3 - Conference Paper published in a book

AN - SCOPUS:84945547755

SN - 9783902661746

T3 - IFAC Proceedings Volumes (IFAC-PapersOnline)

SP - 139

EP - 143

BT - IFAC Proceedings Volumes (IFAC-PapersOnline)

A2 - Ferrate, Gabriel

A2 - Camacho, Eduardo F.

A2 - Basanez, Luis

A2 - de la Puente, Juan. A.

PB - IFAC Secretariat

T2 - 15th World Congress of the International Federation of Automatic Control, 2002

Y2 - 21 July 2002 through 26 July 2002

ER -

Semi-Markov decision problems and performance sensitivity analysis

Abstract

Publication series

Conference

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Cite this