A basic formula for online policy gradient algorithms

Xi Ren Cao

doi:10.1109/TAC.2005.847037

A basic formula for online policy gradient algorithms

Xi Ren Cao^*

^*Corresponding author for this work

Department of Electronic and Computer Engineering

Research output: Contribution to journal › Journal Article › peer-review

37 Citations (Scopus)

Abstract

This note presents a (new) basic formula for sample-path-based estimates for performance gradients for Markov systems (called policy gradients in reinforcement learning literature). With this basic formula, many policy-gradient algorithms, including those that have previously appeared in the literature, can be easily developed. The formula follows naturally from a sensitivity equation in perturbation analysis. New research direction is discussed.

Original language	English
Pages (from-to)	696-699
Number of pages	4
Journal	IEEE Transactions on Automatic Control
Volume	50
Issue number	5
DOIs	https://doi.org/10.1109/TAC.2005.847037
Publication status	Published - May 2005

Keywords

Markov decision processes
Online estimation
Perturbation analysis (PA)
Perturbation realization
Poisson equations
Potentials
Reinforcement learning

Access to Document

10.1109/TAC.2005.847037

Cite this

@article{3a1d9eebab614f6fbda5d22dc91b3bdb,

title = "A basic formula for online policy gradient algorithms",

abstract = "This note presents a (new) basic formula for sample-path-based estimates for performance gradients for Markov systems (called policy gradients in reinforcement learning literature). With this basic formula, many policy-gradient algorithms, including those that have previously appeared in the literature, can be easily developed. The formula follows naturally from a sensitivity equation in perturbation analysis. New research direction is discussed.",

keywords = "Markov decision processes, Online estimation, Perturbation analysis (PA), Perturbation realization, Poisson equations, Potentials, Reinforcement learning",

author = "Cao, \{Xi Ren\}",

year = "2005",

month = may,

doi = "10.1109/TAC.2005.847037",

language = "English",

volume = "50",

pages = "696--699",

journal = "IEEE Transactions on Automatic Control",

issn = "0018-9286",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

number = "5",

}

TY - JOUR

T1 - A basic formula for online policy gradient algorithms

AU - Cao, Xi Ren

PY - 2005/5

Y1 - 2005/5

N2 - This note presents a (new) basic formula for sample-path-based estimates for performance gradients for Markov systems (called policy gradients in reinforcement learning literature). With this basic formula, many policy-gradient algorithms, including those that have previously appeared in the literature, can be easily developed. The formula follows naturally from a sensitivity equation in perturbation analysis. New research direction is discussed.

AB - This note presents a (new) basic formula for sample-path-based estimates for performance gradients for Markov systems (called policy gradients in reinforcement learning literature). With this basic formula, many policy-gradient algorithms, including those that have previously appeared in the literature, can be easily developed. The formula follows naturally from a sensitivity equation in perturbation analysis. New research direction is discussed.

KW - Markov decision processes

KW - Online estimation

KW - Perturbation analysis (PA)

KW - Perturbation realization

KW - Poisson equations

KW - Potentials

KW - Reinforcement learning

UR - https://www.webofscience.com/wos/woscc/full-record/WOS:000229085100018

UR - https://openalex.org/W2096899507

UR - https://www.scopus.com/pages/publications/20344361949

U2 - 10.1109/TAC.2005.847037

DO - 10.1109/TAC.2005.847037

M3 - Journal Article

SN - 0018-9286

VL - 50

SP - 696

EP - 699

JO - IEEE Transactions on Automatic Control

JF - IEEE Transactions on Automatic Control

IS - 5

ER -

A basic formula for online policy gradient algorithms

Abstract

Keywords

Access to Document

Other files and links

Fingerprint

Cite this