Abstraction in Model Based Partially Observable Reinforcement Learning using Extended Sequence Trees

Cilden E., POLAT F.

11th IEEE/WIC/ACM International Joint Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT), Macau, China, 4 - 07 December 2012, pp.348-355 identifier identifier

  • Publication Type: Conference Paper / Full Text
  • Doi Number: 10.1109/wi-iat.2012.161
  • City: Macau
  • Country: China
  • Page Numbers: pp.348-355
  • Middle East Technical University Affiliated: Yes


Extended sequence tree is a direct method for automatic generation of useful abstractions in reinforcement learning, designed for problems that can be modelled as Markov decision process. This paper proposes a method to expand the extended sequence tree method over reinforcement learning to cover partial observability formalized via partially observable Markov decision process through belief state formalism. This expansion requires a reasonable approximation of information state. Inspired by statistical ranking, a simple but effective discretization schema over belief state space is defined. Extended sequence tree method is modified to make use of this schema under partial observability, and effectiveness of resulting algorithm is shown by experiments on some benchmark problems.