Toward Generalization of Automated Temporal Abstraction to Partially Observable Reinforcement Learning

Cilden E., POLAT F.

IEEE TRANSACTIONS ON CYBERNETICS, vol.45, no.8, pp.1414-1425, 2015 (SCI-Expanded) identifier identifier

  • Publication Type: Article / Article
  • Volume: 45 Issue: 8
  • Publication Date: 2015
  • Doi Number: 10.1109/tcyb.2014.2352038
  • Journal Name: IEEE TRANSACTIONS ON CYBERNETICS
  • Journal Indexes: Science Citation Index Expanded (SCI-EXPANDED), Scopus
  • Page Numbers: pp.1414-1425
  • Keywords: Learning abstractions, partially observable Markov decision process (POMDP), reinforcement learning (RL)
  • Middle East Technical University Affiliated: Yes

Abstract

Temporal abstraction for reinforcement learning (RL) aims to decrease learning time by making use of repeated sub-policy patterns in the learning task. Automatic extraction of abstractions during RL process is difficult but has many challenges such as dealing with the curse of dimensionality. Various studies have explored the subject under the assumption that the problem domain is fully observable by the learning agent. Learning abstractions for partially observable RL is a relatively less explored area. In this paper, we adapt an existing automatic abstraction method, namely extended sequence tree, originally designed for fully observable problems. The modified method covers a certain family of model-based partially observable RL settings. We also introduce belief state discretization methods that can be used with this new abstraction mechanism. The effectiveness of the proposed abstraction method is shown empirically by experimenting on well-known benchmark problems.