Toward Generalization of Automated Temporal Abstraction to Partially Observable Reinforcement Learning

Cilden E., POLAT F.

IEEE TRANSACTIONS ON CYBERNETICS, cilt.45, sa.8, ss.1414-1425, 2015 (SCI-Expanded) identifier identifier

  • Yayın Türü: Makale / Tam Makale
  • Cilt numarası: 45 Sayı: 8
  • Basım Tarihi: 2015
  • Doi Numarası: 10.1109/tcyb.2014.2352038
  • Dergi Adı: IEEE TRANSACTIONS ON CYBERNETICS
  • Derginin Tarandığı İndeksler: Science Citation Index Expanded (SCI-EXPANDED), Scopus
  • Sayfa Sayıları: ss.1414-1425
  • Anahtar Kelimeler: Learning abstractions, partially observable Markov decision process (POMDP), reinforcement learning (RL)
  • Orta Doğu Teknik Üniversitesi Adresli: Evet

Özet

Temporal abstraction for reinforcement learning (RL) aims to decrease learning time by making use of repeated sub-policy patterns in the learning task. Automatic extraction of abstractions during RL process is difficult but has many challenges such as dealing with the curse of dimensionality. Various studies have explored the subject under the assumption that the problem domain is fully observable by the learning agent. Learning abstractions for partially observable RL is a relatively less explored area. In this paper, we adapt an existing automatic abstraction method, namely extended sequence tree, originally designed for fully observable problems. The modified method covers a certain family of model-based partially observable RL settings. We also introduce belief state discretization methods that can be used with this new abstraction mechanism. The effectiveness of the proposed abstraction method is shown empirically by experimenting on well-known benchmark problems.