Abstraction in Model Based Partially Observable Reinforcement Learning using Extended Sequence Trees

Cilden E., POLAT F.

11th IEEE/WIC/ACM International Joint Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT), Macau, Çin, 4 - 07 Aralık 2012, ss.348-355 identifier identifier

  • Yayın Türü: Bildiri / Tam Metin Bildiri
  • Doi Numarası: 10.1109/wi-iat.2012.161
  • Basıldığı Şehir: Macau
  • Basıldığı Ülke: Çin
  • Sayfa Sayıları: ss.348-355
  • Orta Doğu Teknik Üniversitesi Adresli: Evet

Özet

Extended sequence tree is a direct method for automatic generation of useful abstractions in reinforcement learning, designed for problems that can be modelled as Markov decision process. This paper proposes a method to expand the extended sequence tree method over reinforcement learning to cover partial observability formalized via partially observable Markov decision process through belief state formalism. This expansion requires a reasonable approximation of information state. Inspired by statistical ranking, a simple but effective discretization schema over belief state space is defined. Extended sequence tree method is modified to make use of this schema under partial observability, and effectiveness of resulting algorithm is shown by experiments on some benchmark problems.