Positive impact of state similarity on reinforcement learning performance

Girgin S., Polat F., Alhaj R.

IEEE TRANSACTIONS ON SYSTEMS MAN AND CYBERNETICS PART B-CYBERNETICS, cilt.37, sa.5, ss.1256-1270, 2007 (SCI-Expanded) identifier identifier identifier

  • Yayın Türü: Makale / Tam Makale
  • Cilt numarası: 37 Sayı: 5
  • Basım Tarihi: 2007
  • Doi Numarası: 10.1109/tsmcb.2007.899419
  • Dergi Adı: IEEE TRANSACTIONS ON SYSTEMS MAN AND CYBERNETICS PART B-CYBERNETICS
  • Derginin Tarandığı İndeksler: Science Citation Index Expanded (SCI-EXPANDED), Scopus
  • Sayfa Sayıları: ss.1256-1270
  • Anahtar Kelimeler: action-value function, learning performance, optimal policies, reinforcement learning (RL), similarity function, state similarity
  • Orta Doğu Teknik Üniversitesi Adresli: Evet

Özet

In this paper, we propose a novel approach to identify states with similar subpolicies and show how they can be integrated into the reinforcement learning framework to improve learning performance. The method utilizes a specialized tree structure to identify common action sequences of states, which are derived from possible optimal policies, and defines a similarity function between two states based on the number of such sequences. Using this similarity function, updates on the action-value function of a state are reflected onto all similar states. This allows experience that is acquired during learning to be applied to a broader context. The effectiveness of the method is demonstrated empirically.