Unsupervised Meta-Testing With Conditional Neural Processes for Hybrid Meta-Reinforcement Learning
IEEE ROBOTICS AND AUTOMATION LETTERS, cilt.9, sa.10, ss.8427-8434, 2024 (SCI-Expanded, Scopus)
- Yayın Türü: Makale / Tam Makale
- Cilt numarası: 9 Sayı: 10
- Basım Tarihi: 2024
- Doi Numarası: 10.1109/lra.2024.3443496
- Dergi Adı: IEEE ROBOTICS AND AUTOMATION LETTERS
- Derginin Tarandığı İndeksler: Science Citation Index Expanded (SCI-EXPANDED), Scopus
- Sayfa Sayıları: ss.8427-8434
- Açık Arşiv Koleksiyonu: AVESİS Açık Erişim Koleksiyonu
- Orta Doğu Teknik Üniversitesi Adresli: Hayır
Özet
We introduce Unsupervised Meta-Testing with Conditional Neural Processes (UMCNP), a novel hybrid few-shot meta-reinforcement learning (meta-RL) method that uniquely combines, yet distinctly separates, parameterized policy gradient-based (PPG) and task inference-based few-shot meta-RL. Tailored for settings where the reward signal is missing during meta-testing, our method increases sample efficiency without requiring additional samples in meta-training. UMCNP leverages the efficiency and scalability of Conditional Neural Processes (CNPs) to reduce the number of online interactions required in meta-testing. During meta-training, samples previously collected through PPG meta-RL are efficiently reused for learning task inference in an offline manner. UMCNP infers the latent representation of the transition dynamics model from a single test task rollout with unknown parameters. This approach allows us to generate rollouts for self-adaptation by interacting with the learned dynamics model. We demonstrate our method can adapt to an unseen test task using significantly fewer samples during meta-testing than the baselines in 2D-Point Agent and continuous control meta-RL benchmarks, namely, cartpole with unknown angle sensor bias, walker agent with randomized dynamics parameters.