Cross-lingual few-shot sign language recognition

Bilge Y. C., Ikizler-Cinbis N., CİNBİŞ R. G.

Pattern Recognition, cilt.151, 2024 (SCI-Expanded) identifier identifier

  • Yayın Türü: Makale / Tam Makale
  • Cilt numarası: 151
  • Basım Tarihi: 2024
  • Doi Numarası: 10.1016/j.patcog.2024.110374
  • Dergi Adı: Pattern Recognition
  • Derginin Tarandığı İndeksler: Science Citation Index Expanded (SCI-EXPANDED), Scopus, Academic Search Premier, PASCAL, Applied Science & Technology Source, BIOSIS, Compendex, Computer & Applied Sciences, INSPEC, MLA - Modern Language Association Database, zbMATH
  • Anahtar Kelimeler: Cross-lingual sign language recognition, Few-shot learning, Sign language recognition
  • Orta Doğu Teknik Üniversitesi Adresli: Evet

Özet

There are over 150 sign languages worldwide, each with numerous local variants and thousands of signs. However, collecting annotated data for each sign language to train a model is a laborious and expert-dependent task. To address this issue, this paper introduces the problem of few-shot sign language recognition (FSSLR) in a cross-lingual setting. The central motivation is to be able to recognize a novel sign, even if it belongs to a sign language unseen during training, based on a small set of examples. To tackle this problem, we propose a novel embedding-based framework that first extracts a spatio-temporal visual representation based on video and hand features, as well as hand landmark estimates. To establish a comprehensive test bed, we propose three meta-learning FSSLR benchmarks that span multiple languages, and extensively evaluate the proposed framework. The experimental results demonstrate the effectiveness and superiority of the proposed approach for few-shot sign language recognition in both monolingual and cross-lingual settings.