Named entity recognition in Turkish: A comparative study with detailed error analysis

Ozcelik O., Toraman Ç.

INFORMATION PROCESSING & MANAGEMENT, cilt.59, sa.6, 2022 (SCI-Expanded) identifier identifier

  • Yayın Türü: Makale / Tam Makale
  • Cilt numarası: 59 Sayı: 6
  • Basım Tarihi: 2022
  • Doi Numarası: 10.1016/j.ipm.2022.103065
  • Dergi Adı: INFORMATION PROCESSING & MANAGEMENT
  • Derginin Tarandığı İndeksler: Science Citation Index Expanded (SCI-EXPANDED), Social Sciences Citation Index (SSCI), Scopus, FRANCIS, Periodicals Index Online, ABI/INFORM, Applied Science & Technology Source, Business Source Elite, Business Source Premier, Communication Abstracts, Computer & Applied Sciences, EBSCO Education Source, Education Abstracts, Information Science and Technology Abstracts, INSPEC, Library and Information Science Abstracts, Library Literature and Information Science, Library, Information Science & Technology Abstracts (LISTA), Linguistics & Language Behavior Abstracts, MLA - Modern Language Association Database, zbMATH
  • Orta Doğu Teknik Üniversitesi Adresli: Hayır

Özet

Named entity recognition aims to detect pre-determined entity types in unstructured text. There is a limited number of studies on this task for low-resource languages such as Turkish. We provide a comprehensive study for Turkish named entity recognition by comparing the performances of existing state-of-the-art models on the datasets with varying domains to understand their generalization capability and further analyze why such models fail or succeed in this task. Our experimental results, supported by statistical tests, show that the highest weighted F1 scores are obtained by Transformer-based language models, varying from 80.8% in tweets to 96.1% in news articles. We find that Transformer-based language models are more robust to entity types with a small sample size and longer named entities compared to traditional models, yet all models have poor performance for longer named entities in social media. Moreover, when we shuffle 80% of words in a sentence to imitate flexible word order in Turkish, we observe more performance deterioration, 12% in well-written texts, compared to 7% in noisy text.