Diacritics correction in Turkish with context-aware sequence to sequence modeling


Creative Commons License

Köksal A. T., Bozal Ö., ÖZGE U.

Turkish Journal of Electrical Engineering and Computer Sciences, vol.30, no.6, pp.2433-2445, 2022 (SCI-Expanded) identifier identifier identifier

  • Publication Type: Article / Article
  • Volume: 30 Issue: 6
  • Publication Date: 2022
  • Doi Number: 10.55730/1300-0632.3948
  • Journal Name: Turkish Journal of Electrical Engineering and Computer Sciences
  • Journal Indexes: Science Citation Index Expanded (SCI-EXPANDED), Scopus, Academic Search Premier, Applied Science & Technology Source, Compendex, Computer & Applied Sciences, INSPEC, TR DİZİN (ULAKBİM)
  • Page Numbers: pp.2433-2445
  • Keywords: Natural language processing, diacritics restoration, diacritics correction, sequence to sequence learning, LSTM, RESTORATION
  • Middle East Technical University Affiliated: Yes

Abstract

© TÜBITAK.Digital texts in many languages have examples of missing or misused diacritics which makes it hard for natural language processing applications to disambiguate the meaning of words. Therefore, diacritics restoration is a crucial step in natural language processing applications for many languages. In this study we approach this problem as bidirectional transformation of diacritical letters and their ASCII counterparts, rather than unidirectional diacritic restoration. We propose a context-aware character-level sequence to sequence model for this transformation. The model is language independent in the sense that no language-specific feature extraction is necessary other than the utilization of word embeddings and is directly applicable to other languages. We trained the model for Turkish diacritics correction task and for the assessment we used Turkish tweets benchmark dataset. Our best setting for the proposed model improves the state-of-the-art results in terms of F1 score by 4.7% on ambiguous words and 1.24% over all cases.