Accuracy Barrier (ACCBAR): A novel performance indicator for binary classification

Canbek G., Taşkaya Temizel T., Sağiroğlu Ş.

15th International Conference on Information Security and Cryptography, ISCTURKEY 2022, Ankara, Türkiye, 19 - 20 Ekim 2022, ss.92-97 identifier

  • Yayın Türü: Bildiri / Tam Metin Bildiri
  • Doi Numarası: 10.1109/iscturkey56345.2022.9931888
  • Basıldığı Şehir: Ankara
  • Basıldığı Ülke: Türkiye
  • Sayfa Sayıları: ss.92-97
  • Anahtar Kelimeler: confirmation bias, performance evaluation, performance indicators, performance measures, publication bias
  • Orta Doğu Teknik Üniversitesi Adresli: Evet

Özet

© 2022 IEEE.Although several binary classification performance metrics have been defined, a few of them are used for performance evaluation of classifiers and performance comparison/reporting in the literature. Specifically, F1 and Accuracy (ACC) are the most known and conventionally used metrics. Despite their popularity and easy-to-understand characteristics, those metrics exhibit critical robustness issues. This paper suggests a new instrument category named 'performance indicators' and proposes a novel indicator named accuracy barrier (ACCBAR for short) that works to uncover confounding problems in performance reporting of ACC metric. The given case study in mobile malware classification, which is a domain of cyber security, has shown that the indicator gives an accurate interpretation of the results presented in terms of ACC. This study also recommends that researchers should use ACCBAR to eliminate potential publication or confirmation bias in classification performance evaluation.