Evaluating the Robustness of Vulnerability Detection Models under Semantic-Preserving Code Transformations

17. Ulusal Yazılım Mühendisliği Sempozyumu,, Muğla, Türkiye, 14 - 16 Mayıs 2026, ss.25, (Özet Bildiri)

Software vulnerability detection models are typically evaluated on original source code, while their robustness to

semantic-preserving code modifications remains largely unexplored. In this study, we evaluate the robustness

of vulnerability detection models under semantic-preserving code transformations using the SySeVR dataset,

covering arithmetic expression, API function call, array usage, and pointer usage categories. We consider

whitespace perturbation, dead code injection, and their sequential combination, all of which preserve program

semantics while altering surface-level code characteristics. The evaluation includes classical machine learning

models with TF-IDF representations (Random Forest, Support Vector Machine), deep learning models with

Word2Vec embeddings (BiLSTM, TextCNN), and large language models (CodeBERT, GraphCodeBERT, and

Qwen2.5-Coder). Model performance is assessed primarily using F1-score for detection effectiveness and training

time for efficiency analysis. The results show that although several models achieve strong performance on

original code, detection performance generally degrades under semantic-preserving transformations, with the

combined application of transformations causing the most pronounced impact. Training efficiency is also affected,

but the effect varies by transformation and model family. These findings highlight the need for robustness-aware

and efficiency-aware evaluation of vulnerability detection models beyond clean code settings.