DETECTING MALICIOUS API CALL SEQUENCES IN BINARY PROGRAMS USING DYNAMIC SYMBOLIC EXECUTION


Tatar F. T. , Betin Can A.

Other, pp.1-17, 2022

  • Publication Type: Other Publication / Other
  • Publication Date: 2022
  • Page Numbers: pp.1-17
  • Middle East Technical University Affiliated: Yes

Abstract

As malicious software gets more stealthy and smarter, software analysis has become an essential part of malware detection. Modern malware does not immediately display its malicious behavior, especially if they are aware that it is being analyzed. For instance, malware can detect the runtime environment and use certain triggers, such as time, to avoid detection. Static analysis fails on obfuscated code whereas dynamic analysis struggles to find the right actions and conditions to trigger malicious activity of software that can sense being monitored. In this paper, we propose a behavior-based malware detection methodology using API call sequence analysis based on dynamic symbolic execution. We propose API function models with the symbolic execution engine to extract possible call sequences of a given binary program; identify if there is a malicious sequence even if it is hidden, and provide evidence by showing what data values ​​will reveal this malicious API sequence. Our experiments showed that our methodology detects suspicious behavior hiding behind evasion techniques and its applicability to a real malware.