Other, pp.1-17, 2022
As malicious software gets more stealthy and smarter, software analysis has become an essential part of malware detection. Modern malware does not immediately display its malicious behavior, especially if they are aware that it is being analyzed. For instance, malware can detect the runtime environment and use certain triggers, such as time, to avoid detection. Static analysis fails on obfuscated code whereas dynamic analysis struggles to find the right actions and conditions to trigger malicious activity of software that can sense being monitored. In this paper, we propose a behavior-based malware detection methodology using API call sequence analysis based on dynamic symbolic execution. We propose API function models with the symbolic execution engine to extract possible call sequences of a given binary program; identify if there is a malicious sequence even if it is hidden, and provide evidence by showing what data values will reveal this malicious API sequence. Our experiments showed that our methodology detects suspicious behavior hiding behind evasion techniques and its applicability to a real malware.