Research Information System
Other, pp.1-17, 2022
As
malicious software gets more stealthy and smarter, software analysis has become
an essential part of malware detection. Modern malware does not immediately
display its malicious behavior, especially if they are aware that it is being
analyzed. For instance, malware can detect the runtime environment and use
certain triggers, such as time, to avoid detection. Static analysis fails on
obfuscated code whereas dynamic analysis struggles to find the right actions
and conditions to trigger malicious activity of software that can sense being
monitored. In this paper, we propose a behavior-based malware detection
methodology using API call sequence analysis based on dynamic symbolic
execution. We propose API function models with the symbolic execution engine to
extract possible call sequences of a given binary program; identify if there is
a malicious sequence even if it is hidden, and provide evidence by showing what
data values will reveal this malicious API sequence. Our experiments showed
that our methodology detects suspicious behavior hiding behind evasion
techniques and its applicability to a real malware.