Malicious code detection using penalized splines on OPcode frequency

Recently, malicious software are gaining exponential growth due to the innumerable obfuscations of extended x86 IA-32 (OPcodes) that are being employed to evade from traditional detection methods. In this paper, we design a novel distinguisher to separate malware from benign that combines Multivariate Logistic Regression model using kernel HS in Penalized Splines along with OPcode frequency feature selection technique for efficiently detecting obfuscated malware. The main advantage of our penalized splines based feature selection technique is its performance capability achieved through the efficient filtering and identification of the most important OPcodes used in the obfuscation of malware. This is demonstrated through our successful implementation and experimental results of our proposed model on large malware datasets. The presented approach is effective at identifying previously examined malware and non-malware to assist in reverse engineering.