Zero-day malware detection based on supervised learning algorithms of API call signatures

Mamoun Alazab, Sitalakshmi Venkatraman, Paul Watters, Moutaz Alazab

Research output: Chapter in Book/Report/Conference proceedingConference proceeding contributionpeer-review

86 Citations (Scopus)
56 Downloads (Pure)


Zero-day or unknown malware are created using code obfuscation techniques that can modify the parent code to produce offspring copies which have the same functionality but with different signatures. Current techniques reported in literature lack the capability of detecting zero-day malware with the required accuracy and efficiency. In this paper, we have proposed and evaluated a novel method of employing several data mining techniques to detect and classify zero-day malware with high levels of accuracy and efficiency based on the frequency of Windows API calls. This paper describes the methodology employed for the collection of large data sets to train the classifiers, and analyses the performance results of the various data mining algorithms adopted for the study using a fully automated tool developed in this research to conduct the various experimental investigations and evaluation. Through the performance results of these algorithms from our experimental analysis, we are able to evaluate and discuss the advantages of one data mining algorithm over the other for accurately detecting zero-day malware successfully. The data mining framework employed in this research learns through analysing the behavior of existing malicious and benign codes in large datasets. We have employed robust classifiers, namely Naïve Bayes (NB) Algorithm, k-Nearest Neighbor (kNN) Algorithm, Sequential Minimal Optimization (SMO) Algorithm with 4 differents kernels (SMO - Normalized PolyKernel, SMO - PolyKernel, SMO - Puk, and SMO- Radial Basis Function (RBF)), Backpropagation Neural Networks Algorithm, and J48 decision tree and have evaluated their performance. Overall, the automated data mining system implemented for this study has achieved high true positive (TP) rate of more than 98.5%, and low false positive (FP) rate of less than 0.025, which has not been achieved in literature so far. This is much higher than the required commercial acceptance level indicating that our novel technique is a major leap forward in detecting zero-day malware. This paper also offers future directions for researchers in exploring different aspects of obfuscations that are affecting the IT world today.

Original languageEnglish
Title of host publication9th Australasian Data Mining Conference, AusDM 2011
EditorsPeter Vamplew, Andrew Stranieri, Kok-Leong Ong, Peter Christen, Paul J. Kennedy
Place of PublicationSydney, NSW
PublisherAustralian Computer Society
Number of pages12
ISBN (Print)9781921770029
Publication statusPublished - Dec 2010
Externally publishedYes
Event9th Australasian Data Mining Conference, AusDM - 2011 - Ballarat, Australia
Duration: 1 Dec 20112 Dec 2011

Publication series

NameInformation Technology Series


Other9th Australasian Data Mining Conference, AusDM - 2011

Bibliographical note

Copyright © 2011, Australian Computer Society, Inc. This paper appeared at the 9th Australasian Data Mining Conference (AusDM 2011), Ballarat, Australia. Conferences in Research and Practice in Information Technology (CRPIT), Vol. 121, Peter Vamplew, Andrew Stranieri, Kok-Leong Ong, Peter Christen and Paul Kennedy, Eds. Reproduction for academic, not-for profit purposes permitted provided this text is included.


Dive into the research topics of 'Zero-day malware detection based on supervised learning algorithms of API call signatures'. Together they form a unique fingerprint.

Cite this