Mutual information and feature importance gradient boosting: automatic byte n-gram feature reranking for Android malware detection

The fast pace evolving of Android malware demands for highly efficient strategy. That is, for a range of malware types, a malware detection scheme needs to be resilient and with minimum computation performs efficient and precise. In this paper, we propose Mutual Information and Feature Importance Gradient Boosting (MIFIBoost) tool that uses byte n-gram frequency. MIFIBoost consists of four steps in the model construction phase and two steps in the prediction phase. For training, first, n-grams 2 ≤ n ≤ 4 of both the classes.dex and AndroidManifest.xml binary files are obtained. Then, MIFIBoost uses Mutual Information (MI) to determine the top most informative items from the entire n-gram vocabulary. In the third phase, MIFIBoost utilizes the Gradient Boosting algorithm to re-rank these top n-grams. For testing, MIFIBoost uses the learned vocabulary of byte n-grams term-frequency (tf) to feed into the classifier for prediction. Thus, MIFIBoost does not require reverse engineering. A key insight from this work is that filtering using XGBoost helps us to address the hard problem of detecting obfuscated malware better while having a negligible impact on nonobfuscated malware. We have conducted a wide range of experiments on four different datasets one of which is obfuscated, and MIFIBoost outperforms state-of-the-art tools. MIFIBoost's f1-score for Drebin, DexShare, and AMD datasets is 99.1%, 98.87%, and 99.62%, respectively, a False Positive Rate of 0.41% using AMD dataset. On average, the False Negative Rate of MIFIBoost is 2.1% for the PRAGuard dataset in which seven different obfuscation techniques are implemented. In addition to fast run-time performance and resiliency against obfuscated malware, the experiments show that MIFIBoost performs quite efficiently for five zero-day families with 99.78% AUC.