Out-VM monitoring for Malicious Network Packet Detection in cloud

Preeti Mishra, Emmanuel S. Pilli, Vijay Varadharajan, Udaya Tupakula

Research output: Chapter in Book/Report/Conference proceedingConference proceeding contributionpeer-review

23 Citations (Scopus)


Cloud security is one of the biggest challenge in today's technological world. Researchers have proposed some solutions for cloud security. Virtual Machine (VM)-level solutions are configured and controlled at VM. They are less robust and can be easily subverted by attackers. In this paper, we propose an out-VM monitoring security approach named as Malicious Network Packet Detection (MNPD) which monitors the VMs from outside at both network and virtualization layer in cloud environment. MNPD performs the behavioral analysis of network traffic at Cloud Networking Server (CNS); providing primary defense from intrusions at network level. MNPD does the VM traffic validation at hypervisor of Cloud Compute Server (CCoS) to detect spoofing attacks, originated from VMs. The non-spoofed packets are further analyzed using behavior analysis of network traffic to detect any abnormality in the virtual traffic; providing second level of defense from intrusions at virtualization level. MNPD employs statistical learning technique (Random Forest) with ensemble of feature selection approach to learn the behavior of traffic patterns. MNPD does not involve overhead incurred in monitoring extensive memory writes or instruction-level traces. It is a more secure solution to detect attacks which never pass through physical interface and hence not detected by traditional IDS. The proposed approach has been validated with latest datasets (UNSW-NB and ITOC) and results seem to be promising.

Original languageEnglish
Title of host publicationISEASP 2017
Subtitle of host publicationProceedings of the 2017 ISEA Asia Security and Privacy Conference
Place of PublicationPiscataway, NJ
PublisherInstitute of Electrical and Electronics Engineers (IEEE)
Number of pages10
ISBN (Electronic)9781509059423
ISBN (Print)9781509059430
Publication statusPublished - 12 Jul 2017
Event2017 ISEA Asia Security and Privacy Conference, ISEASP 2017 - Surat, India
Duration: 29 Jan 20171 Feb 2017


Other2017 ISEA Asia Security and Privacy Conference, ISEASP 2017


Dive into the research topics of 'Out-VM monitoring for Malicious Network Packet Detection in cloud'. Together they form a unique fingerprint.

Cite this