Dynamic state-based security architecture for detecting security attacks in virtual machines

Udaya Kiran Tupakula, Vijay Varadharajan*

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

5 Citations (Scopus)


Current Internet environment is vulnerable to a range of different types of attacks, and furthermore, new types of attacks are being discovered on a daily basis. In this paper, we address the design of comprehensive intrusion detection for virtual-machine-based systems. We propose a novel security architecture using a virtual machine monitor-based intrusion detection system called Virtual machine Intrusion deteCTOR (VICTOR), which takes into account the specific characteristics of operating system and applications running in each virtual machine at a fine granular level to detect attacks. The components of our architecture are designed to deal with different types of malicious behaviour. The entity validation component is used for capturing information of the operating system and applications running in the virtual machines, secure logging and detection of attacks that are generated with spoofed source address. The intrusion detection engine component is used for the detection of known attacks and suspicious behaviour of the entities by monitoring the incoming and outgoing traffic of virtual machines. The dynamic analyser is used for detection and validation of hidden processes, detection of zero-day attacks and fine granular isolation of malicious process that is generating the attack traffic. After a zero-day attack is detected, interactive virtual machine technique is used to determine whether the zero-day attack exhibits polymorphic or metamorphic behaviour and develop attack signatures to deal with the attack. We have analysed our architecture with different types of attacks such as hidden processes and attacks such as Slammer. In this paper, we illustrate the operation of our system architecture by considering Slammer worm attack in detail.

Original languageEnglish
Pages (from-to)397-409
Number of pages13
JournalComputer Journal
Issue number4
Publication statusPublished - Apr 2012


Dive into the research topics of 'Dynamic state-based security architecture for detecting security attacks in virtual machines'. Together they form a unique fingerprint.

Cite this