VMM-based Process Shepherding
Haibo Chen,Pengcheng Liu,Rong Chen,Binyu Zang
2007-01-01
Abstract:Processes in commodity operating systems are “wild” 1 in nature: They are usually granted with excessive privileges, yet can be easily compromised and abused. Unfortunately, since commodity operating systems are big, complex, thus inherently untrusted, monitoring process behaviors within them is inherently insecure and could be circumvented or tampered. In this paper, we present an approach, named VMM-based process shepherding, to prevent, detect and isolate harmful behaviors (e.g. intrusions) of wild processes. The key idea of our approach is using a virtual machine monitor (VMM) to shepherd all privileged operations made by a wild process, in terms of system calls. As a VMM can easily be adjusted to intercept all system calls from a process running within operating systems thereon, such interception is mandatory and can not be bypassed. Further, as our approach is completely implemented in VMMs, it can result in good operating system transparency and portability. We provide three techniques as building blocks to process shepherding. First, the VMM prohibits any unauthorized accesses to privileged resources (e.g. system configuration files) using policy-based system call auditing. Second, the VMM uses system call sequences to detect possible malicious behaviors. Unexpected system call sequences should be considered as evidences for misbehaving. Third, the VMM isolates all suspect operations and discard them once a wild process is identified as malicious. Based on the three techniques, our approach tends to be safe and non-intrusive, that is, it can isolate damages made by a wild process. We present Shepherd, a prototype system based on Xen VMM, and evaluate it against real-life applications. According to our evaluation, Shepherd is resistant against several recent real-life attacks. Performance measurements show that our implementation incurs only a small amount of performance overhead.