VMSPY: An Automatic Guest Virtual Machine System Function Hook and Control Program

Jin YU,Hao HUANG
DOI: https://doi.org/10.11897/SP.J.1016.2017.00414
2017-01-01
Abstract:How to effectively ensure the operation security of the guest virtual machine on a cloud computing platform is a hot topic at present.The system function hook and control method is one of the key technologies that are used to monitor the client system.Although the function hook and control method adopted in the security monitoring program based on the kernel interface of the operation system and the introspection program of the virtual machine based on the virtualization technology can meet the requirements of the security monitoring,the system still has some defects:the function hook can be easily bypassed;the system call hook method is single and limited;the internal function of the client application can't be hooked;the executing process of the function cannot be controlled;the security mechanism may result in extra large performance overhead.In this paper,an automatic function hook and control program of the guest virtual machine system based on virtualization technology (VMSPY) is proposed.The main functions of the modules are realized in the VMM,the codes of the guest system are analyzed automatically and generated dynamically via the disassembly engine.Besides,the privileged instruction sequence designed is inserted in a proper position to realize the system call hook.The internal function of the application is intercepted under the condition that it is not affected by the address space layout randomization (ASLR) technology.The code instruction sequence of the hooked function is simulated and executed automatically in the VMM according to the strategy to realize the control of the executing process of the system call and the application function.The privileged instruction sequence inserted in the guest system is protected through the memory page authorization mechanism to prevent the impact of the guest system.A cache mechanism is used to reduce the extra performance overhead as much as possible.
What problem does this paper attempt to address?