Abstract:
Software is prone to contain bugs and vulnerabilities. To protect it, researchers normally
go to a lower layer, such as protecting the applications from the kernel or protecting the
operating systems from the hypervisor, because the upper layer is controlled and depends
on the lower layer. However, even a small hypervisor, which partitions the system hardware
resources into different domains to support and isolate multiple virtual machines, may
contain some vulnerabilities and is hard to protect within itself.
In this dissertation, we use a hardware-assisted method to monitor the integrity of
the software running on top it. We present HyperCheck, a hardware-assisted tampering
detection framework designed to protect the integrity of hypervisors or operating systems
(OS). HyperCheck leverages the CPU System Management Mode (SMM), present in x86
systems and a dedicated commercial network card, to securely generate and transmit the
full state of the protected machine to an external server. Using HyperCheck, we were able
to ferret-out rootkits that targeted the integrity of both the Xen hypervisor and traditional
OSes. Moreover, HyperCheck is robust against attacks that aim to disable or block its
operation. Our experimental results show that HyperCheck can produce and communicate
a scan of the state of the protected software in less than 40ms.
In addition to detecting the intrusion, another promising approach to protect the end
user's computer is to separate sensitive tasks, such as financial-related activities, from un-sensitive tasks. For this purpose, we designed a system which has two operating systems
installed: one trusted and the other untrusted. The trusted OS runs only the trusted
applications and is guaranteed to be separated from the untrusted OS. Without using a
hypervisor, we leverage the commercial hardware and the BIOS to enforce the isolation
between the two OSes. By utilizing the standard ACPI S3 sleep, we also achieve a short
delay when switching between the two OSes.
