Running VirtualBox/VMWare on Hardened Kernel

Running VirtualBox or Vmware on Hardened Kernel

In this article we'll take a look at how to run VirtualBox/Vmware on a hardened kernel, which provides various security features increasing the security of the entire system. The default installation of Linux kernel is not as secure as it can be, which is why there are various kernel patches that further secure the kernel and overall system. Existing solutions are presented below and differ mainly in their integration into the system and administration of their features.

There are various articles on the Internet, like Linux Kernel Security (SELinux vs AppArmor vs Grsecurity) that specify features and comparison of every security implementation. A table from the mentioned article is presented below, which clearly states that Grsecurity is the easiest to use, AppArmor is easy to understand and administer, while the SELinux provides most powerful access control mechanisms.


In this article we'll take a look at Grsecurity, which also includes PaX that provides hardening of user/kernel-space against memory corruption vulnerabilities and exploits. PaX is not developed by Grsecurity developers, but provides additional protections such as: marking stack as non-executable, marking program memory as non-writable, etc, which protects against buffer overflow vulnerabilities. PaX features in the kernel are presented in the picture below, where it's clearly visible that PaX provides enforcement of non-executable memory pages, provides address space layout randomization and miscellaneous hardening features.


Note that the best reference about all PaX/Grsecurity options can be found Grsecurity and PaX Configuration Options

First we have to install VirtualBox by using the virtualbox package. There's a higher change of a working VirtualBox if we use the virtualbox instead of virtualbox-bin package, but I've tested with both packages in the past and both seem to work after an initial hassle of setting everything up.

[code] # emerge app-emulation/virtualbox app-emulation/virtualbox-additions app-emulation/virtualbox-modules [/code]

PaX/Grsecurity Features

In order to be able to boot virtualization software like VirtualBox/Vmware, we have to disable some PaX/Grsecurity options. Note that I didn't have any problems with all of the options enabled when using Qemu/KVM, which runs without any problems despite having all of the PaX/Grsecurity features mentioned below enabled.

Prevent invalid userland pointer dereference (PAX_MEMORY_UDEREF)

This feature needs to be disabled in the kernel, otherwise the whole host operating system will crash and we won't be able to use VirtualBox/Vmware. This option uses x86 segmentation to create strict separation of user and kernel addresses. By disabling this option, a vulnerability or an exploit will be able to access data and execute code residing in the userland portion of the process address space. Therefore, many exploits gaining access to the kernel via vulnerability will be able to access and execute the code from userland process.


The options presented above control the miscellaneous memory protections and are described in PaX Quickstart. The table describing all of the options is also presented below for clarity and understanding of the article. The option that must be disabled is UDEREF, which is described as the second entry in the table below. Note that the option cannot be changed on a running kernel, which consequentially requires a kernel recompilation when disabling it.


Paging based non-executable pages (PAX_PAGEEXEC)

This option uses the paging algorithm for managing memory pages, rather than segmentation. This option enables virtual machines to be somewhat faster than if segmentation is used. I haven't extensively tested this, so if somebody has additional information regarding the performance impact of one or the other alternative, please add such observations to comments below.


Enforce non-executable kernel pages (PAX_KERNEXEC)

This feature needs to be disabled in the kernel, otherwise the whole host operating system will crash and we won't be able to use VirtualBox/Vmware. The option protects against injecting and executing code in the kernel itself.


The options presented above control the ability to enforce non-executable pages and are described in PaX Quickstart. The table of the options is also presented below for clarity and understanding of this article. In order to run virtualization technology the KERNEXEC needs to be disabled in the kernel (the last option in the table below). Note that the KERNEXEC cannot be controlled on a per ELF object with PaX flags and therefore cannot be changed while the kernel is running. In order to disable this option, the kernel needs to be recompiled.


Sanitize all freed memory/Sanitize kernel stack (PAX_MEMORY_SANITIZE/PAX_MEMORY_STACKLEAK)

This option instructs kernel to erase memory pages and slab objects as soon as they are freed, which reduces the amount of time the data is stored in them after they are no longer in use. Note that this option does not strictly need to be disabled in order for virtualization technology to work properly, but will speed up the VMs by 4%.


Note that VirtualBox/Vmware are running fine when both options presented above are enabled, so if you don't mind a little slowdown for increased security, you should enable both options.

Address Space Layout Randomization (PAX_ASLR)

This option randomizes parts of the program in memory, which makes it considerably harder for attackers to obtain the knowledge of certain addresses in the program. We can randomize the following areas of memory: top of task's kernel stack, top of task's userland stack, base address for mmap() requests, base address of main executables.


The options presented above control the address space layout randomization and are described in PaX Quickstart. From the table below, we can see that RANDMMAP and RANDUSTACK can be controlled on per ELF binaries when the kernel is already running. Therefore, both options can be enabled in the kernel. Note that RANDKSTACK needs to be disabled at all times, since it will not allow virtual machine to boot, but will instead display a critical error during booting.


In order to test the hypothesis, we need to enable the RANDMMAP and RANDUSTACK in the kernel. Then we can enable/disable those option on ELF binary basis. In order to do that, we must first determine whether SOFTMODE or NON-SOFTMODE is enabled in the kernel. On the picture below, we can see that “Support soft mode” is disabled, which effectively uses non-softmode.


With SOFTMODE enabled, PaX protections are not enforced by default, but can be turned on/off at runtime for every ELF binary – we have to instruct every binary to use PaX protections manually. Therefore, if we would like to enforce PaX protections by default on every binary, we have to use NON-SOFTMODE. With this mode, we can still disable certain PaX protections on binary ELF executables manually. When ELF is loaded into memory, the kernel will read PaX flags set on the binary and apply them upon execution of the binary.

If your virtualization software doesn't work, try experimenting with RANDMMAP/RANDUSTACK to see whether VirtualBox/Vmware starts working and booting the virtual machines properly. For completeness of the article let's take a look at the default PaX options in the kernel and PaX options for VirtualBox binary, which are presented on the picture below.



In this article we've seen the options that need to be enabled/disabled on PaX/Grsecurity enabled hardened kernel. I've spend a few days switching the kernel configuration options on and off and recompiling the kernel every time in order to figure out the right flags causing problems with VirtualBox/Vmware - therefore, quite some work has already been done in order to figure out the right kernel options. If you're found the article informative and useful, please respond with a comment below. If you're running VirtualBox/Vmware on PaX/Grsecurity hardened kernel, I would advise that you share your thoughts and experiences below in order to make this article a reference for future users trying to do the same thing.