PAE/NX, software virtualization, and Linux guest hangs in VirtualBox

Gullible Jones · Oct 31, 2015

If you're like me and tend to put the cart before the horse, you may have tried running Linux VMs with PAE and VT-x, and found them to work great everywhere.

Likewise, you may have tried running Windows VMs with PAE and software virtualization only, and found them to work without a hitch.

So... That should work on Linux, right?

Wrong. There is a reason that VirtualBox PAE support is considered experimental. If your Linux guests have been running slow, crashing, or mysteriously hanging on boot, PAE might be why. If you want to use hardware NX support on a 32-bit VM, you need hardware virtualization.

Just FYI...

lotuseclat79 · Nov 1, 2015

Gullible Jones said: ↑

If you're like me and tend to put the cart before the horse, you may have tried running Linux VMs with PAE and VT-x, and found them to work great everywhere.

Likewise, you may have tried running Windows VMs with PAE and software virtualization only, and found them to work without a hitch.

So... That should work on Linux, right?

Wrong. There is a reason that VirtualBox PAE support is considered experimental. If your Linux guests have been running slow, crashing, or mysteriously hanging on boot, PAE might be why. If you want to use hardware NX support on a 32-bit VM, you need hardware virtualization.

Just FYI...
Click to expand...

Hi Gullible,

So, by "hardware virtualization" you mean a processor with VT-x, VT-d, VT-i, etc. in addition to PAE eh? IOW, you need to buy the processor (horse) according to all of its specific hardware virtualization support before trying out using the (cart) aka VirtualBox PAE.

-- Tom

Gullible Jones · Nov 1, 2015

@lotuseclat79 I've actually had some success virtualizing Win2k/XP on older laptops, without hardware virt. But... yeah, that's pretty much it. Hardware PAE/NX cannot be effectively made available to guest OSes without hardware VT-x etc. even though it can be enabled in the GUI for such guests. That ought to teach me to read manuals.

Speaking of which, I've been reading the VBox end-user and SDK manuals; and the remarks therein on VBox software virtualization are interesting...

10.5. Details about software virtualization

Implementing virtualization on x86 CPUs with no hardware virtualization support is an extraordinarily complex task because the CPU architecture was not designed to be virtualized. The problems can usually be solved, but at the cost of reduced performance. Thus, there is a constant clash between virtualization performance and accuracy.

The x86 instruction set was originally designed in the 1970s and underwent significant changes with the addition of protected mode in the 1980s with the 286 CPU architecture and then again with the Intel 386 and its 32-bit architecture. Whereas the 386 did have limited virtualization support for real mode operation (V86 mode, as used by the "DOS Box" of Windows 3.x and OS/2 2.x), no support was provided for virtualizing the entire architecture.

In theory, software virtualization is not overly complex. In addition to the four privilege levels ("rings") provided by the hardware (of which typically only two are used: ring 0 for kernel mode and ring 3 for user mode), one needs to differentiate between "host context" and "guest context".

In "host context", everything is as if no hypervisor was active. This might be the active mode if another application on your host has been scheduled CPU time; in that case, there is a host ring 3 mode and a host ring 0 mode. The hypervisor is not involved.

In "guest context", however, a virtual machine is active. So long as the guest code is running in ring 3, this is not much of a problem since a hypervisor can set up the page tables properly and run that code natively on the processor. The problems mostly lie in how to intercept what the guest's kernel does.

There are several possible solutions to these problems. One approach is full software emulation, usually involving recompilation. That is, all code to be run by the guest is analyzed, transformed into a form which will not allow the guest to either modify or see the true state of the CPU, and only then executed. This process is obviously highly complex and costly in terms of performance. (VirtualBox contains a recompiler based on QEMU which can be used for pure software emulation, but the recompiler is only activated in special situations, described below.)

Another possible solution is paravirtualization, in which only specially modified guest OSes are allowed to run. This way, most of the hardware access is abstracted and any functions which would normally access the hardware or privileged CPU state are passed on to the hypervisor instead. Paravirtualization can achieve good functionality and performance on standard x86 CPUs, but it can only work if the guest OS can actually be modified, which is obviously not always the case.

VirtualBox chooses a different approach. When starting a virtual machine, through its ring-0 support kernel driver, VirtualBox has set up the host system so that it can run most of the guest code natively, but it has inserted itself at the "bottom" of the picture. It can then assume control when needed -- if a privileged instruction is executed, the guest traps (in particular because an I/O register was accessed and a device needs to be virtualized) or external interrupts occur. VirtualBox may then handle this and either route a request to a virtual device or possibly delegate handling such things to the guest or host OS. In guest context, VirtualBox can therefore be in one of three states:

Guest ring 3 code is run unmodified, at full speed, as much as possible. The number of faults will generally be low (unless the guest allows port I/O from ring 3, something we cannot do as we don't want the guest to be able to access real ports). This is also referred to as "raw mode", as the guest ring-3 code runs unmodified.

For guest code in ring 0, VirtualBox employs a nasty trick: it actually reconfigures the guest so that its ring-0 code is run in ring 1 instead (which is normally not used in x86 operating systems). As a result, when guest ring-0 code (actually running in ring 1) such as a guest device driver attempts to write to an I/O register or execute a privileged instruction, the VirtualBox hypervisor in "real" ring 0 can take over.

The hypervisor (VMM) can be active. Every time a fault occurs, VirtualBox looks at the offending instruction and can relegate it to a virtual device or the host OS or the guest OS or run it in the recompiler.

In particular, the recompiler is used when guest code disables interrupts and VirtualBox cannot figure out when they will be switched back on (in these situations, VirtualBox actually analyzes the guest code using its own disassembler). Also, certain privileged instructions such as LIDT need to be handled specially. Finally, any real-mode or protected-mode code (e.g. BIOS code, a DOS guest, or any operating system startup) is run in the recompiler entirely.

Unfortunately this only works to a degree. Among others, the following situations require special handling:

Running ring 0 code in ring 1 causes a lot of additional instruction faults, as ring 1 is not allowed to execute any privileged instructions (of which guest's ring-0 contains plenty). With each of these faults, the VMM must step in and emulate the code to achieve the desired behavior. While this works, emulating thousands of these faults is very expensive and severely hurts the performance of the virtualized guest.

There are certain flaws in the implementation of ring 1 in the x86 architecture that were never fixed. Certain instructions that should trap in ring 1 don't. This affects, for example, the LGDT/SGDT, LIDT/SIDT, or POPF/PUSHF instruction pairs. Whereas the "load" operation is privileged and can therefore be trapped, the "store" instruction always succeed. If the guest is allowed to execute these, it will see the true state of the CPU, not the virtualized state. The CPUID instruction also has the same problem.

A hypervisor typically needs to reserve some portion of the guest's address space (both linear address space and selectors) for its own use. This is not entirely transparent to the guest OS and may cause clashes.

The SYSENTER instruction (used for system calls) executed by an application running in a guest OS always transitions to ring 0. But that is where the hypervisor runs, not the guest OS. In this case, the hypervisor must trap and emulate the instruction even when it is not desirable.

The CPU segment registers contain a "hidden" descriptor cache which is not software-accessible. The hypervisor cannot read, save, or restore this state, but the guest OS may use it.

Some resources must (and can) be trapped by the hypervisor, but the access is so frequent that this creates a significant performance overhead. An example is the TPR (Task Priority) register in 32-bit mode. Accesses to this register must be trapped by the hypervisor, but certain guest operating systems (notably Windows and Solaris) write this register very often, which adversely affects virtualization performance.

To fix these performance and security issues, VirtualBox contains a Code Scanning and Analysis Manager (CSAM), which disassembles guest code, and the Patch Manager (PATM), which can replace it at runtime.

Before executing ring 0 code, CSAM scans it recursively to discover problematic instructions. PATM then performs in-situ patching, i.e. it replaces the instruction with a jump to hypervisor memory where an integrated code generator has placed a more suitable implementation. In reality, this is a very complex task as there are lots of odd situations to be discovered and handled correctly. So, with its current complexity, one could argue that PATM is an advanced in-situ recompiler.

In addition, every time a fault occurs, VirtualBox analyzes the offending code to determine if it is possible to patch it in order to prevent it from causing more faults in the future. This approach works well in practice and dramatically improves software virtualization performance.
Click to expand...

I had figured that it was a full platform emulator optimized for x86 only, with the guest extensions providing something similar to VirtIO. But it is actually a good deal more complicated than that. I wonder about the dangers that might be posed by corner cases.

e.g. Could there be situations where the VBox driver might fail to recognize a dangerous instruction, and it would pass into ring 0 unmodified? That might allow arbitrary guest tampering with the host OS.

I'm thinking it might not be a good idea to trust Virtualbox VM security, even without guest extensions, in the absence of hardware virtualization capability.

(And in cases where that capability is available, it might be a good idea to constrain Virtualbox with AppArmor or such; or at least run it in its own user account. It does not sound like the userspace code is restricted all that much, with or without hardware virtualization.)

Log in or Sign up

PAE/NX, software virtualization, and Linux guest hangs in VirtualBox

Gullible Jones Registered Member

lotuseclat79 Registered Member

Gullible Jones Registered Member

Log in or Sign up

PAE/NX, software virtualization, and Linux guest hangs in VirtualBox

Gullible Jones Registered Member

lotuseclat79 Registered Member

Gullible Jones Registered Member

Useful Searches