PCI Virtualisation Terminology

My last post was titled "Building qemu-dm for HVM VT-d PCI Pass-Through", a piece about building a component of Xen. But what does all this terminology mean?

Firstly, a very brief overview of the Xen architecture. The three main components of a system running Xen are the hypervisor, privileged domains and unprivileged domains. The hypervisor is run when the system first boots and controls access to the hardware. Privileged domains essentially provide both drivers and a management interface to xen. Usually there is one privileged domain, called dom0. They hypervisor by itself is not a functional system and generally the hypervisor + dom0 is required to boot the system. Unprivileged domains are the guest OSes on the system, often referred to as domU. As the name suggests they generally have to access hardware via the hypervisor - which may delegate the task to a privileged domain.

Xen supports two methods of domain (guest) virtualisation. In the beginning there were paravirtualised (PV) domains. In paravirtualised domains portions of the guest OS that execute privileged instructions are replaced by hypercalls. The hypercall asks the hypervisor to perform the relevant action. At the time that Xen was originally developed this was the only type of virtualisation that was possible on x86 and ia64.

With the advent of Intel's VT-x for x86 and VT-i for ia64, and AMD's AMD-V it has became possible to run unmodified operating systems as guests. This is called full-virtualisation or hardare virtual machine (HVM). It has obvious advantages when the guest operating sytem can't be modified.

There are several techniques used to give unprivileged domains access to PCI devices. For PV domains the common approach is to split the driver into front-end and back-end portions. The front-end driver sits in the unprivileged domain and the back-end driver sits in a privileged domain. The front-end driver communicates with the back-end driver. And the back-end driver communicates with the hardware. The back-end driver may multiplex IO from multiple front-end drivers. And thus a PCI device may be shared between domains.

Another approach used with PV domains is PCI pass-through. This basically involves giving a PCI device exclusively to an unprivileged domain. The advantage of this is a potential performance increase, as I/O no longer has to go through a back-end driver sitting in a privileged domain. The disadvantages of this approach are:

1. The PCI device in question can no longer be shared between domains
2. Hotplug needs to be used to achieve live migration - not all devices are supported
3. As the owner of a PCI card can potentially DMA main memory, passing through a PCI device to an unprivileged domain effectively makes the domain privileged

Systems that have Intel VT-d have IOMMU capabilities. The IOMMU constrains the area to with a device passed-through to a guest may DMA. Which removes the security problem relating to passing-through PCI devices to unprivileged guests. Xen allows pass-through to paravirtualised to optionally use the IOMMU if present.

For fully virtualised domains devices are commonly emulated. This is done by running the device emulation code from qemu as qemu-dm in a privileged domain. In order to improve I/O performance in fully virtualised domains it is possible to use PV drivers. These essentially provide front-end drivers to fully virtualised domains. And with the advent of Intel's VT-d it is possible to pass-through PCI devices to fully virtualised domains.