PCI device passthrough

Industrial PCs with IOMMU virtualization functions allow physical PCI devices to be explicitly assigned to a virtual machine (see table: Device support for TwinCAT/BSD Hypervisor, device and GPU passthrough.). PCI devices such as the GPU, network interfaces or USB controllers can be explicitly assigned to a virtual machine as passthru devices.

To assign a PCI device to a virtual machine, its PCI address is needed first. The command pciconf -l lists all PCI devices and their addresses.

$ pciconf -l
...
vgapci0@pci0:0:2:0: class=0x030000 rev=0x00 hdr=0x00 vendor=0x8086 device=0x3e92 subvendor=0x8086 subdevice=0x2212
xhci0@pci0:0:20:0: class=0x0c0330 rev=0x10 hdr=0x00 vendor=0x8086 device=0xa36d subvendor=0x8086 subdevice=0x7270
igb0@pci0:1:0:0: class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x1533 subvendor=0x8086 subdevice=0x1533
...

In the following sample, the three listed devices are to be assigned to a virtual machine.

To isolate the devices from the TwinCAT/BSD host, devices with devctl are assigned the ppt (PCI PassThrough) driver.

doas devctl set driver -f pci0:0:2:0 ppt
doas devctl set driver -f pci0:0:20:0 ppt
doas devctl set driver -f pci0:1:0:0 ppt

To set the drivers already at system boot the PCI addresses can be added as pptdevs to the /boot/loader.conf file:

pptdevs="1/0/0 0/2/0 0/20/0"

A new output of pciconf -l now shows that the ppt drivers have been assigned to the devices:

$ pciconf -l
...
ppt0@pci0:0:2:0: class=0x030000 rev=0x00 hdr=0x00 vendor=0x8086 device=0x3e92 subvendor=0x8086 subdevice=0x2212
ppt1@pci0:0:20:0: class=0x0c0330 rev=0x10 hdr=0x00 vendor=0x8086 device=0xa36d subvendor=0x8086 subdevice=0x7270
ppt2@pci0:1:0:0: class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x1533 subvendor=0x8086 subdevice=0x1533
...

The PCI devices can now be passed to bhyve with the parameter -s [slot],passthru,[slot/bus/function]. The values for [slot/bus/function] refer to the PCI addresses of the pciconf -l output. Since passthrough devices use fixed memory addresses, byhve must also be passed the flag -S as a parameter to disable memory swapping for the process.

The bhyve call with assigned on-board GPU, Ethernet and USB controller results for the sample as follows:

doas bhyve \
-c sockets=1,cores=2,threads=1 \
-m 4G \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_UEFI.fd,fwcfg=qemu \
-s 0:0,hostbridge \
-s 2:0,passthru,0/2/0 \
-s 3:0,passthru,0/20/0 \
-s 10:0,nvme,/vms/samplevm/disk0.img \
-s 20:0,passthru,1/0/0 \
-s 31:0,lpc \
-A -H -P -S \
samplevm

Extended parameters for the pass-through of the integrated GPU

The pass-through of the integrated GPU on supported IPCs requires extended bhyve parameters so that the EFI firmware BHYVE_BHF_UEFI.fd can correctly initialize the integrated graphics card during the boot process of the virtual machine and make it available to the guest operating system.

doas bhyve \
-c sockets=1,cores=2,threads=1 \
-m 4G \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_UEFI.fd,fwcfg=qemu \
-s 0:0,hostbridge \
-s 2:0,passthru,0/2/0,rom=/vms/samplevm/gop.rom \
-s 3:0,passthru,0/20/0 \
-s 10:0,nvme,/vms/samplevm/disk0.img \
-s 20:0,passthru,1/0/0 \
-s 31:0,lpc \
-o pci.0.31.0.pcireg.vendor=host \
-o pci.0.31.0.pcireg.device=host \
-o pci.0.31.0.pcireg.revid=host \
-o pci.0.31.0.pcireg.subvendor=host \
-o pci.0.31.0.pcireg.subdevice=host \
-A -H -P -S \
samplevm

The passthru parameter for the integrated graphics card ( -s 2:0,passthru,0/2/0 ) is extended by ,rom=/vms/samplevm/gop.rom.
The file gop.rom is required for the initialization of the graphics card by the EFI and can be created with the following command:

doas gop-dump /vms/samplevm/gop.rom

In addition, as shown above, the PCI registers of the emulated LPC bridge must be set to the value host when calling bhyve:

-o pci.0.31.0.pcireg.vendor=host \
-o pci.0.31.0.pcireg.device=host \
-o pci.0.31.0.pcireg.revid=host \
-o pci.0.31.0.pcireg.subvendor=host \
-o pci.0.31.0.pcireg.subdevice=host \