Start and manage virtual machines

Virtual machines are started and managed with the programs bhyve and bhyvectl. Before a virtual machine can be started with bhyve, the kernel module vmm.ko must be loaded:

doas kldload -n vmm.ko

So that this step does not have to be repeated after each restart, the kernel module can already be loaded during the system startup of TwinCAT/BSD by setting vmm_load="YES" in the /boot/loader.conf:

doas sysrc -f /boot/loader.conf vmm_load="YES"

Once the kernel module is loaded, a virtual machine can be started by calling bhyve:

bhyve [OPTIONS] <vm_instance>

The parameters [OPTIONS] determine the configuration of the virtual machine, which can be used, for example, to specify the number of virtual CPUs used, the size of the main memory or the storage location of local files. The last parameter <vm_instance> of the call specifies the virtual machine instance name.

Start VM instance with simple basic configuration

A UEFI-based virtual machine with two virtual CPUs and 2 GB of main memory can be started with the following command:

doas bhyve \
-c sockets=1,cores=2,threads=1 \
-m 2G \
-s 0,hostbridge \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_UEFI.fd \
-l com1,stdio \
-A -H -P \
samplevm

In simplified terms, the basic configuration of the VM instance is as follows:

Start and manage virtual machines 1: — Basic configuration of a VM instance.

As soon as the Bhyve process is started with the parameters, the virtual machine with the instance name samplevm is available. Bhyve starts the UEFI firmware stored as bootrom. The UEFI shell is then output via the virtual interface com1 and redirected to the standard streams of the TwinCAT/BSD host via the parameter stdio, so that the UEFI shell is displayed on the command line:

UEFI Interactive Shell v2.2
EDK II
UEFI v2.70 (BHYVE, 0x00010000)
map: No mapping found.
Press ESC in 1 seconds to skip startup.nsh or any other key to continue.
Shell>

By entering reset -s in the UEFI shell, the virtual machine can be shut down again. The Bhyve process is then terminated with the return value 1. To restart the virtual machine, the complete Bhyve command must be called with the same parameters.

Explanation of the parameters

The meaning of the individual parameters can be called with bhyve -h. Detailed descriptions of the parameters can be found in the manual for bhyve. Alternatively, the manual can be called up via the command line with the command man bhyve.

The parameters used in the sample above are briefly explained below.

Parameter	Description
`-c sockets=1,cores=2, threads=1`	Configuration of the virtual CPU topology. In this sample, a CPU socket with two cores and one thread per core.
`-m 2G`	Main memory available to the virtual machine. In this sample 2 GB.
`-s 0,hostbridge`	A virtual host bridge to connect the virtual CPU to the virtual PCI bus. By convention, the host bridge should always be configured at PCI address -s 0:0:0 (-s 0 for short).
`-s 31,lcp`	LPC/PCI ISA bridge for connecting emulated LPC devices. By convention, the LPC/PCI ISA bridge should always be configured at PCI address -s 0:31:0 (-s 31 for short).
`-l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_UEFI.fd`	An emulated bootrom on the LPC bus. The UEFI firmware is passed as ROM in the file `/usr/local/share/uefi-firmware/BHYVE_BHF_UEFI.fd`.
`-l com1,stdio`	A serial interface on the LPC bus whose inputs and outputs are redirected to bhyve's standard input and output streams.
`-A`	Creates bhyve ACPI tables for the virtual machine.
`-H`	Releases the virtual CPU thread when an HLT instruction is detected. Otherwise, the virtual CPUs will use 100% of the host CPUs.
`-P`	Forces the virtual guest CPU to terminate when a PAUSE instruction is detected.

Parameters starting with -s are used to configure virtual PCI slots to which in turn emulated PCI devices can be assigned (for examples see: Advanced VM configuration). Parameters starting with -l are used to configure emulated LPC devices behind the LPC/PCI-ISA bridge.

Manage virtual machines

Started virtual machines are listed as Bhyve processes on the TwinCAT/BSD host. Accordingly, running virtual machines can be listed with ps(1):

ps -a | grep bhyve

7048 0 SC 0:31.06 bhyve: samplevm (bhyve)
7642 0 SC 0:01.83 bhyve: debian11 (bhyve)

Running virtual machines can be shut down or terminated via the TwinCAT/BSD host by sending signals via kill(1) to the respective Bhyve process. The TERM signal can be used to send an ACPI shutdown request to the virtual machine to trigger the virtual machine shutdown:

doas kill -s TERM $(pgrep -f "bhyve: samplevm")

If the virtual machine does not respond to ACPI shutdown requests, the KILL signal can be used to terminate the Bhyve process directly:

doas kill -s KILL $(pgrep -f "bhyve: samplevm")

After a Bhyve process has been terminated, its return value (exit code) can be queried by the shell variable $?:

echo $?

Return values greater than 1 indicate that the virtual machine could not be shut down properly. If the virtual machine is to be restarted or the configuration of a VM instance has been changed between bhyve calls, the VM instance must first be removed via bhyvectl:

doas bhyvectl --vm=samplevm --destroy

Virtual machine instances are listed as device files at /dev/vmm and can be further managed using bhyvectl. Via ls -al /dev/vmm it can also be determined which virtual machines are currently created on the TwinCAT/BSD host:

ls /dev/vmm

samplevm