bhyve VM usage in Jenkins cluster

Contents

bhyve VM usage in Jenkins cluster

Installation of FreeBSD into bhyve VM for use in Jenkins cluster

Make sure that in /boot/loader.conf you have:

# Include vmm driver for bhyve
vmm_load="YES"
# Load nmdm driver to communicate with bhyve VM's
nmdm_load="YES"
# For bhyve VM's networking
if_bridge_load="YES"
bridgestp_load="YES"
if_tap_load="YES"

Obtain a release ISO image from FreeBSD download site.
Create a ZFS dataset which will hold the VM
```
   zfs create zroot/vm
```
Set ZFS property so that dataset will export ZVOL's as block devices only, and not try to probe them with GEOM:
```
   zfs set volmode=dev zroot/vm
   zfs get volmode zroot/vm
```
Create a 200G zvol:
```
   zfs create -V 200G zroot/vm/disk.img
```

Boot the release ISO image and install it into the disk image:

 sh /usr/share/examples/bhyve/vmrun.sh -i -I FreeBSD-9.2-RELEASE-amd64-disc1.iso -d /dev/zvol/zroot/vm/disk.img -m 2G vm_test

Do the initial installation, using disk.img as the backing hard drive

Edit /etc/ttys and add this line:

 ttyu0   "/usr/libexec/getty std.9600"   vt100   on secure

or this, if the guest is running 11-CURRENT newer than r264175

 ttyu0  "/usr/libexec/getty 3wire"  vt100  on  secure

shutdown the VM

Reboot the VM without the ISO image:

 sh /usr/share/examples/bhyve/vmrun.sh -d disk.img -m 2G vm_test

Set up basic networking inside the VM by editing /etc/rc.conf
Reboot the VM.
Install necessary packages such as devel/jenkins

Configure tap devices for bhyve VM's

On the host machine which is running the VM's, pick a network interface which will be used in the bridge. This example will use igb0.

Add the following to /etc/rc.conf on the host machine:

#####################################################
# Create tap devices, one tap interface per bhyve VM.
# Add the tap interfaces to bridge0
####################################################
cloned_interfaces="bridge0 tap0 tap1"

autobridge_interfaces="bridge0"
autobridge_bridge0="tap* igb0"

Each VM should have a separate tap device, all connected to the same bridge. If you add more VM's, remember to add another tap device to cloned_interfaces

Add the following to /etc/sysctl.conf on the host machine:

# bhyve needs this for tap interfaces
net.link.tap.user_open=1
net.link.tap.up_on_open=1

Refer to: http://www.freebsd.org/doc/handbook/network-bridging.html

Accessing console of VM

Type the following to see the list of VM's running:
```
 ls /dev/vmm/*
```
The bhyve VM's are configured to send their console output to the nmdm(4) nullmodem driver. Type the following to see the list of nullmodem devices in use:
```
 ls /dev/nmdm*
```
Each nmdm(4) has two endpoints A and B. You must connect to the B endpoint using the cu utility. For example:
```
 cu -l /dev/nmdm0B
```
To disconnect, type:
```
 ~..
```

Startup scripts

The scripts used to start the bhyve VM's are located at Github: https://github.com/freebsd/freebsd-ci/tree/master/vm

Configure rc.d scripts to boot VM

Read this: README for VM scripts