• bhyveContainerLinux

bhyve Container Linux Virtual Machines

Creating The Container Linux VM

Download the raw container linux image from CoreOS.

fetch https://stable.release.core-os.net/amd64-usr/current/coreos_production_image.bin.bz2
fetch https://stable.release.core-os.net/amd64-usr/current/coreos_production_image.bin.bz2.DIGESTS
fetch https://stable.release.core-os.net/amd64-usr/current/coreos_production_image.bin.bz2.DIGESTS.asc
fetch https://stable.release.core-os.net/amd64-usr/current/coreos_production_image.bin.bz2.DIGESTS.sig
fetch https://stable.release.core-os.net/amd64-usr/current/coreos_production_image.bin.bz2.sig

Probably good practice to Verify the image, not described here.

Optionally create a ZFS volumne for the disk drive

zfs create -V 32G -s -o reservation=none -o volmode=dev zroot/coreos

bzcat the image into your new drive, if no ZFS then simply un-bzip the file.

bzcat coreos_production_image.bin.bz2 > /dev/zvol/zroot/coreos

Optionally snapshot if you are using ZFS

zfs snap zroot/coreos@vanilla

At this point you can boot the container linux VM, but it will not be accessible.

bhyve -c 1 -m 4G -A -H -P \
  -s 0,hostbridge \
  -s 3,ahci-hd,/dev/zvol/zroot/coreos \
  -s 31,lpc \
  -l com1,stdio \
  -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
  coreos

Accessing the Container Linux VM

There are a couple of options here for accessing and configuring the VM:

1. Add ssh key(s) to ~core/.ssh/authorized_keys
2. Configure via hosted configuration

Either option can be accomplished by booting from Linux ISO.

Boot from Linux ISO

This example uses SystemRescueCD, I've also added a network interface so I can copy things over to the VM:

bhyve -c 1 -m 4G -A -H -P \
  -s 0,hostbridge \
  -s 2,ahci-cd,/usr/home/amarks/distfiles/systemrescuecd-x86-5.2.2.iso \
  -s 3,ahci-hd,/dev/zvol/zroot/coreos \
  -s 4,virtio-net,tap1 \
  -s 29,fbuf,w=1280,h=1024,tcp=0.0.0.0:5900 \
  -s 30,xhci,tablet \
  -s 31,lpc \
  -l com1,stdio \
  -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
  coreos

Adding ssh key(s)

This section is accomplished running from the Linux ISO.

From your live disk, mount the root partition

mount /dev/sda9 /media

Copy your public key(s) over to the VM and add them to ~core/.ssh/authorized_keys

Restart the VM without the live disc and ssh using public key authentication.

ssh -l core <ip address>

Hosting A Configuration

This section is accomplished from any host on the same network as the VM.

A web-server is required to host a configuration file. I'm using the same FreeBSD host system to host the Container Linux configuration.

pkg install nginx

Either write the configuration in json, or write it in a yml format then convert it to json using https://github.com/coreos/container-linux-config-transpiler.

YML:

passwd:
  users:
    - name: core
      ssh_authorized_keys:
        - "ssh-rsa AAAAB3NzaC1...

JSON:

{
    "ignition": {
        "config": {},
        "security": {
            "tls": {}
        },
        "timeouts": {},
        "version": "2.2.0"
    },
    "networkd": {},
    "passwd": {
        "users": [
            {
                "name": "core",
                "sshAuthorizedKeys": [
                    "ssh-rsa AAAA..."
                ]
            }
        ]
    },
    "storage": {},
    "systemd": {}
}

Configure Kernel Parameters for hosted configuration

This section is accomplished running from the Linux ISO.

To add kernel parameters to instruct container Linux to use the hosted configuration, add a grub.cfg to the OEM partition of the Container Linux VM.

ls -l /dev/disk/by-partlabel

lrwxrwxrwx 1 root root 10 Aug 19 13:42 BIOS-BOOT -> ../../sda2
lrwxrwxrwx 1 root root 10 Aug 19 13:42 EFI-SYSTEM -> ../../sda1
lrwxrwxrwx 1 root root 10 Aug 19 13:42 OEM -> ../../sda6
lrwxrwxrwx 1 root root 10 Aug 19 13:43 OEM-CONFIG -> ../../sda7
lrwxrwxrwx 1 root root 10 Aug 19 13:42 ROOT -> ../../sda9
lrwxrwxrwx 1 root root 10 Aug 19 13:42 USR-A -> ../../sda3
lrwxrwxrwx 1 root root 10 Aug 19 13:42 USR-B -> ../../sda4

mount /dev/sda6 /media

/media/grub.cfg

linux_append="coreos.autologin coreos.first_boot=1 coreos.config.url=http://<IP Address>/config.json"

Container Linux specific options are documented here: https://coreos.com/os/docs/latest/booting-with-pxe.html

With your web-server running and the hosted configuration in place, reboot the VM without the Linux Live ISO.

Where to go from here

1. Enable dockerd, either manually or via the hosted configuration, and you can use the docker client command on another machine to run containers on this VM.
2. Clone the ZFS disk and make clusters of Container Linux VMs running dockerd, swarm, rkt, flannel, kubernetes, etc.