Package Building for the FreeBSD.org Cluster
The Big Picture
The basic idea is that we need to make certain that when port upgrades happen on cluster machines, we need to be able to somewhat reasonably guarantee that the ports being upgraded are compatible with the existing ports, and nothing is missed. poudriere handles this quite nicely and reliably.
Poudriere
Local Setup
The cluster package build repository is separate from the default clusteradm subversion repository. It is located at: svn+ssh://pb00.FreeBSD.org/var/svn/pkgbuild/keg
This repository contains the /usr/local/etc/poudriere.d/ directory on pb00.FreeBSD.org.
This directory contains all the magic needed for FreeBSD.org cluster machine package builds as noted in the Local Configuration and Local Scripts sections.
As we already know, pre-built packages were never one-size-fits-all. What I have tended to do when setting up such package build systems is to do package builds based on either a per-machine basis, or based on a set of machine roles. Both of these have their pros and cons:
- With per-role builds, time spent building packages is reduced, however you lose flexibility with machine-independent configuration.
- With per-machine builds, more time is spent building (potentially duplicate) packages, however each can have independent configurations.
My preference has always been per-machine package builds.
Local Configurations
/usr/local/etc/poudriere.d/ is set up according to what poudriere expects to find, and where it will look. These files are all under revision control, for the inevitable case of needing to look back to see what changed with a port build options, or needing to revert an entire set of changed options.
Each build has its own:
options/ directory
make.conf file
pkglist.txt file
These are linked to a particular build by prefixing the jail name and target machine name, for example:
10-amd64-pb00-make.conf
10-amd64-pb00-options/
10-amd64-pb00-pkglist.txt
In this case, 10-amd64 is the name of the jail used for the port build and pb00 is the FreeBSD.org cluster machine that will be the consumer of the resulting packages.
The idea for jail names is effectively ${BRANCH_VERSION}-${ARCH}
The idea for the "sets" poudriere will build is effectively the machine hostname, less "freebsd.org". In cases where there are multiple machines per site, or a machine hostname is site specific, the "." is replaced with "-", for example: cook-isc
Local Scripts
There are two scripts right now to help automate the maintenance of package builds:
clusterbuild.sh: Runs the package build for all configured systems.
regen.sh: Updates the ports tree and, if TERM is set, runs make config for the ports in the pkglist.txt files (and dependencies).
There are a few configuration files that are used by the scripts:
jails.conf: List of jails to map to build "sets"
sets-${jail}.conf: List of target machines to build using a particular build jail.
There is also a pkgbootstrap.sh script that is intended to fetch the sample pkg.conf file and update as necessary, as well as the public pkg.cert file from pbdist.FreeBSD.org, and put them into the correct locations. After fetching the files, the SHA256 is calculated and verified against known values.
Package Distribution
Local Setup
pb00.FreeBSD.org has a single jail for web services, pbdist.FreeBSD.org. Its web root directory is a read-only nullfs(5) mount from pb00.FreeBSD.org, containing an empty index.html file, a sample pkg.conf file, and the public certificate file (pkg.cert) for the SSL key used to sign the packages.
Note: If ever changing the pkg.cert (or the signing key) or pkg.conf, the SHA256 sums must be updated in pkgbootstrap.sh.
Package Repository Creation
poudriere will automatically create the repo.txz using the signing key after a package build completes. In general, there should be nothing to do here, but if manually signing a package build is needed, this can be done as:
# pkg repo /poudrieredata/data/packages/${JAIL}-cluster-${TARGET_HOST}/ /root/ssl/pkg.key
Pkgng
Cluster Server Bootstrapping
A "bootstrapping a system for cluster pkgng builds" example:
root@pb00:~ # fetch http://pbdist.freebsd.org/pkgbootstrap.sh pkgbootstrap.sh 100% of 1465 B 1346 kBps 00m00s root@pb00:~ # sh ./pkgbootstrap.sh /tmp/pkg.cert 100% of 1491 B 4282 kBps 00m00s /tmp/pkg.conf 100% of 479 B 1480 kBps 00m00s SHA256 for pkg.conf and pkg.cert match. Creating /etc/ssl/ directory for pkg.cert... Installing the public certificate file... Done. Running FreeBSD:10 amd64 This hostname is: pb00 Installing pkg.conf... Done. Cleaning up. Done. root@pb00:~ # grep ^PACKAGESITE /usr/local/etc/pkg.conf PACKAGESITE : https://pbdist.FreeBSD.org/10-amd64-cluster-pb00 root@pb00:~ # pkg update Updating repository catalogue digests.txz 100% 2280 2.2KB/s 2.2KB/s 00:00 packagesite.txz 100% 7580 7.4KB/s 7.4KB/s 00:00 Incremental update completed, 0 packages processed: 0 packages updated, 0 removed and 21 added.