When bringing up future freebsd.org sites
Prerequisites:
- Clear site contact information
- Operational ownership and responsibility established and documented
- Clarity between foundation and project over the site
- Added to clusteradm site index page (Link TBD)
Basic standards:
- Remote power control
- Remote console
- BIOS console redirection enabled
- 115200 baud console
- Switch with ability to partition and remotely change vlan settings on a per-port basis
Internet for tier 1 sites:
- Naked internet, similar to how a site would provide a hosted customer.
- NO upstream firewall. We use a pf/pfsync/igw setup.
- Such sites can be part of global geo-routed services like pkg, www, svn etc.
Internet for tier 2 sites:
- If the upstream is providing firewalls, we can skip the igw/pf/pfsync setup.
- If the upstream is providing firewalls, our recourse for a site problem is to pull the site offline.
- Such sites MAY NOT be used for tier-1 services as we can't do immediate updates if required.
- Such sites MAY NOT carry cluster secrets - eg: copies of kerberos database, password hashes, etc.
- Such sites may be useful for country or site specific mirrors.
Initial bringup:
- Initial network access, basic services and netboot cabability
all subsequent installs done via netboot to prove remote power, console and reinstall capability
- no memory stick / out of band installs unless it is absolutely impossible to netboot install, and documented as an unmanaged machine
unmanaged machines are NOT used for anything critical.
Naming standards:
- 3 letter dns prefix
- understandable hostnames. not obscure acronyms.
- paired / redundant machines with a numerical suffix.
- if at all possible, use an existing naming scheme. Don't get cute. (especially peter@.. did you hear that?)
Jail naming and assignments:
- No **magic**, hacks, needless complexity, including pf hacks, loopback, baked in hostnames etc.
- Only use suffix of jail host for inherently non-portable jails. eg: an input mirror feed for per-machine use.
- machines like blue.freebsd.org can never happen again.
- If there is any possibility of a jail being portable or cloneable, then set it up that way. Don't bake in assumptions about its jail host.
- nullfs does NOT share locks, you cannot share databases across nullfs. Notably including svn.
Subnetting:
- Partition ipv4 space to allow mapping subnets to vlans
- Partition according to similar function. Don't put world+dog developers in the same vlan as security critical machines.
Cluster security:
- ssh keys for initial access
- ksu for elevated access
- NO password hashes, aside from root for console use.
- NO sudo
- NO g+w as a means of accessing a privileged subsystem's files
- NO logging in directly as role or privileged accounts, use ksu
- Privilege containment is to be done by having privileged processes safely pulling from work areas
- NO automated privilege escalation by scripts etc.
ssh PermitRootLogin no
- ssh password authentication disabled.
- use integrated clusteradm scripts.
- Beat somebody up if clusteradm scripts are missing roles, don't use roles as an excuse to not use them.
IPv4 address assignment:
- Link TBD
IPv6 address assignment:
- Link TBD
Vlan assignment:
Cluster foothold requirements:
- A pair of routers
- An admin shell / netboot / general work box
- 3+ jail servers for cluster integration:
- hosts/jails for replication of network services (dns, ntp, ldap, kerberos). these need to be redundant
- a jail for sensitive things.. snmp monitoring, consoles, etc. (This may be able to be hosted on admin server.)
"Special" machines:
- Noted on the site index page.