User:Mjb/FreeBSD 8 installation and upgrade

These notes apply to FreeBSD 8, so they are pretty outdated now.

FreeBSD 8 OS installation

Disk space allocation

This is only if you are installing from scratch using the FreeBSD installer, as opposed to just copying a snapshot.

You need roughly 15 GB for the OS, ports, and upgrades.

You also need space for "userland" (e.g. home directories, websites, 3rd-party software) as well as mail and temporary files. This kind of content totals another 5 GB on my modest system with very few users.

You also need swap space, the ideal amount of which depends on how much physical RAM you have and how much RAM your system will ever need at one time; "twice the amount of physical RAM" used to be the recommendation, and 4 GB seems to be a fairly standard amount these days, but I find that 500 MB is plenty. Ideally, swap space should be a partition, not a file.

It used to be recommended to have separate partitions and slices (sub-partitions) for certain directories, but in my experience, it is perfectly fine to have one partition for swap and one partition for everything else. You might consider using a separate drive for userland, though, so that you can recover more easily in case a drive dies.

OS upgrade

Upgrade to a new patch level

This is only if you are running FreeBSD from a RELEASE branch.

A FreeBSD RELEASE version may include a patch level, e.g. FreeBSD 10.1-RELEASE-p10 indicates patch level 10. This level correlates with security patches that were released as replacement source code and binaries which were installed using freebsd-update. It does not take into account any binaries you patched and rebuilt yourself as directed by security advisories.

These security patches are only available for OS versions that are still "supported", i.e. not more than 2 years old. Therefore, you may first have to do a minor version update (see next section) or new patches won't even be available for your system.

Prepare the environment

If you have "-v" in your GZIP environment variable, this really clutters the output of freebsd-update, so unset it:

• unsetenv GZIP

Get patches

• freebsd-update fetch

The new files will be downloaded to a temporary location and listed so you know what's going to change. If you have the OS source code installed in /usr/src, source patches will be included in the update as well (unless you modified /etc/freebsd-update.conf to omit src).

Install patches

• freebsd-update install

Rebuild customized system binaries

If you previously recompiled any of your system binaries with custom options, then freebsd-update may well replace those files, so you will have to recompile them. Otherwise, you will suddenly be running the standard version.

For example, enabling SMTP Authentication used to require building a custom version of Sendmail, so after running freebsd-update, I'd often have to run this script I wrote (I call it rebuild_sendmail):

#!/bin/sh
cd /usr/src/lib/libsmutil
make cleandir && make obj && make
cd /usr/src/lib/libsm
make cleandir && make obj && make
cd /usr/src/usr.sbin/sendmail
make cleandir && make obj && make && make install
cd /etc/mail
make restart

Reboot

Whether a reboot is needed depends on what was updated. You have to decide that yourself. Obviously anything kernel-related should make you want to do a reboot. If you don't do a reboot, but system daemons were updated, you'll need to restart those.

Upgrade to a new minor version of the OS

Reference: FreeBSD Update section of the FreeBSD Handbook

The following info is based on my upgrade from 8.1-RELEASE to 8.3-RELEASE, and from 8.3-RELEASE to 8.4-RELEASE (assumes generic kernel):

Prepare the environment

I normally have "-v" in my GZIP environment variable, and this really clutters the output of freebsd-update, so unset it:

• unsetenv GZIP

Get new files

• freebsd-update -r 8.3-RELEASE upgrade

Takes several hours.

Merge files

Most merges will happen automatically, but some un-mergeable files like /etc/passwd will be reported, and you need to answer 'y' and merge them manually...but (as of the time of this writing) you don't get a nice merge interface, you just get dumped into an empty text editor! What you are expected to do here is create a merged file. Be very careful!

The goal is to compare and merge each old file from the directory tree rooted at /var/db/freebsd-update/merge/old (copied from the live system) with the corresponding new file in /var/db/freebsd-update/merge/XXX, where XXX is the new FreeBSD version you're upgrading to (e.g. 8.4-RELEASE). You need to put each merged file into the same relative location under /var/db/freebsd-update/merge/new, which is where the empty text editor will be saving to.

In my upgrade to 8.3-RELEASE, I just elected to go into the editor (you have no choice, really), load the old file, and save it as-is. I didn't bother merging in the new one! Not ideal, but the least amount of hassle, right?

In my upgrade to 8.4-RELEASE, I tried a new approach: merge the files in a separate window, pre-populating the new folder, so that when the editor is opened, it's not empty, but rather has the merged file in it. Then I can just give it a once-over and save the result.

To accomplish this, in a separate terminal, as root, it would be nice to be able to run mergemaster. So I tried to do it like this:

• mergemaster -w 100 -ciFv -m /var/db/freebsd-update/merge/8.4-RELEASE -D /var/db/freebsd-update/merge/new

However, it didn't work. I have asked about it on the freebsd-questions mailing list. Here is another, cruder method I tried, which did work:

• cd /var/db/freebsd-update/merge/8.4-RELEASE
• find -X . -type f | xargs -n 1 -o -I % sh -c '{ echo Now processing %. left=current, right=new, help="?"; sdiff -d -w 100 -o ../new/% ../old/% %; }'

The downside of this method is that it assumes you want to do an interactive merge (sdiff) of every file, whereas sometimes you are really going to want to save time and just choose to use the old or new file without merging; mergemaster would give you that ability.

Regardless of how you do your merge, once you've saved all the files in the editor, you'll be prompted to approve a diff for each one. If you answer "n" to any of these prompts, it will abort the entire upgrade and you will have to start over! So hopefully the merges are all OK, and you can continue.

However, among the changes you're asked to approve may be unspecified differences in /etc/pwd.db and /etc/spwd.db, the binary files that contain your password database. You have no choice but to answer "y", but for God's sake, rebuild those files before rebooting! (see below).

Review changes

freebsd-update now presents you with lists of all the files that will be deleted, all the files that will be added, and all the files that will be modified.

Pay special attention to the changes in /etc.

After showing you the lists, that's it, nothing happens. The changes are ready to be made, but nothing has actually happened yet.

Install the new files

You are about to overwrite your real system files. I suggest making a backup of /etc first:

• cp -pr /etc /tmp/etc.backup

Cross your fingers:

• freebsd-update install

Rebuild soon-to-be-clobbered databases

Now, unless you got mergemaster to work, you probably have to do the things that mergemaster normally would do for you.

It seems things don't get replaced until after reboot. This may be a real problem!

If /etc/passwd or /etc/master.passwd were changed or if /etc/pwd.db or (most importantly, I think) /etc/spwd.db changed (e.g., as in 8.4-RELEASE, got set to new defaults), then a pwd_mkdb run will be necessary to regenerate the .db files, and you want to do this before your shutdown or you'll never get to log back in.

Normally you would do this:

• pwd_mkdb -p /etc/master.passwd

This will use /etc/master.passwd as the source file, and the -p means generate a new /etc/passwd from it, in addition the the .db files.

However, the files in /etc are, at this stage, untouched. The new versions are sitting gzipped in /var/db/freebsd-update/files, a huge dumping ground with no sub-structure. An index to the files is in /var/db/freebsd-update/install.XXXXX/INDEX-NEW, where XXXXX is a random ID; look at the directory creation date to figure out which one is current, if there's more than one.

So I think what you need to do is something like this, to inspect the new files:

• cd /var/db/freebsd-update
• mkdir -m 0700 /tmp/oldpwdfiles
• zcat files/`grep '^/etc/master\.passwd' install.LYQAJQ/INDEX-NEW | cut -d \| -f 7`.gz > /tmp/oldpwdfiles/master.passwd
• zcat files/`grep '^/etc/passwd' install.LYQAJQ/INDEX-NEW | cut -d \| -f 7`.gz > /tmp/oldpwdfiles/passwd
• zcat files/`grep '^/etc/pwd\.db' install.LYQAJQ/INDEX-NEW | cut -d \| -f 7`.gz > /tmp/oldpwdfiles/pwd.db
• zcat files/`grep '^/etc/spwd\.db' install.LYQAJQ/INDEX-NEW | cut -d \| -f 7`.gz > /tmp/oldpwdfiles/spwd.db
• ls -l /tmp/oldpwdfiles

total 10
 6 -rw-r--r--  1 root  wheel   4.0k Jun 25 00:48 master.passwd
 4 -rw-r--r--  1 root  wheel   3.2k Jun 25 00:49 passwd
 0 -rw-r--r--  1 root  wheel     0B Jun 25 00:49 pwd.db
 0 -rw-r--r--  1 root  wheel     0B Jun 25 00:49 spwd.db

Obviously pwd.db and spwd.db are crap and we'd be in trouble if we installed those empty files!

If /tmp/oldpwdfiles/master.passwd looks OK, then try generating a new passwd file and pair of .db files:

• mkdir -m 0700 /tmp/newpwdfiles
• pwd_mkdb -d /tmp/newpwdfiles -p /tmp/oldpwdfiles/master.passwd
• ls -l /tmp/newpwdfiles

total 138
  6 -rw-------  1 root  wheel   4.0k Jun 25 00:48 master.passwd
  4 -rw-r--r--  1 root  wheel   3.2k Jun 25 00:53 passwd
 68 -rw-r--r--  1 root  wheel    68k Jun 25 00:53 pwd.db
 60 -rw-------  1 root  wheel    60k Jun 25 00:53 spwd.db

Quite a bit better. As you can see, master.passwd was just moved over, and the other three files were generated. Now to replace them:

• gzip /tmp/newpwdfiles/*
• mv /tmp/newpwdfiles/master.passwd.gz files/`grep '^/etc/master\.passwd' install.LYQAJQ/INDEX-NEW | cut -d \| -f 7`.gz
• mv /tmp/newpwdfiles/passwd.gz files/`grep '^/etc/passwd' install.LYQAJQ/INDEX-NEW | cut -d \| -f 7`.gz
• mv /tmp/newpwdfiles/pwd.db.gz files/`grep '^/etc/pwd\.db' install.LYQAJQ/INDEX-NEW | cut -d \| -f 7`.gz
• mv /tmp/newpwdfiles/spwd.db.gz files/`grep '^/etc/spwd\.db' install.LYQAJQ/INDEX-NEW | cut -d \| -f 7`.gz

And finally, clean up:

• rm -fr /tmp/oldpwdfiles /tmp/newpwdfiles

You'll have to go through a similar process if you use sendmail and you merged in any changes to /etc/mail/aliases or /etc/mail/*.cf files. Ordinarily, the most thorough way is this:

• cd /etc/mail; make all
• make install
• make restart

But as before, the files haven't been installed yet!

Likewise, changes to /etc/login.conf require rebuilding a database:

• cap_mkdb (see the man page for exact syntax)

Same for /etc/services:

• services_mkdb (see the man page for exact syntax)

There's a bug report filed about this: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=165954

Reboot and continue

OK, now reboot to try out the new kernel:

• shutdown -r now (again, this assumes you want the generic kernel)

Hope & pray it comes back up. If it does, do this again to get world installed:

• freebsd-update install

This worked for me, for the upgrade to 8.3-RELEASE.

For the 8.4 upgrade, after this stage, it said:

Completing this upgrade requires removing old shared object files.
Please rebuild all installed 3rd party software (e.g., programs
installed from the ports tree) and then run "/usr/sbin/freebsd-update install"
again to finish installing updates.

Worry about that in a minute. First, realize that at this point, /etc has been modified, so it's a good idea to make sure you like the look of the new files, especially these:

• /etc/master.passwd
• /etc/group
• /etc/mail/* (if changed, you need to run the appropriate make command in /etc/mail ...perhaps make all install restart)
• /etc/services (if changed, you need to run services_mkdb -q to rebuild /var/db/services.db)
• /etc/login.conf (if changed, freebsd-update should've run cap_mkdb to rebuild login.conf.db)

If anything's amiss, remember you made a backup in /tmp/etc.

OK, now you can follow the directions below to update your ports tree and rebuild everything(!). Personally I don't like doing this because things tend to go wrong if you don't do it piecemeal. The downside is that some things will be left un-updated. But you can deal with that; read on...

Check for cruft

After the upgrade, you might want to see if anything out-of-date got left behind:

• cd /usr/src && make check-old

If there's anything, you can run make delete-old to get rid of it; it will ask you about each file, normally. Ref: http://www.freebsd.org/doc/handbook/make-delete-old.html

There are a couple of options for checking the installed shared libraries:

• If you install the sysutils/bsdadminscripts port, you can run pkg_libchk to check for missing libraries. It even tells you which ports are affected.
• If you install the sysutils/libchk port (note: requires Ruby), you can run libchk to check for missing libraries, check for unused libraries, and see exactly which binaries use each library. To figure out which port installed the file needing the library, you need to run pkg info -W /path/to/the/file.

Sample output of pkg_libchk:

gamin-0.1.10_4: /usr/local/libexec/gam_server misses libpcre.so.0
gio-fam-backend-2.28.8_1: /usr/local/lib/gio/modules/libgiofam.so misses libpcre.so.1

Rebuilding these two ports should be sufficient to get them linked to the current libpcre library. (Double-checking /usr/local/lib shows that there's a libpcre.so.3 now).

Why did I have these ports installed? pkg info -R gamin-0.1.10_4 tells me gamin is required by gio-fam-backend, and pkg info -R gio-fam-backend-2.28.8_1 reveals that gio-fam-backend isn't required by anything that I currently have installed. This is a weird port, though, and it is not something you want to deinstall. It is FreeBSD-specific, and is kind of a companion to the glib port. (Though apparently they decommissioned it - see the 20130731 entry in UPDATING). pkg info -R glib-2.34.3 reveals what's using glib: ImageMagick & MediaWiki.

Anyway, portmaster --update-if-newer gio-fam gamin takes care of the problem. Now when I run pkg_libchk gamin-0.1.10_5 and pkg_libchk gio-fam-backend-2.34.3 (the new versions), there are no problems. The question now is whether I need to update ImageMagick. The lack of problems reported by pkg_libchk ImageMagick-nox11-6.8.0.7_1 suggests the answer is no.

Reboot to restart daemons

After upgrading from 8.3-RELEASE to 8.4-RELEASE, /var/log/messages started accumulating error messages from sshd, every time someone tried to log in:

error: Could not load host key: /etc/ssh/ssh_host_ecdsa_key

Indeed, that key file didn't exist until after another reboot, which didn't happen until a mysterious, probably unrelated crash a month after the upgrade.

Web searches suggest that most people running into this problem aren't able to log in at all until they run a special ssh_keygen command to create the missing files, but I was having no such trouble.

I think that for me, the only problem was that after finishing the OS upgrade, sshd needed to actually be restarted. This makes me think that maybe it's a good idea to restart all the daemons as the penultimate step in upgrading the OS. To do that, you could run service -R, but it might be easier to just reboot.