Basics:

  • pool/root/version is for a given OS version
  • pool/root/all is for data independent of OS version
  • pool/tld is for new top-level directories - use your judgment

Installation:

gpart create -s gpt ada0
gpart add -s 94 -t freebsd-boot -l boot ada0
gpart add -s 4194304 -t freebsd-swap -l swap ada0
gpart add -l root ada0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0

zpool create -o cachefile=/tmp/zpool.cache -o altroot=/mnt -O compression=lz4 -O checksum=sha256 test /dev/gpt/root
zfs create -o mountpoint=legacy test/root
zfs create test/root/9.0
zfs create test/root/all
zfs create test/root/all/tmp
zfs create test/root/all/var
zfs create test/root/all/var/crash
zfs create test/root/all/var/empty
zfs create -o compression=gzip test/root/all/var/log
zfs create -o compression=off test/root/all/var/ports
zfs create test/root/all/var/ports/distfiles
zfs create test/root/all/var/ports/packages
zfs create test/root/all/var/tmp

zfs set mountpoint=legacy test/root
zpool set bootfs=test/root/9.0 test

mount -t zfs test/root/9.0 /mnt
mount -t zfs test/root/all/tmp /mnt/tmp
mount -t zfs test/root/all/var /mnt/var
mount -t zfs test/root/all/var/crash /mnt/var/crash
mount -t zfs test/root/all/var/empty /mnt/var/empty
mount -t zfs test/root/all/var/log /mnt/var/log
mount -t zfs test/root/all/var/tmp /mnt/var/tmp

Perform installation here, and modify config files:

/mnt/etc/fstab:

test/root/9.0            /           zfs  rw 0 0
test/root/all/tmp        /tmp        zfs  rw 0 0
test/root/all/var        /var        zfs  rw 0 0
test/root/all/var/crash  /var/crash  zfs  rw 0 0
test/root/all/var/empty  /var/empty  zfs  rw 0 0
test/root/all/var/log    /var/log    zfs  rw 0 0
test/root/all/var/tmp    /var/tmp    zfs  rw 0 0

/mnt/boot/loader.conf:

zfs_load="YES"
kern.geom.label.gptid.enable="0"
vfs.root.mountfrom="zfs:test/root/9.0"

If you intend on building ports or src on here:

zfs create -o mountpoint=legacy test/devel
zfs create -o mountpoint=/usr/src -o compression=gzip test/devel/src
zfs create -o mountpoint=/usr/ports -o compression=gzip test/devel/ports
zfs create -o mountpoint=/usr/obj test/devel/obj

Then modify some config files to direct changes to the appropriate datasets:

/etc/make.conf:

DISTDIR=/var/ports/distfiles
PACKAGES=/var/ports/packages
WRKDIRPREFIX=/usr/obj

Each upgrade uses the following sequence:

  • Snapshot all filesystems
zfs snapshot -r mtumishi/root/8.2p2@2011-10-29_20:00:00
  • Clone the snapshots to the new version
zfs clone mtumishi/root/8.2p2@2011-10-29_20:00:00 mtumishi/root/8.2p4
zfs clone mtumishi/root/8.2p2/tmp@2011-10-29_20:00:00 mtumishi/root/8.2p4/tmp
...
  • Perform upgrade
mergemaster -D /zfs/mtumishi/root/8.2p4 -p
make installworld DESTDIR=/zfs/mtumishi/root/8.2p4
make installkernel DESTDIR=/zfs/mtumishi/root/8.2p4
mergemaster -D /zfs/mtumishi/root/8.2p4 -iFU
  • Update the bootcode
gpart bootcode -b /zfs/mtumishi/root/8.2p4/boot/pmbr -p /zfs/mtumishi/root/8.2p4/boot/gptzfsboot -i 1 ada0
gpart bootcode -b /zfs/mtumishi/root/8.2p4/boot/pmbr -p /zfs/mtumishi/root/8.2p4/boot/gptzfsboot -i 1 ada1
  • Point to the new install
echo 'vfs.root.mountfrom="zfs:mtumishi/root/8.2p4"' >> /zfs/mtumishi/root/8.2p4/boot/loader.conf
sed -i 's/^mtumishi\/root/8.2p2/mtumishi\/root/8.2p4/g' /zfs/mtumishi/root/8.2p4/etc/fstab
zpool set bootfs=mtumishi/root/8.2p4 mtumishi
  • Hide mountpoints
zfs set mountpoint=legacy mtumishi/root/8.2p4
  • Reboot to test!
  • Roll back if something went wrong!
zpool set bootfs=mtumishi/root/8.2p2 mtumishi

Notes:

  • It may be useful to place /usr and /usr/local on separate datasets, so you can snapshot before a port upgrade without having to snapshot all of /usr and / as well
NAME                                    USED  AVAIL  REFER  MOUNTPOINT
mtumishi                               2.73G   214G    24K  /zfs/mtumishi
mtumishi/home                          21.1M   214G  21.1M  /home
mtumishi/obj                            875M   214G   875M  /usr/obj
mtumishi/ports                          170M   214G   170M  /usr/ports
mtumishi/root/8.2p2                     426M   214G   423M  legacy
mtumishi/root/8.2p2/tmp                94.5K   214G  66.5K  legacy
mtumishi/root/8.2p2/var                1.86M   214G  1.67M  legacy
mtumishi/root/8.2p2/var/crash          22.5K   214G  22.5K  legacy
mtumishi/root/8.2p2/var/empty            21K   214G    21K  legacy
mtumishi/root/8.2p2/var/tmp              44K   214G    23K  legacy
mtumishi/root/8.2p4                     256M   214G   423M  legacy
mtumishi/root/8.2p4/tmp                  34K   214G  66.5K  legacy
mtumishi/root/8.2p4/var                  57K   214G    21K  legacy
mtumishi/root/8.2p4/var/crash            16K   214G  22.5K  legacy
mtumishi/root/8.2p4/var/tmp              20K   214G    23K  legacy
mtumishi/root/all                      45.6M   214G    22K  legacy
mtumishi/root/all/var                  45.6M   214G  1.69M  /var
mtumishi/root/all/var/empty              15K   214G    21K  /var/empty
mtumishi/root/all/var/log               194K   214G   194K  /var/log
mtumishi/root/all/var/ports            45.2M   214G    24K  /var/ports
mtumishi/root/all/var/ports/distfiles  45.1M   214G  45.1M  /var/ports/distfiles
mtumishi/root/all/var/ports/packages     21K   214G    21K  /var/ports/packages
mtumishi/src                            293M   214G   293M  /usr/src
mtumishi/srv                            354M   214G    25K  /srv
mtumishi/srv/iscsi                       21K   214G    21K  /srv/iscsi
mtumishi/srv/jail                        21K   214G    21K  /srv/jail
mtumishi/srv/srcs                       353M   214G    23K  /srv/srcs
mtumishi/srv/srcs/freebsd               353M   214G    23K  /srv/srcs/freebsd
mtumishi/srv/srcs/freebsd/9             353M   214G   353M  /srv/srcs/freebsd/9