Experiences setting up FreeBSD 6.2 to run as a router entirely off flash media.
A lot of these instructions assume a single go. Rebooting before the end can cause problems with missing files or incorrect permissions, which can be difficult to get out of. It should also be done in single user mode, to reduce the chance of daemons scribbling during preperation.
Patch rc.initdiskless to mount /conf from things other than nfs.
--- rc.initdiskless.orig 2009-11-30 02:55:12.000000000 +0000
+++ rc.initdiskless 2010-01-03 16:02:27.000000000 +0000
@@ -190,8 +190,21 @@ # handle_remount() { # $1 = mount point
log "nfspt ${nfspt} mountopts ${mountopts}"
# prepend the nfs root if not present
[ `expr "$nfspt" : '\(.\)'` = "/" ] && nfspt="${nfsroot}${nfspt}"
- mount_nfs $mountopts $nfspt $b
- chkerr $? "mount_nfs $nfspt $b"
+
+ # <CyberLeo> Hack to allow mounting of any filesystem type, not just NFS
+ # (thumbdrive w/glabel *hint hint*)
+ # ufs:/dev/label/conf (or nfs:server:/root -- server:/root works too, if
+ # server isn't named ufs, msdosfs, etc...)
+ eval $(echo ${nfspt} | sed -E 's/^([^:]+):(.*)$/fstype="\1" device="\2"/')
+ case "${fstype}" in
+ ufs|msdosfs|cd9660|nfs) ;;
+ # Backwards compatibility. Hopefully someoen didn't name their NFS server 'ufs'...
+ *) device="${fstype}:${device}"; fstype="nfs" ;;
+ esac
+ mount -t "${fstype}" "${device}" "${b}"
+ chkerr $? "mount -t ${fstype} ${device} ${b}"
+ # </CyberLeo>
+
to_umount="$b ${to_umount}"
}
Disable sendmail, and replace with something better
cat >> /etc/rc.conf <<EOF sendmail_enable="NONE" sendmail_submit_enable="NO" sendmail_outbound_enable="NO" sendmail_msp_queue_enable="NO" EOF cat >> /etc/periodic.conf <<EOF daily_clean_hoststat_enable="NO" daily_status_mail_rejects_enable="NO" daily_status_include_submit_mailq="NO" daily_submit_queuerun="NO" EOF
I chose mini_sendmail (/usr/ports/mail/mini_sendmail) because a router doesn't need a full MTA.
(I nfs-mounted /usr/ports from another server, so that I wouldn't clutter up the read-only image.)
cd /usr/ports/mail/mini_sendmail && make install clean cat > /etc/mail/mailer.conf <<EOF # Send outgoing mail to a smart relay using mini_sendmail sendmail /usr/local/bin/mini_sendmail -s<host> -p<port> send-mail /usr/local/bin/mini_sendmail -s<host> -p<port> EOF
Get things ready to be read-only. This includes moving all configuration directories together, and moving tmp into a writable space. This is best done in single-user mode.
# Move /tmp into to-be-writable space mv /tmp/* /tmp/.??* /var/tmp/ rmdir /tmp ln -svf /var/tmp /tmp # Move 'local' configuration into /etc mv /usr/local/etc/rc.d /etc/rc.local # An arbitrary name for the local init scripts directory mv /usr/local/etc/* /etc/ rmdir /usr/local/etc ln -svf /etc /usr/local/etc echo 'local_startup="/etc/rc.local"' >> /etc/rc.conf # Tell rc where to find the new location of the local init scripts
Enable diskless mode (rc.initdiskless)
touch /etc/diskless
Indicate where the config partition resides. Around here, it's also useful to modify /etc/fstab to set / as read-only, so that the changes are propagated properly.
mkdir /conf glabel label -v conf /dev/ad4 newfs -U /dev/label/conf echo "ufs:/dev/label/conf" > /conf/diskless_remount echo "/dev/label/conf /conf ufs ro,noauto,noatime,sync 2 2" >> /etc/fstab
Set up the configuration. Ideally, this should be done on a 'clean' system with as many baseline enhancements as you want to keep. I.e. useful packages and ports, barebones passwd and group files, etc.. Additional configuration will be saved elsewhere, allowing a quick method of 'starting over' by simply removing the changed configuration.
mount -w /conf
mkdir -p /conf/{base,default}/{etc,var}
echo "10240" > /conf/base/etc/md_size # 5MB
echo "131072" > /conf/base/var/md_size # 64MB
There are two methods to handle the files. If you have ample space on your flash media, the file method can save accesses and writes to precious flash cells. If you are short on space, the archive method is tighter, but requires proper timestamping, and it writes all changed files each save.
When staging the base archives or filesets, it's useful to remove files that are not necessary across reboots, or are autogenerated if absent. These include SSH keys from /etc/ssh/ssh_host_* (They will be autogenerated, then saved in /conf/default when saveconfig is run, so that nonconflicting copies can be made without too much trouble), /var/run/sudo/* (most likely to expire across reboots anyways) and so on.
File Method
cp -vpR /etc /conf/base/etc
cp -vpR /var /conf/base/var
for I in /conf/base/var/log; do cat /dev/null > ${I}; done # Zero out logfiles to save space
rm -Rf /conf/base/var/tmp/* /conf/base/var/tmp/.??* # Remove stale temp files. Everything in here gets recreated on reboot anyways.
I haven't written a full saveconfig script for the file method, as I am using the archive method myself. However, the following rsync command line (requires rsync, from packages or ports) does the job, provided /conf is mounted. (It won't be left mounted after booting)
# Copy all changed files, times, permissions, ownership from /etc and /var to /conf/default/{etc,var},
# and consider /conf/base/{etc,var} as additional comparison files. This will keep all the files in /conf/base from being
# copied to /conf/default if they haven't been changed.
rsync --archive --delete --hard-links --exclude='*/tmp/*' --compare-dest=/conf/base /etc /var /conf/default/
Archive Method
cd /tmp # Temporary staging area
cp -vpR /var /tmp/
for I in var/log; do cat /dev/null > ${I}; done # Zero out logfiles to save space
rm -Rf /conf/base/var/tmp/* /conf/base/var/tmp/.??* # Remove stale temp files. Everything in here gets recreated on reboot anyways.
find var -print0 | xargs -0 touch -ht 200605080000.00 # Set the timestamps for all the unchanged files, so we can find the changed ones easier.
tar zcpvf /conf/base/var.cpio.gz var # Structure must be relative to / for extraction to work properly.
rm -Rf var
cp -vpR /etc /tmp/
find etc -print0 | xargs -0 touch -ht 200605080000.00
touch -ht 200605080000.01 etc/diskless # This file is our baseline, as it will (hopefully) never change.
tar zcpvf /conf/base/etc.cpio.gz etc
rm -Rf etc
The saveconfig script (700 root:wheel in /sbin)
#!/bin/sh
echo -n "Saving configuration... "
if [ ! -f /etc/diskless ]
then
echo "No flash setup detected."
exit 1
fi
# Mount /conf read-write, and remount if it already is.
if [ $(grep -c "/conf" /etc/fstab) -gt 0 ]
then
mount -w /conf
if [ $? -ne 1 ]
then
umount /conf
mount -w /conf
fi
fi
# Five oldest backups method:
# Remove the oldest backup
#[ -d /conf/backup/5 ] && rm -Rf /conf/backup/5
# Rotate the backups to make room
#for I in 4 3 2 1
#do
# dest=$(expr "${I}" + 1)
# [ -d "/conf/backup/${I}" ] && mv "/conf/backup/${I}" "/conf/backup/${dest}"
#done
# Back up previous config and create space for the new one
#mv /conf/default /conf/backup/1
# Unlimited history method: (make sure you have provisions for removing the old backups, or this can get HUGE!
dest=$(expr "$(ls -1 /conf/backup | tail -n 1)" + 1)
mv /conf/default "/conf/backup/${dest}"
mkdir /conf/default
cd /
# Copy changed config files to /conf/default (anything younger than /etc/diskless)
find etc var -type f -not -regex '.*/tmp/.*' -newer /etc/diskless -print0 | xargs -0 tar cpf - | tar xpCf /conf/default/ -
# This can be changed to allow compressed configuration here as well:
# find etc -type f -not -regex '.*/tmp/.*' -newer /etc/diskless -print0 | xargs -0 tar cpf /conf/default/etc.cpio.gz
# find var -type f -not -regex '.*/tmp/.*' -newer /etc/diskless -print0 | xargs -0 tar cpf /conf/default/var.cpio.gz
# Umount /conf afterwards
if [ "$(mount |grep -c "/conf")" -gt 0 ]
then
umount /conf
fi
echo "Done!"
Save existing configuration on shutdown, in /etc/rc.local/saveconfig:
#!/bin/sh
# PROVIDE: saveconfig
# REQUIRE: var
# BEFORE: swap1
# KEYWORD: shutdown
name="saveconfig"
stop_cmd="saveconfig_stop"
saveconfig_stop(){
/sbin/saveconfig
}
load_rc_config $name
run_rc_command "$1"
Things left to do:
- Custom kernel with AltQ
- Install DHCPd diskless
- Install BIND diskless
- Install OpenVPN diskless
- PGP/GPG for logmail signing
- Log handler scripts, to sign and mail at regular intervals
Sundry Notes
* /boot/kernel/*.symbols are unstripped copies of their similarly named counterparts. They are only needed for debugging.
- While /boot/loader can load gzip-compressed kernel, it cannot load gzip-compressed modules. It just stalls.