CDNWiki - View Source: KnowledgeBase/FreeBSD/Mosi

*Note:* Pretty much all of this is implemented more elegantly in the firmware generation framework, located at [http://git.cyberleo.net/Mosi.git]
----
Experiences setting up FreeBSD 6.2 to run as a router entirely off flash media.

A lot of these instructions assume a single go. Rebooting before the end can cause problems with missing files or incorrect permissions, which can be difficult to get out of. It should also be done in single user mode, to reduce the chance of daemons scribbling during preperation.

Patch rc.initdiskless to mount /conf from things other than nfs. (8.0 version)
<code brush=bash>
--- rc.initdiskless.orig        2009-11-30 02:55:12.000000000 +0000
+++ rc.initdiskless     2010-01-03 16:02:27.000000000 +0000
@@ -190,8 +190,21 @@ # handle_remount() { # $1 = mount point
     log "nfspt ${nfspt} mountopts ${mountopts}"
     # prepend the nfs root if not present
     [ `expr "$nfspt" : '\(.\)'` = "/" ] && nfspt="${nfsroot}${nfspt}"
-    mount_nfs $mountopts $nfspt $b
-    chkerr $? "mount_nfs $nfspt $b"
+
+    # <CyberLeo> Hack to allow mounting of any filesystem type, not just NFS
+    #  (thumbdrive w/glabel *hint hint*)
+    # ufs:/dev/label/conf (or nfs:server:/root -- server:/root works too, if
+    #  server isn't named ufs, msdosfs, etc...)
+    eval $(echo ${nfspt} | sed -E 's/^([^:]+):(.*)$/fstype="\1" device="\2"/')
+    case "${fstype}" in
+    ufs|msdosfs|cd9660|nfs) ;;
+    # Backwards compatibility. Hopefully someoene didn't name their NFS server 'ufs'...
+    *) device="${fstype}:${device}"; fstype="nfs" ;;
+    esac
+    mount -t "${fstype}" "${device}" "${b}"
+    chkerr $? "mount -t ${fstype} ${device} ${b}"
+    # </CyberLeo>
+
     to_umount="$b ${to_umount}"
 }
</code>

Patch rc.initdiskless to mount /conf from things other than nfs. (6.2 version)
<code brush=bash>
--- /etc/rc.initdiskless.orig        Sun May  7 19:00:00 2006
+++ /etc/rc.initdiskless     Mon Feb 12 08:03:03 2007
@@ -229,8 +229,25 @@
     if [ `expr "$nfspt" : '\(.\)'` = "/" ]; then
        nfspt="${nfsroot}${nfspt}"
     fi
-    mount_nfs $nfspt /conf
-    chkerr $? "mount_nfs $nfspt /conf"
+
+    # <CyberLeo> Hack to allow mounting of any filesystem type, not just NFS (thumbdrive w/glabel *hint hint*)
+    #  ufs:/dev/label/conf (or nfs:server:/root -- server:/root works too, if server isn't named ufs, msdos, etc..)
+    eval $(echo ${nfspt} | sed -E 's/^([^:]+):(.*)$/fstype="\1" device="\2"/')
+    case "${fstype}" in
+    ufs)       ;;
+    msdosfs)     ;;
+    cd9660)    ;;
+    nfs)       ;;
+    # Backwards compatibility. Hopefully someone didn't name their NFS server 'msdos'...
+    *)         device="${fstype}:${device}"; fstype="nfs" ;;
+    esac
+
+    mount -t ${fstype} ${device} /conf
+    chkerr $? "mount -t ${fstype} ${device} /conf"
+
+    # mount_nfs $nfspt /conf
+    # chkerr $? "mount_nfs $nfspt /conf"
+
+    # </CyberLeo>
     to_umount="/conf"
 fi

</code>

Disable sendmail, and replace with something better
<code brush=bash>
cat >> /etc/rc.conf <<EOF
sendmail_enable="NONE"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"
EOF

cat >> /etc/periodic.conf <<EOF
daily_clean_hoststat_enable="NO"
daily_status_mail_rejects_enable="NO"
daily_status_include_submit_mailq="NO"
daily_submit_queuerun="NO"
EOF
</code>

I chose mini_sendmail (/usr/ports/mail/mini_sendmail) because a router doesn't need a full MTA.<br>
<small>(I nfs-mounted /usr/ports from another server, so that I wouldn't clutter up the read-only image.)</small>
<code brush=bash>
cd /usr/ports/mail/mini_sendmail && make install clean
cat > /etc/mail/mailer.conf <<EOF
# Send outgoing mail to a smart relay using mini_sendmail
sendmail	   /usr/local/bin/mini_sendmail -s<host> -p<port>
send-mail	   /usr/local/bin/mini_sendmail -s<host> -p<port>
EOF
</code>

Get things ready to be read-only. This includes moving all configuration directories together, and moving tmp into a writable space.
This is best done in single-user mode.
<code brush=bash>
# Move /tmp into to-be-writable space
mv /tmp/* /tmp/.??* /var/tmp/
rmdir /tmp
ln -svf /var/tmp /tmp

# Move 'local' configuration into /etc
mv /usr/local/etc/rc.d /etc/rc.local # An arbitrary name for the local init scripts directory
mv /usr/local/etc/* /etc/
rmdir /usr/local/etc
ln -svf /etc /usr/local/etc
echo 'local_startup="/etc/rc.local"' >> /etc/rc.conf # Tell rc where to find the new location of the local init scripts
</code>

Enable diskless mode (rc.initdiskless)
<code brush=bash>
touch /etc/diskless
</code>

Indicate where the config partition resides.
Around here, it's also useful to modify /etc/fstab to set / as read-only, so that the changes are propagated properly.
<code brush=bash>
mkdir /conf
glabel label -v conf /dev/ad4
newfs -U /dev/label/conf
echo "ufs:/dev/label/conf" > /conf/diskless_remount
echo "/dev/label/conf    /conf    ufs    ro,noauto,noatime,sync    2    2" >> /etc/fstab
</code>

Set up the configuration. Ideally, this should be done on a 'clean' system with as many baseline enhancements as you want to keep. I.e. useful packages and ports, barebones passwd and group files, etc.. Additional configuration will be saved elsewhere, allowing a quick method of 'starting over' by simply removing the changed configuration.
<code brush=bash>
mount -w /conf
mkdir -p /conf/{base,default}/{etc,var}
echo "10240" > /conf/base/etc/md_size # 5MB
echo "131072" > /conf/base/var/md_size # 64MB
</code>

There are two methods to handle the files. If you have ample space on your flash media, the file method can save accesses and writes to precious flash cells. If you are short on space, the archive method is tighter, but requires proper timestamping, and it writes all changed files each save.

When staging the base archives or filesets, it's useful to remove files that are not necessary across reboots, or are autogenerated if absent. These include SSH keys from /etc/ssh/ssh_host_* (They will be autogenerated, then saved in /conf/default when saveconfig is run, so that nonconflicting copies can be made without too much trouble), /var/run/sudo/* (most likely to expire across reboots anyways) and so on.

!! File Method
<code brush=bash>
cp -vpR /etc /conf/base/etc
cp -vpR /var /conf/base/var
for I in /conf/base/var/log; do cat /dev/null > ${I}; done # Zero out logfiles to save space
rm -Rf /conf/base/var/tmp/* /conf/base/var/tmp/.??* # Remove stale temp files. Everything in here gets recreated on reboot anyways.
</code>

I haven't written a full saveconfig script for the file method, as I am using the archive method myself. However, the following rsync command line (requires rsync, from packages or ports) does the job, provided /conf is mounted. (It won't be left mounted after booting)
<code brush=bash>
# Copy all changed files, times, permissions, ownership from /etc and /var to /conf/default/{etc,var},
#  and consider /conf/base/{etc,var} as additional comparison files. This will keep all the files in /conf/base from being
#  copied to /conf/default if they haven't been changed.
rsync --archive --delete --hard-links --exclude='*/tmp/*' --compare-dest=/conf/base /etc /var /conf/default/
</code>

!! Archive Method
<code brush=bash>
cd /tmp # Temporary staging area

cp -vpR /var /tmp/
for I in var/log; do cat /dev/null > ${I}; done # Zero out logfiles to save space
rm -Rf /conf/base/var/tmp/* /conf/base/var/tmp/.??* # Remove stale temp files. Everything in here gets recreated on reboot anyways.
find var -print0 | xargs -0 touch -ht 200605080000.00 # Set the timestamps for all the unchanged files, so we can find the changed ones easier.
tar zcpvf /conf/base/var.cpio.gz var # Structure must be relative to / for extraction to work properly.
rm -Rf var

cp -vpR /etc /tmp/
find etc -print0 | xargs -0 touch -ht 200605080000.00
touch -ht 200605080000.01 etc/diskless # This file is our baseline, as it will (hopefully) never change.
tar zcpvf /conf/base/etc.cpio.gz etc
rm -Rf etc
</code>

The saveconfig script (700 root:wheel in /sbin)
<code brush=bash>
#!/bin/sh

echo -n "Saving configuration... "

if [ ! -f /etc/diskless ]
then
    echo "No flash setup detected."
    exit 1
fi

# Mount /conf read-write, and remount if it already is.
if [ $(grep -c "/conf" /etc/fstab) -gt 0 ]
then
        mount -w /conf
        if [ $? -ne 1 ]
        then
                umount /conf
                mount -w /conf
        fi
fi

# Five oldest backups method:
# Remove the oldest backup
#[ -d /conf/backup/5 ] && rm -Rf /conf/backup/5

# Rotate the backups to make room
#for I in 4 3 2 1
#do
#        dest=$(expr "${I}" + 1)
#        [ -d "/conf/backup/${I}" ] && mv "/conf/backup/${I}" "/conf/backup/${dest}"
#done
# Back up previous config and create space for the new one
#mv /conf/default /conf/backup/1

# Unlimited history method: (make sure you have provisions for removing the old backups, or this can get HUGE!
dest=$(expr "$(ls -1 /conf/backup | tail -n 1)" + 1)
mv /conf/default "/conf/backup/${dest}"

mkdir /conf/default

cd /
# Copy changed config files to /conf/default (anything younger than /etc/diskless)
find etc var -type f -not -regex '.*/tmp/.*' -newer /etc/diskless -print0 | xargs -0 tar cpf - | tar xpCf /conf/default/ -
# This can be changed to allow compressed configuration here as well:
# find etc -type f -not -regex '.*/tmp/.*' -newer /etc/diskless -print0 | xargs -0 tar cpf /conf/default/etc.cpio.gz
# find var -type f -not -regex '.*/tmp/.*' -newer /etc/diskless -print0 | xargs -0 tar cpf /conf/default/var.cpio.gz

# Umount /conf afterwards
if [ "$(mount |grep -c "/conf")" -gt 0 ]
then
        umount /conf
fi

echo "Done!"
</code>

Save existing configuration on shutdown, in /etc/rc.local/saveconfig:
<code brush=bash>
#!/bin/sh

# PROVIDE: saveconfig
# REQUIRE: var
# BEFORE: swap1
# KEYWORD: shutdown

name="saveconfig"
stop_cmd="saveconfig_stop"

saveconfig_stop(){
	/sbin/saveconfig
}

load_rc_config $name
run_rc_command "$1"
</code>

!Things left to do:

* Custom kernel with AltQ
* Install DHCPd diskless
* Install BIND diskless
* Install OpenVPN diskless
* PGP/GPG for logmail signing
* Log handler scripts, to sign and mail at regular intervals

! Sundry Notes
* /boot/kernel/*.symbols are unstripped copies of their similarly named counterparts. They are only needed for debugging.
* While /boot/loader can load gzip-compressed kernel, it cannot load gzip-compressed modules. It just stalls.
* Pare down root/etc because it cannot possibly need all that just to locate and extract the real etc.
** Perhaps just /etc/rc, /etc/diskless, and /etc/rc.initdiskless?
* Copy SSH keys into tree, to avoid annoying SSH conflicts