CDNWiki
View Source:
KnowledgeBase/FreeBSD/Mosi
*Note:* Pretty much all of this is implemented more elegantly in the firmware generation framework, located at [http://git.cyberleo.net/Mosi.git] ---- Experiences setting up FreeBSD 6.2 to run as a router entirely off flash media. A lot of these instructions assume a single go. Rebooting before the end can cause problems with missing files or incorrect permissions, which can be difficult to get out of. It should also be done in single user mode, to reduce the chance of daemons scribbling during preperation. Patch rc.initdiskless to mount /conf from things other than nfs. (8.0 version) <code brush=bash> --- rc.initdiskless.orig 2009-11-30 02:55:12.000000000 +0000 +++ rc.initdiskless 2010-01-03 16:02:27.000000000 +0000 @@ -190,8 +190,21 @@ # handle_remount() { # $1 = mount point log "nfspt ${nfspt} mountopts ${mountopts}" # prepend the nfs root if not present [ `expr "$nfspt" : '\(.\)'` = "/" ] && nfspt="${nfsroot}${nfspt}" - mount_nfs $mountopts $nfspt $b - chkerr $? "mount_nfs $nfspt $b" + + # <CyberLeo> Hack to allow mounting of any filesystem type, not just NFS + # (thumbdrive w/glabel *hint hint*) + # ufs:/dev/label/conf (or nfs:server:/root -- server:/root works too, if + # server isn't named ufs, msdosfs, etc...) + eval $(echo ${nfspt} | sed -E 's/^([^:]+):(.*)$/fstype="\1" device="\2"/') + case "${fstype}" in + ufs|msdosfs|cd9660|nfs) ;; + # Backwards compatibility. Hopefully someoene didn't name their NFS server 'ufs'... + *) device="${fstype}:${device}"; fstype="nfs" ;; + esac + mount -t "${fstype}" "${device}" "${b}" + chkerr $? "mount -t ${fstype} ${device} ${b}" + # </CyberLeo> + to_umount="$b ${to_umount}" } </code> Patch rc.initdiskless to mount /conf from things other than nfs. (6.2 version) <code brush=bash> --- /etc/rc.initdiskless.orig Sun May 7 19:00:00 2006 +++ /etc/rc.initdiskless Mon Feb 12 08:03:03 2007 @@ -229,8 +229,25 @@ if [ `expr "$nfspt" : '\(.\)'` = "/" ]; then nfspt="${nfsroot}${nfspt}" fi - mount_nfs $nfspt /conf - chkerr $? "mount_nfs $nfspt /conf" + + # <CyberLeo> Hack to allow mounting of any filesystem type, not just NFS (thumbdrive w/glabel *hint hint*) + # ufs:/dev/label/conf (or nfs:server:/root -- server:/root works too, if server isn't named ufs, msdos, etc..) + eval $(echo ${nfspt} | sed -E 's/^([^:]+):(.*)$/fstype="\1" device="\2"/') + case "${fstype}" in + ufs) ;; + msdosfs) ;; + cd9660) ;; + nfs) ;; + # Backwards compatibility. Hopefully someone didn't name their NFS server 'msdos'... + *) device="${fstype}:${device}"; fstype="nfs" ;; + esac + + mount -t ${fstype} ${device} /conf + chkerr $? "mount -t ${fstype} ${device} /conf" + + # mount_nfs $nfspt /conf + # chkerr $? "mount_nfs $nfspt /conf" + + # </CyberLeo> to_umount="/conf" fi </code> Disable sendmail, and replace with something better <code brush=bash> cat >> /etc/rc.conf <<EOF sendmail_enable="NONE" sendmail_submit_enable="NO" sendmail_outbound_enable="NO" sendmail_msp_queue_enable="NO" EOF cat >> /etc/periodic.conf <<EOF daily_clean_hoststat_enable="NO" daily_status_mail_rejects_enable="NO" daily_status_include_submit_mailq="NO" daily_submit_queuerun="NO" EOF </code> I chose mini_sendmail (/usr/ports/mail/mini_sendmail) because a router doesn't need a full MTA.<br> <small>(I nfs-mounted /usr/ports from another server, so that I wouldn't clutter up the read-only image.)</small> <code brush=bash> cd /usr/ports/mail/mini_sendmail && make install clean cat > /etc/mail/mailer.conf <<EOF # Send outgoing mail to a smart relay using mini_sendmail sendmail /usr/local/bin/mini_sendmail -s<host> -p<port> send-mail /usr/local/bin/mini_sendmail -s<host> -p<port> EOF </code> Get things ready to be read-only. This includes moving all configuration directories together, and moving tmp into a writable space. This is best done in single-user mode. <code brush=bash> # Move /tmp into to-be-writable space mv /tmp/* /tmp/.??* /var/tmp/ rmdir /tmp ln -svf /var/tmp /tmp # Move 'local' configuration into /etc mv /usr/local/etc/rc.d /etc/rc.local # An arbitrary name for the local init scripts directory mv /usr/local/etc/* /etc/ rmdir /usr/local/etc ln -svf /etc /usr/local/etc echo 'local_startup="/etc/rc.local"' >> /etc/rc.conf # Tell rc where to find the new location of the local init scripts </code> Enable diskless mode (rc.initdiskless) <code brush=bash> touch /etc/diskless </code> Indicate where the config partition resides. Around here, it's also useful to modify /etc/fstab to set / as read-only, so that the changes are propagated properly. <code brush=bash> mkdir /conf glabel label -v conf /dev/ad4 newfs -U /dev/label/conf echo "ufs:/dev/label/conf" > /conf/diskless_remount echo "/dev/label/conf /conf ufs ro,noauto,noatime,sync 2 2" >> /etc/fstab </code> Set up the configuration. Ideally, this should be done on a 'clean' system with as many baseline enhancements as you want to keep. I.e. useful packages and ports, barebones passwd and group files, etc.. Additional configuration will be saved elsewhere, allowing a quick method of 'starting over' by simply removing the changed configuration. <code brush=bash> mount -w /conf mkdir -p /conf/{base,default}/{etc,var} echo "10240" > /conf/base/etc/md_size # 5MB echo "131072" > /conf/base/var/md_size # 64MB </code> There are two methods to handle the files. If you have ample space on your flash media, the file method can save accesses and writes to precious flash cells. If you are short on space, the archive method is tighter, but requires proper timestamping, and it writes all changed files each save. When staging the base archives or filesets, it's useful to remove files that are not necessary across reboots, or are autogenerated if absent. These include SSH keys from /etc/ssh/ssh_host_* (They will be autogenerated, then saved in /conf/default when saveconfig is run, so that nonconflicting copies can be made without too much trouble), /var/run/sudo/* (most likely to expire across reboots anyways) and so on. !! File Method <code brush=bash> cp -vpR /etc /conf/base/etc cp -vpR /var /conf/base/var for I in /conf/base/var/log; do cat /dev/null > ${I}; done # Zero out logfiles to save space rm -Rf /conf/base/var/tmp/* /conf/base/var/tmp/.??* # Remove stale temp files. Everything in here gets recreated on reboot anyways. </code> I haven't written a full saveconfig script for the file method, as I am using the archive method myself. However, the following rsync command line (requires rsync, from packages or ports) does the job, provided /conf is mounted. (It won't be left mounted after booting) <code brush=bash> # Copy all changed files, times, permissions, ownership from /etc and /var to /conf/default/{etc,var}, # and consider /conf/base/{etc,var} as additional comparison files. This will keep all the files in /conf/base from being # copied to /conf/default if they haven't been changed. rsync --archive --delete --hard-links --exclude='*/tmp/*' --compare-dest=/conf/base /etc /var /conf/default/ </code> !! Archive Method <code brush=bash> cd /tmp # Temporary staging area cp -vpR /var /tmp/ for I in var/log; do cat /dev/null > ${I}; done # Zero out logfiles to save space rm -Rf /conf/base/var/tmp/* /conf/base/var/tmp/.??* # Remove stale temp files. Everything in here gets recreated on reboot anyways. find var -print0 | xargs -0 touch -ht 200605080000.00 # Set the timestamps for all the unchanged files, so we can find the changed ones easier. tar zcpvf /conf/base/var.cpio.gz var # Structure must be relative to / for extraction to work properly. rm -Rf var cp -vpR /etc /tmp/ find etc -print0 | xargs -0 touch -ht 200605080000.00 touch -ht 200605080000.01 etc/diskless # This file is our baseline, as it will (hopefully) never change. tar zcpvf /conf/base/etc.cpio.gz etc rm -Rf etc </code> The saveconfig script (700 root:wheel in /sbin) <code brush=bash> #!/bin/sh echo -n "Saving configuration... " if [ ! -f /etc/diskless ] then echo "No flash setup detected." exit 1 fi # Mount /conf read-write, and remount if it already is. if [ $(grep -c "/conf" /etc/fstab) -gt 0 ] then mount -w /conf if [ $? -ne 1 ] then umount /conf mount -w /conf fi fi # Five oldest backups method: # Remove the oldest backup #[ -d /conf/backup/5 ] && rm -Rf /conf/backup/5 # Rotate the backups to make room #for I in 4 3 2 1 #do # dest=$(expr "${I}" + 1) # [ -d "/conf/backup/${I}" ] && mv "/conf/backup/${I}" "/conf/backup/${dest}" #done # Back up previous config and create space for the new one #mv /conf/default /conf/backup/1 # Unlimited history method: (make sure you have provisions for removing the old backups, or this can get HUGE! dest=$(expr "$(ls -1 /conf/backup | tail -n 1)" + 1) mv /conf/default "/conf/backup/${dest}" mkdir /conf/default cd / # Copy changed config files to /conf/default (anything younger than /etc/diskless) find etc var -type f -not -regex '.*/tmp/.*' -newer /etc/diskless -print0 | xargs -0 tar cpf - | tar xpCf /conf/default/ - # This can be changed to allow compressed configuration here as well: # find etc -type f -not -regex '.*/tmp/.*' -newer /etc/diskless -print0 | xargs -0 tar cpf /conf/default/etc.cpio.gz # find var -type f -not -regex '.*/tmp/.*' -newer /etc/diskless -print0 | xargs -0 tar cpf /conf/default/var.cpio.gz # Umount /conf afterwards if [ "$(mount |grep -c "/conf")" -gt 0 ] then umount /conf fi echo "Done!" </code> Save existing configuration on shutdown, in /etc/rc.local/saveconfig: <code brush=bash> #!/bin/sh # PROVIDE: saveconfig # REQUIRE: var # BEFORE: swap1 # KEYWORD: shutdown name="saveconfig" stop_cmd="saveconfig_stop" saveconfig_stop(){ /sbin/saveconfig } load_rc_config $name run_rc_command "$1" </code> !Things left to do: * Custom kernel with AltQ * Install DHCPd diskless * Install BIND diskless * Install OpenVPN diskless * PGP/GPG for logmail signing * Log handler scripts, to sign and mail at regular intervals ! Sundry Notes * /boot/kernel/*.symbols are unstripped copies of their similarly named counterparts. They are only needed for debugging. * While /boot/loader can load gzip-compressed kernel, it cannot load gzip-compressed modules. It just stalls. * Pare down root/etc because it cannot possibly need all that just to locate and extract the real etc. ** Perhaps just /etc/rc, /etc/diskless, and /etc/rc.initdiskless? * Copy SSH keys into tree, to avoid annoying SSH conflicts
