Linux device-mapper has a module named 'snapshot', which affords Copy-on-Write layers to block devices.

Stack a COW atop a disk, and test out your damaging changes on that instead. If it doesn't work, throw away the COW and try again.

While syncing a raid5 to a new disk, the server's video card failed and, inexplicably, disabled one of the SATA ports, causing two disks to fail out of the RAID5.

Assembling the raid with --force should have ignored the 'stale' status of one of the members, and rebuilt it. This failed for some reason.

The only way to recover from this would be to recreate the array with the exact same parameters as the original, and leave out the disk that never finished synchronizing.

for I in a b c d e
do
  dd if=/dev/zero of=sd${I}.cow bs=1048576 count=1024
  losetup -a sd${I}.cow
done

echo "0 976773168 snapshot /dev/sda /dev/loop1 n 4096" | dmsetup create sda.cow
echo "0 976773168 snapshot /dev/sdb /dev/loop2 n 4096" | dmsetup create sdb.cow
echo "0 976773168 snapshot /dev/sdc /dev/loop3 n 4096" | dmsetup create sdc.cow
echo "0 976773168 snapshot /dev/sdd /dev/loop4 n 4096" | dmsetup create sdd.cow
echo "0 976773168 snapshot /dev/sde /dev/loop5 n 4096" | dmsetup create sde.cow

mdadm --create /dev/md/0 --assume-clean --metadata=1 --level=5 --chunk=1024 --raid-devices=5 /dev/mapper/sde.cow /dev/mapper/sdd.cow missing /dev/mapper/sdb.cow /dev/mapper/sda.cow

In the future, when swapping a functional non-failing RAID member for another, the following procedure would greatly reduce the problem window:

Possible problems with the above:

PHP Warning

Warning: "preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead"

Warning: "preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead:"

Warning: "preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead"

Warning: "preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead"

Warning: "preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead"

Warning: "preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead"

Warning: "preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead"

Warning: "preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead"

Warning: "preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead:"