ugh

Last night chicken seemed very slow for command line use, but I put it down to too much load. This morning I saw all the nightly tests had failed with various missing file errors. Then I found I couldn't log in. Yep, one of the two 250 GB disks that I have lvmed together failed. SMART is whining about it now, but didn't give me any advance warning.

Most of the stuff on there was media files and not backed up, so I'm currently rsyncing hundreds of gigabytes of data that seems ok to a different machine, then I'll yank the disk. I'm not looking forward to reinstalling, or re-downloading my Debian mirror, or to living with 250 gb less disk space on my file server until I can get the disk replaced. It would probably be prudent to add a third disk and go for raid 5, but I doubt I will. At least I have all the important info, including /etc, safely in offsite svn repositories.

Also, Dani is having a really hard time and sounds completly torn up about it. I knew I shouldn't get out of bed today..