Winning the West...

This morning I discovered that the UPS had played games during the night and my server was thus off. Turns out the machine endured a hard power drop complements to the APC 1500 UPS. I must say, less than impressed is a euphemistic rating for that particular rack mount UPS. On the other hand, the little 620VA APC unit I have works fine.

Anyway, so the server didn't want to boot, one of the XFS filesystems was wedged. Setting aside the fsck.xfs == nothing glory, at times you do need to xfs_repair. I've hit this once before, having to repair and zero the metadata log to regain mount ability. No log zeroing was needed this time it turned out.

The trick here is that during booting of a Fedora machine, if a local filesystem complains during fsck -A loud enough then you get offered a root shell assuming you have the password. Unfortunately, every character typed seemed to be taken as the password, so if root has a password of length > 1 char then you can't login this way. And with 20 local filesystems and no indication of which was the problem this makes for hours of bliss. Forget rides in the mountains, nice scotch, or a good book, Russian roulette with fsck is the edge of the seat fun way to spend a few hours.

Anyways, having a 4gb live Fedora USB stick helped. Having access to rpms for XFS helped too, and those were extracted and installed on the USB stick should the need arise again. For those finding this through the libferris internet searches, the command you are after is

rpm2cpio foo.rpm | cpio -id

to extract the foo.rpm to paths relative to the current directory. Though if that command escapes you then having a wedged ext? or XFS not permitting you to boot up would likely be an epic problem. FWIW I also managed to wedge btrfs into a charming state recently due to a non UPS backed power failure. This is not a slur on any of these filesystems. Trying to get performance and durability in the face of arbitrary hardware and firmware paths is not hard, its rock hard^TM. There is no substitute for a good UPS if you care about your byteses.