Save of the Century

Being an IT professional, I have to hang my head in shame and admit to you that I nearly lost data today. I was resigned to restoring from the last full backup I took, which was ages ago. In the end a last ditch attempt came to fruition.

It all started innocently enough with the upgrade of the main server from FC3 to FC5.  I hate upgrading TurboTas so much that i missed out FC4 completly.   The upgrade began and (as with FC3), the installer baulked at my lvm drives.  not at all of them, just the striped 160s.  I guess this happened coz i use them native, not with any partition table. 

The installer warned me that it wouldzero the partition table on hdc and hdd if i continued.  I selected cancel and the install proceeded anyway.  Lo and behold, after the upgrade Linux booted fine but no LVM on the two big drives.

All LVM tools suggested that there was nothing LVMish about the two drives which left me a bit stumped.  The net was symathetic, but no actual help.  I tried vgrestorecfg, but this reported that the vg memebers were not correct.  In desperation, I wrote new signatures to the PVs using pvcreate.  I had to use -ff as lvm already thought they were in a vg.  

Next I tried the vgrestorecfg tool again, but now was told that the signatures were wrong.  In desperation I edited the backup file and changed the drive signates to those now reported by pvdisplay /dev/hdc and hdd.  Amazingly, vgrestorecfg worked. 

A few minutes later and I had mounted the drives and discovered that I seem to have lost nothing at all other than a few clumps of hair. 

The lesson, dear readers is to do those backups even if they are manual and a pain in the butt.  I've had my warning: you may not be so lucky.