TL;DR: Always have an up-to-date backup of your system, no matter how much you trust it. This type of recovery is a poor second option..
- Use testdisk to recover lvm backup metadata from the /etc directory, otherwise hexedit to scrape the information manually from the disk
- Use the pvcreate and vgcfgrestore commands to restore the lvm metadata
- Fix the MBR/grub if it was your boot disk
Background
Last year I upgraded and cross-graded my home PC’s disks from the following configuration:
- 1x128GB SSD (half Linux LVM, half Windows 7)
- 1x256GB HDD (Linux LVM)
- 1x512GB HDD (NTFS data drive)
to the following:
- 1x128GB SSD (Windows 10)
- 1x256GB SSD (Linux LVM)
- 1x512GB HDD (NTFS data drive)
To do so, I moved the LVM partitions from the 128GB SSD and 256GB HDD to the new 256GB SSD (*), then reformatted and reinstalled Windows on the 128GB SSD (Windows being only used for Elite Dangerous, Linux really being the primary OS I use).
(*) I can’t remember the precise steps. Definitely at one stage the 256GB HDD contained the entire Linux VG.
Disaster
Fast forward to this year, and the 256GB SSD recently packed it in. Panick ensues.Read on..
So naturally, first steps first, try to take a snapshot of the disk. Initially an rsync of the entire filesystem, then a dd, then a ddrescue. Unfortunately it looks like the SSD is too far gone; only a few hundred MB into the copy the drive just goes offline with hard errors on the ATA link.
Next steps; try to find the original HDD amongst my pile of old drives. I didn’t hold out much hope for this as I had recently gathered them all and done secure wipes in preparation to selling/freecycling/dumping the disks.
Sure enough, disk after disk turned out to be blank with either nothing on it, random garbage, or remnants of other systems.
Hope
Finally I found a 256GB disc which was empty, but actually partitioned. Closer examination with a hex editor showed that it used to contain an LVM VG created by my lost machine..
First steps: take a dd of the disk. Taking absolutely no chances with this one!
Second steps: take a copy of the disk image to work with.
Fig 1. The disk was partitioned as follows:# fdisk -l /data/archives/DriveBackups/20170509_urutu/20170509_disk1.img Disk /data/archives/DriveBackups/20170509_urutu/20170509_disk1.img: 233.8 GiB, 251000193024 bytes, 490234752 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x03080307 Device Boot Start End Sectors Size Id Type /data/archives/DriveBackups/20170509_urutu/20170509_disk1.img1 * 2048 490233855 490231808 233.8G ef EFI (FAT
Recovery
Firstly I found this article for restoring the LVM on a disk which is no longer configured as a PV. This proved to be a very useful starting point!
Initially I followed the suggestions and used hexedit to save the various lvm metadata dumps on the disk (hint: use search), but after dumping half a dozen or so I got bored.
Fig 2: LVM details found using hexedit
001065F0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00106600 76 67 31 20 7B 0A 69 64 20 3D 20 22 7A 63 73 41 vg1 {.id = "zcsA
00106610 4A 64 2D 57 62 6E 79 2D 75 6C 70 49 2D 6C 4C 4B Jd-Wbny-ulpI-lLK
00106620 62 2D 4D 78 77 55 2D 52 61 71 4A 2D 69 72 57 6C b-MxwU-RaqJ-irWl
00106630 59 68 22 0A 73 65 71 6E 6F 20 3D 20 31 32 0A 66 Yh".seqno = 12.f
00106640 6F 72 6D 61 74 20 3D 20 22 6C 76 6D 32 22 20 23 ormat = "lvm2" #
00106650 20 69 6E 66 6F 72 6D 61 74 69 6F 6E 61 6C 0A 73 informational.s
00106660 74 61 74 75 73 20 3D 20 5B 22 52 45 53 49 5A 45 tatus = ["RESIZE
00106670 41 42 4C 45 22 2C 20 22 52 45 41 44 22 2C 20 22 ABLE", "READ", "
00106680 57 52 49 54 45 22 5D 0A 66 6C 61 67 73 20 3D 20 WRITE"].flags =
00106690 5B 5D 0A 65 78 74 65 6E 74 5F 73 69 7A 65 20 3D [].extent_size =
001066A0 20 38 31 39 32 0A 6D 61 78 5F 6C 76 20 3D 20 32 8192.max_lv = 2
001066B0 35 36 0A 6D 61 78 5F 70 76 20 3D 20 32 35 36 0A 56.max_pv = 256.
001066C0 6D 65 74 61 64 61 74 61 5F 63 6F 70 69 65 73 20 metadata_copies
001066D0 3D 20 30 0A 0A 70 68 79 73 69 63 61 6C 5F 76 6F = 0..physical_vo
001066E0 6C 75 6D 65 73 20 7B 0A 0A 70 76 30 20 7B 0A 69 lumes {..pv0 {.i
001066F0 64 20 3D 20 22 57 31 48 47 68 77 2D 32 6C 44 56 d = "W1HGhw-2lDV
00106700 2D 6C 4E 42 77 2D 36 69 42 76 2D 38 7A 57 67 2D -lNBw-6iBv-8zWg-
--- 20170509_disk1.img --0x106600/0x3A70C70000--------------------------
I had already used testdisk on the disk and found a bunch of partitions – obviously not real ones as they are LVM logical volumes so testdisk can’t be used to restore them. But I re-ran testdisk and used it to successfully dump out the /etc/lvm/backup folder. Now I had access to the last valid configuration of the LVM volume group in the system.
Fig 3: Showing partitions as found by testdisk:TestDisk 6.14, Data Recovery Utility, July 2013 Christophe GRENIER <grenier@cgsecurity.org> http://www.cgsecurity.org Disk /data/archives/DriveBackups/20170509_urutu/20170509_disk1.img - 251 GB / 23 Partition Start End Size in sectors > Linux 0 65 2 32 227 3 524288 Linux Swap 32 227 4 555 14 35 8388608 Linux 555 14 36 1207 195 12 10485760 Linux 1207 195 13 3165 227 6 31457280 Linux 3165 227 7 3818 152 46 10485760 Linux 3818 152 47 4471 78 23 10485760 Linux 4471 78 24 5776 184 40 20971520 Linux 5124 4 1 18178 46 44 209715200 Structure: Ok. Use Up/Down Arrow keys to select partition. Use Left/Right Arrow keys to CHANGE partition characteristics: *=Primary bootable P=Primary L=Logical E=Extended D=Deleted Keys A: add partition, L: load backup, T: change type, P: list files, Enter: to continue ext4 blocksize=1024 Sparse superblock, 268 MB / 256 MiB
Fig 4: Showing some of the contents of the /etc directory in the root partition (3rd partition in the list above):
TestDisk 6.14, Data Recovery Utility, July 2013 Christophe GRENIER <grenier@cgsecurity.org> http://www.cgsecurity.org Linux 555 14 36 1207 195 12 10485760 Directory /etc Previous >-rw-r--r-- 0 0 1222 1-Aug-2014 23:32 fstab drwxr-xr-x 0 0 4096 24-Jul-2014 02:21 ld.so.conf.d -rw-r--r-- 0 0 11 5-May-2013 11:00 debian_version drwxr-xr-x 0 0 4096 29-Jul-2014 21:31 default drwxr-xr-x 0 0 4096 29-Jun-2014 12:28 profile.d -rw-r--r-- 0 0 35 6-May-2013 11:33 issue lrwxrwxrwx 0 0 21 20-Jul-2014 11:11 os-release drwxr-xr-x 0 0 4096 1-Aug-2014 22:09 cron.daily -rw-r--r-- 0 0 1863 5-Mar-2014 04:18 bash.bashrc drwxr-xr-x 0 0 4096 27-Apr-2014 13:58 apt -rw-r--r-- 0 0 956 5-Feb-2014 04:41 mke2fs.conf -rw-r--r-- 0 0 367 19-May-2014 10:50 bindresvport.blacklist drwxr-xr-x 0 0 4096 1-Aug-2014 22:09 pam.d -rw-r--r-- 0 0 552 26-Jul-2008 21:44 pam.conf Next Use Left arrow to go back, Right to change directory, h to hide deleted files q to quit, : to select the current file, a to select all files C to copy the selected files, c to copy the current file
Mounting the disk-image partition
The disk contained a single partition starting at offset 2048 of type EFI, which matched the physical volume information in the LVM metadata I had recovered.
First I used cfdisk to change the partition type to Linux LVM (8e), although not sure that was entirely necessary, and then mounted the partition using losetup and kpartx, which resulted in a device node for the partition in /dev/mapper/loop0p1. This device node will be used later with the lvm commands.
# cfdisk disk.img# losetup /dev/loop0 disk.img# kpartx -a /dev/loop0
Recovering the Volume Group
HINT: Use the -t (test) parameter to try out the commands first!
Now it was a fairly simple procedure to recreate the volume group.
Firstly, this is an excerpt of the data contained in the lvm backup, take a note of the volume group name, and the physical volume id.
Fig 5. LVM Text Format Volume Group
# Generated by LVM2 version 2.02.106(2) (2014-04-10): Fri Aug 1 22:29:02 2014
contents = "Text Format Volume Group"
version = 1
description = "Created *after* executing 'vgreduce vg1 /dev/sda1'"
creation_host = "urutu" # Linux urutu 3.14-1-amd64 #1 SMP Debian 3.14.12-1 (2014-07-11) x86_64
creation_time = 1406928542 # Fri Aug 1 22:29:02 2014
vg1 {
id = "zcsAJd-Wbny-ulpI-lLKb-MxwU-RaqJ-irWlYh"
seqno = 51
format = "lvm2" # informational
status = ["RESIZEABLE", "READ", "WRITE"]
flags = []
extent_size = 8192 # 4 Megabytes
max_lv = 256
max_pv = 256
metadata_copies = 0
physical_volumes {
pv0 {
id = "W1HGhw-2lDV-lNBw-6iBv-8zWg-yLGZ-KXNUm8"
device = "/dev/sdb1" # Hint only
status = ["ALLOCATABLE"]
flags = []
dev_size = 490231808 # 233.761 Gigabytes
pe_start = 2048
pe_count = 59842 # 233.758 Gigabytes
}
}
logical_volumes {
boot {
id = "C6YCzW-itiQ-R7uo-xEZD-0J2B-XSWc-9BzET3"
status = ["READ", "WRITE", "VISIBLE"]
flags = []
creation_host = "mint"
Take a note of the physical volume’s id (highlighted above), and plug it into the following command to re-create the physical volume.
# pvcreate -ff -u W1HGhw-2lDV-lNBw-6iBv-8zWg-yLGZ-KXNUm8 --restorefile etc/lvm/backup/vg1 /dev/mapper/loop0p1
Use pvscan to verify the physical volume information. You can use pvdisplay to verify the ID matches before continuing! In my case it looked like this:
# pvscan PV /dev/sdc1 VG vg1 lvm2 [233.76 GiB / 89.51 GiB free] Total: 1 [233.76 GiB] / in use: 1 [233.76 GiB] / in no VG: 0 [0 ] # pvdisplay /dev/mapper/loop0p1 --- Physical volume --- PV Name /dev/mapper/loop0p1 VG Name PV Size 233.76 GiB Allocatable NO PE Size 0 Total PE 0 Free PE 0 Allocated PE 0 PV UUID W1HGhw-2lDV-lNBw-6iBv-8zWg-yLGZ-KXNUm8
Now rewrite the volume group information using the following command:
# vgcfgrestore -f etc/lvm/backup/vg1 -v vg1
You can now verify your volume group exists using lvdisplay, check the filesystems using fsck, and mount them as normal, for example:
# mount /dev/mapper/vg1-root /mnt/recovered
