6.3. Recovering from LVM Mirror Failure
This section provides an example of recovering from a situation where one leg of an LVM mirrored volume fails because the underlying device for a physical volume goes down. When a mirror leg fails, LVM converts the mirrored volume into a linear volume, which continues to operate as before but without the mirrored redundancy. At that point, you can add a new disk device to the system to use as a replacement physical device and rebuild the mirror.
The following command creates the physical volumes which will be used for the mirror.
[[email protected] ~]# pvcreate /dev/sd[abcdefgh][12]
Physical volume "/dev/sda1" successfully created
Physical volume "/dev/sda2" successfully created
Physical volume "/dev/sdb1" successfully created
Physical volume "/dev/sdb2" successfully created
Physical volume "/dev/sdc1" successfully created
Physical volume "/dev/sdc2" successfully created
Physical volume "/dev/sdd1" successfully created
Physical volume "/dev/sdd2" successfully created
Physical volume "/dev/sde1" successfully created
Physical volume "/dev/sde2" successfully created
Physical volume "/dev/sdf1" successfully created
Physical volume "/dev/sdf2" successfully created
Physical volume "/dev/sdg1" successfully created
Physical volume "/dev/sdg2" successfully created
Physical volume "/dev/sdh1" successfully created
Physical volume "/dev/sdh2" successfully created
The following commands creates the volume group vg and the mirrored volume groupfs.
[[email protected] ~]# vgcreate vg /dev/sd[abcdefgh][12]
Volume group "vg" successfully created
[[email protected] ~]# lvcreate -L 750M -n groupfs -m 1 vg /dev/sda1 /dev/sdb1 /dev/sdc1
Rounding up size to full physical extent 752.00 MB
Logical volume "groupfs" created
You can use the lvs command to verify the layout of the mirrored volume and the underlying devices for the mirror leg and the mirror log. Note that in the first example the mirror is not yet completely synced; you should wait until the Copy% field displays 100.00 before continuing.
[[email protected] ~]# lvs -a -o +devices
LV VG Attr LSize Origin Snap% Move Log Copy% Devices
groupfs vg mwi-a- 752.00M groupfs_mlog 21.28 groupfs_mimage_0(0),groupfs_mimage_1(0)
[groupfs_mimage_0] vg iwi-ao 752.00M /dev/sda1(0)
[groupfs_mimage_1] vg iwi-ao 752.00M /dev/sdb1(0)
[groupfs_mlog] vg lwi-ao 4.00M /dev/sdc1(0)
[[email protected] ~]# lvs -a -o +devices
LV VG Attr LSize Origin Snap% Move Log Copy% Devices
groupfs vg mwi-a- 752.00M groupfs_mlog 100.00 groupfs_mimage_0(0),groupfs_mimage_1(0)
[groupfs_mimage_0] vg iwi-ao 752.00M /dev/sda1(0)
[groupfs_mimage_1] vg iwi-ao 752.00M /dev/sdb1(0)
[groupfs_mlog] vg lwi-ao 4.00M i /dev/sdc1(0)
In this example, the primary leg of the mirror /dev/sda1 fails. Any write activity to the mirrored volume causes LVM to detect the failed mirror. When this occurs, LVM converts the mirror into a single linear volume. In this case, to trigger the conversion, we execute a dd command
[[email protected] ~]# dd if=/dev/zero of=/dev/vg/groupfs count=10
10+0 records in
10+0 records out
You can use the lvs command to verify that the device is now a linear device. Because of the failed disk, I/O errors occur.
[[email protected] ~]# lvs -a -o +devices
/dev/sda1: read failed after 0 of 2048 at 0: Input/output error
/dev/sda2: read failed after 0 of 2048 at 0: Input/output error
LV VG Attr LSize Origin Snap% Move Log Copy% Devices
groupfs vg -wi-a- 752.00M /dev/sdb1(0)
At this point you should still be able to use the logical volume, but there will be no mirror redundancy.
To rebuild the mirrored volume, you replace the broken drive and recreate the physical volume. If you use the same disk rather than replacing it with a new one, you will see "inconsistent" warnings when you run the pvcreate command.
[[email protected] ~]# pvcreate /dev/sda[12]
Physical volume "/dev/sda1" successfully created
Physical volume "/dev/sda2" successfully created
[[email protected] ~]# pvscan
PV /dev/sdb1 VG vg lvm2 [67.83 GB / 67.10 GB free]
PV /dev/sdb2 VG vg lvm2 [67.83 GB / 67.83 GB free]
PV /dev/sdc1 VG vg lvm2 [67.83 GB / 67.83 GB free]
PV /dev/sdc2 VG vg lvm2 [67.83 GB / 67.83 GB free]
PV /dev/sdd1 VG vg lvm2 [67.83 GB / 67.83 GB free]
PV /dev/sdd2 VG vg lvm2 [67.83 GB / 67.83 GB free]
PV /dev/sde1 VG vg lvm2 [67.83 GB / 67.83 GB free]
PV /dev/sde2 VG vg lvm2 [67.83 GB / 67.83 GB free]
PV /dev/sdf1 VG vg lvm2 [67.83 GB / 67.83 GB free]
PV /dev/sdf2 VG vg lvm2 [67.83 GB / 67.83 GB free]
PV /dev/sdg1 VG vg lvm2 [67.83 GB / 67.83 GB free]
PV /dev/sdg2 VG vg lvm2 [67.83 GB / 67.83 GB free]
PV /dev/sdh1 VG vg lvm2 [67.83 GB / 67.83 GB free]
PV /dev/sdh2 VG vg lvm2 [67.83 GB / 67.83 GB free]
PV /dev/sda1 lvm2 [603.94 GB]
PV /dev/sda2 lvm2 [603.94 GB]
Total: 16 [2.11 TB] / in use: 14 [949.65 GB] / in no VG: 2 [1.18 TB]
Next you extend the original volume group with the new physical volume.
[[email protected] ~]# vgextend vg /dev/sda[12]
Volume group "vg" successfully extended
[[email protected] ~]# pvscan
PV /dev/sdb1 VG vg lvm2 [67.83 GB / 67.10 GB free]
PV /dev/sdb2 VG vg lvm2 [67.83 GB / 67.83 GB free]
PV /dev/sdc1 VG vg lvm2 [67.83 GB / 67.83 GB free]
PV /dev/sdc2 VG vg lvm2 [67.83 GB / 67.83 GB free]
PV /dev/sdd1 VG vg lvm2 [67.83 GB / 67.83 GB free]
PV /dev/sdd2 VG vg lvm2 [67.83 GB / 67.83 GB free]
PV /dev/sde1 VG vg lvm2 [67.83 GB / 67.83 GB free]
PV /dev/sde2 VG vg lvm2 [67.83 GB / 67.83 GB free]
PV /dev/sdf1 VG vg lvm2 [67.83 GB / 67.83 GB free]
PV /dev/sdf2 VG vg lvm2 [67.83 GB / 67.83 GB free]
PV /dev/sdg1 VG vg lvm2 [67.83 GB / 67.83 GB free]
PV /dev/sdg2 VG vg lvm2 [67.83 GB / 67.83 GB free]
PV /dev/sdh1 VG vg lvm2 [67.83 GB / 67.83 GB free]
PV /dev/sdh2 VG vg lvm2 [67.83 GB / 67.83 GB free]
PV /dev/sda1 VG vg lvm2 [603.93 GB / 603.93 GB free]
PV /dev/sda2 VG vg lvm2 [603.93 GB / 603.93 GB free]
Total: 16 [2.11 TB] / in use: 16 [2.11 TB] / in no VG: 0 [0 ]
Convert the linear volume back to its original mirrored state.
[[email protected] ~]# lvconvert -m 1 /dev/vg/groupfs /dev/sda1 /dev/sdb1 /dev/sdc1
Logical volume mirror converted.
You can use the lvs command to verify that the mirror is restored.
[[email protected] ~]# lvs -a -o +devices
LV VG Attr LSize Origin Snap% Move Log Copy% Devices
groupfs vg mwi-a- 752.00M groupfs_mlog 68.62 groupfs_mimage_0(0),groupfs_mimage_1(0)
[groupfs_mimage_0] vg iwi-ao 752.00M /dev/sdb1(0)
[groupfs_mimage_1] vg iwi-ao 752.00M /dev/sda1(0)
[groupfs_mlog] vg lwi-ao 4.00M /dev/sdc1(0)
