Create LVM using Striping I/O for more IOPS performance

Updated on : May 12, 2021 Category : Linux/ Unix

LVM is a tool first introduced in Fedora Linux for Logical Volume Management which allows useful feature likes allocating disks, striping, mirroring and resizing logical volumes. LVM reduces the tedious work of manual disk management on the servers and Linux systems. This guide will explain how we can create LVM using Striping I/O for more IOPS performance. LVM Striping is one of the features which will write the data over multiple disks instead of constantly writing on a single Physical volume. This will help to reduce the Disk IO Wait, which caused huge load spikes on the system. If we see in terms of speed, LVM Striping roughly gives double the IOPS than LVM Linear.

Assume we have two disk drives and pointed to two physical volumes; if each physical volume is capable of 100 IOPS totally our volume group will get 200 IOPS with LVM Striping, but with LVM Linear, we will get only 100 IOPS for both disks in LVM and no matter how many disks we add in LVM.

Below is a basic diagrammatic and visual expression of LVM

         sda1  sdb1    (Physical Volumes on partitions or whole disks)
           \    /
            \  /
           VG-SSD      (Volume Group)
           / | \
         /   |   \
      home boot  tmp   (Logical Volumes)
       |     |    |
      ext4  ext3 ext4  (File Systems)

create lvm using striping i/o

LVM Striping features can be summed up to the following:

  •  Increases the performance of disks by increasing I/O.
  • Reduces Disk IO Wait by writing data over multiple disks simultaneously.
  • Disk fill-up can be reduced using striping over multiple disks.

Let’s now proceed to learn how we can create LVM using Striping I/O for more IOPS performance.

Requirments

  • Root access to Linux System with Additional Disks attached.

Procedure

  1. Check disks attached to the system using the command lsblk.
    We have a 6 x 2TB disk with max-supported IOPS of 600 attached to our system. With Striping I/O LVM, we will get 6*600 IOPS or at least a value of approximately 3400-3500.

    #> lsblk
NAME   MAJ:MIN  RM  SIZE  RO  TYPE  MOUNTPOINT
sda      8:0     0   100G    0   disk
sda1     8:1     0   100G    0   part        
sdb      8:16    0   1.9T  0   disk
sdc      8:32    0   1.9T  0   disk
sdd      8:48    0   1.9T  0   disk
sde      8:64    0   1.9T  0   disk
sdf      8:80    0   1.9T  0   disk
sdg      8:96    0   1.9T  0   disk

    Here, sda is the main 100GB disk and sda1 root partition, sd[b-g] are the attached HDD disks. sd[b-g] Means all disk sdb to sbg.

  2. Create the Physical Volume (PV) using the command pvcreate /dev/sd[b-g]. 
    #> pvcreate /dev/sd[b-g]
Physical volume "/dev/sdb" successfully created.
Physical volume "/dev/sdc" successfully created.
Physical volume "/dev/sdd" successfully created.
Physical volume "/dev/sde" successfully created.
Physical volume "/dev/sdf" successfully created.
Physical volume "/dev/sdg" successfully created.
  3. Confirm PV status using the command pvs. 
    #> pvs
PV VG Fmt Attr PSize PFree
/dev/vdb lvm2 --- 1.91t 1.91t
/dev/vdc lvm2 --- 1.91t 1.91t
/dev/vdd lvm2 --- 1.91t 1.91t
/dev/vde lvm2 --- 1.91t 1.91t
/dev/vdf lvm2 --- 1.91t 1.91t
/dev/vdg lvm2 --- 1.91t 1.91t
  4. Create the Volume Group (VG) using the command vgcreate -s 1M vgbz /dev/sd[b-g] -v. 
    #> vgcreate -s 1M vgbz /dev/sd[b-g] -v
Wiping internal VG cache
Wiping cache of LVM-capable devices
Wiping signatures on new PV /dev/sdb.
Wiping signatures on new PV /dev/sdc.
Wiping signatures on new PV /dev/sdd.
Wiping signatures on new PV /dev/sde.
Wiping signatures on new PV /dev/sdf.
Wiping signatures on new PV /dev/sdg.
Adding physical volume '/dev/sdb' to volume group 'vgbz'
Adding physical volume '/dev/sdc' to volume group 'vgbz'
Adding physical volume '/dev/sdd' to volume group 'vgbz'
Adding physical volume '/dev/sde' to volume group 'vgbz'
Adding physical volume '/dev/sdf' to volume group 'vgbz'
Adding physical volume '/dev/sdg' to volume group 'vgbz'
Archiving volume group "vgbz" metadata (seqno 0).
Creating volume group backup "/dev/vgbz" (seqno 1).
Volume group "vgbz" successfully created
  5. Confirm Volume Group (VG) status using the command vgdisplay -v. 
    #> vgdisplay -v  
  --- Volume group ---
  VG Name               vgbz
  System ID
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  2
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                1
  Open LV               1
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               11.46 TiB
  PE Size               1.00 MiB
  Total PE              12259804
  Alloc PE / Size       0 / 0
  Free  PE / Size       12259804 / 11.46 TiB
  VG UUID               erxpzQ-MRe9-Tivg-p7d5-eOZc-XCsj-d3v2eF
  6. Volume Group (VG) is now ready, its time to create Logical Volume (VG) using the command lvcreate -L 11.46T -I 16k -i 6 -n storage vgbz. 
    #> lvcreate -L 11.46T -I 128k -i 6 -n storage vgbz
Rounding size 11.40 TiB (12259804 extents) up to stripe boundary size 11.40 TiB (12259804 extents).
Logical volume "storage" created.

    -L Volume size
    -I Strip size
    -i Equivalent to the number of disks
    -n LV name
    vgbz volume group to use

    Check Logical Volume complete view using lvdisplay -m where you will see Stripes as 6, which is a total of 6 disks attached to Strip LVM along with a Strip size of 128 KB.

    #> lvdisplay -m
  --- Logical volume ---
  LV Path                /dev/vgdisk/storage
  LV Name                storage
  VG Name                vgdisk
  LV UUID                OyOR31-sIO3-FOPF-QJoK-pmGw-l4qB-v3l5LA
  LV Write Access        read/write
  LV Creation host, time server.basezap.com, 2021-04-30 01:16:09 +0530
  LV Status              available
  # open                 1
  LV Size                11.40 TiB
  Current LE             12259804
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192
  Block device           253:2

  --- Segments ---
  Logical extents 0 to 12259804:
    Type                striped
    Stripes             6
    Stripe size         128.00 KiB
  7. Now, we can format the Logical Volume using the command mkfs.ext4for the ext4 partition. 
    #> mkfs.ext4 /dev/mapper/vgdisk-storage
  8. Following mount point options can be used in /etc/fstab. 
    /dev/mapper/vgdisk-storage /storage ext4 defaults 1 2
  9. We can check IO Benchmarks using the command fio. 
    #> fio --randrepeat=1 --name=randrw --rw=randrw --direct=1 --ioengine=libaio --bs=16k --numjobs=10 --size=512M --runtime=60 --time_based --iodepth=64 --group_reporting

    Results:

    read : io=1467.8MB, bw=54679KB/s, iops=1782, runt= 60903msec
slat (usec): min=3, max=1362.7K, avg=148.74, stdev=8772.92
clat (msec): min=2, max=6610, avg=233.47, stdev=356.86
lat (msec): min=2, max=6610, avg=233.62, stdev=357.65

    write: io=1465.1MB, bw=54634KB/s, iops=1779, runt= 60903msec
slat (usec): min=4, max=1308.1K, avg=162.97, stdev=8196.09
clat (usec): min=551, max=5518.4K, avg=180989.83, stdev=316690.67
lat (usec): min=573, max=5526.4K, avg=181152.80, stdev=317708.30

We achieved desired IOPS of ~3.5k of the total by LVM Striping rather than the LVM Linear IOPS of 600 of the total, which can be seen in the above fio I/O benchmark results.

If you want to Extend EXT2/3/4 and XFS root partition, you can find the following guide helpful.
Extend EXT2/3/4 and XFS root partition without LVM