An Introduction to RAID in Linux

1. OverviewRAIDstands forRedundantArray ofInexpensive/IndependentDisks.We build our storage with redundancy — duplication of critical functions — so that no one part can fail and bring down our whole system.Because the data reads and writes are spread out over more than one disk, RAID can also provide us performance benefits.Modern filesystems like ZFS and btrfs have built-in RAID functionality. It’s also important we remember what RAID is not: it’s not a backup. For example, if our database gets wiped or corrupted, a mirrored RAID gives us two copies of our blank or broken database. A separate backup gives us a recovery option.In this tutorial, we’ll explore ways to use RAID in Linux.2. Types of RAIDRAID can be implemented with a dedicated hardware controller or entirely in software. Software RAID is more common today.We refer to different kinds of RAID via a standard numbering system of “raid levels”. The numbers do not refer to how many disks are used.RAID’s biggest advantage comes from the replication of data. Our data exists in more than one place on our RAID system, so we can avoid downtime during hardware failure. The replication may be via mirroring (keeping duplicate copies of everything) or parity (checksum calculations of our data).2.1. Hardware vs. SoftwareIn this guide, we’ll explore the RAID options built into Linux via software. Hardware RAID is beyond the scope of this article; just be aware that it is only useful on Linux in special cases, and we may need to turn it off in our computer’s BIOS.2.2. Striped And/or Mirrored (RAID 0, 1, or 10)RAID level 0 has an appropriate number: it has zero redundancy!Instead,in RAID 0, data is written across the drives, or “striped”.This means it can potentially be read from more than one drive concurrently. That can give us a real performance boost.But at the same time, now we have two drives that could fail, taking out all our data. So, RAID 0 is only useful if we want a performance boost but don’t care about long-term storage.We refer to RAID level 1 as “mirrored”because it is created with a pair of equal drives.Each time data is written to a RAID 1 device, it goes to both drives in the pair.Write performance is thus slightly slower, but read performance can be much faster as data is concurrently read from both disks.These two levels of RAID can be combined or nested, creating what’s called RAID 10 or just RAID 10. (There are other permutations, but RAID 10 is the most common.)We can create a RAID 10 device with four disks: one pair of disks in RAID 0, mirroring another pair of disks in RAID 0.This RAID of RAIDs attempts to combine RAID 0’s performance with RAID 1’s redundancy, to be both speedy and reliable.2.3. Parity (RAID 5 or RAID 6)Instead of storing complete copies of our data, we can save space by storing parity data. Parity allows our RAIDs to reconstruct data stored on failed drives.RAID 5 requires at least three equal-size drives to function. In practice, we can add several more, though rarely more than ten are used.RAID 5 sets aside one drive’s worth of space for checksum parity data. It is not all kept on one drive, however; instead, the parity data is striped across all of the devices along with the filesystem data.This means we usually want to build our RAID out of a set of drives of identical size and speed. Adding a larger drive won’t get us more space, as the RAID will just use the size of the smallest member. Similarly, the RAID’s performance will be limited by its slowest member.RAID 5 can recover and rebuild with no data loss if one drive dies. If two or more drives crash, we’ll have to restore the whole thing from backups.RAID 6 is similar to RAID 5 but sets aside two disks’ worth for parity data. That means a RAID 6 can recover from two failed members.RAID 5 gives us more usable storage than mirroring does, but at the price of some performance. A quick way to estimate storage is the total amount of equal-sized drives, minus one drive. For example, if we have 6 drives of 1 terabyte, our RAID 5 will have 5 terabytes of usable space. That’s 83%, compared to 50% of our drives were mirrored in RAID 1.At one time, server manufacturers considered RAID 5 the best practice in storage. It has fallen out of favor to some degree due to the so-called “RAID 5 write hole”, a problem addressed by next-generation filesystems and RAIDZ.3. Linux Kernel RAID(mdraid)Let’s create some new RAIDs with the mdadm tool.3.1. Your Basic RAIDWe’ll start with two identical disks or partitions, and create a striped RAID 0 device.First, let’s make sure we have the correct partitions. We don’t want to destroy something important:# lsblk -o NAME,SIZE,TYPE NAME SIZE TYPE sdb 931.5G disk └─sdb1 4G part sdc 931.5G disk └─sdc1 4G partCopyWe’ll use the mdadm command (multi-disk administrator):# mdadm --verbose --create /dev/md0 --level0 --raid-devices2 /dev/sdb1 /dev/sdc1 mdadm: chunk size defaults to 512K mdadm: Defaulting to version 1.2 metadata mdadm: array /dev/md0 started.CopyOur first RAID device has been created! Let’s break down the options we use withmdadm:–verbosetells us more about what is happening.–createtellsmdadmto create a new RAID device, naming it whatever we want (in this case,md0).–level0is our RAID level, as discussed above. Level 0 is just striped, with no redundancy.–raid-devices2letsmdadmknow to expect two physical disks for this array./dev/sdb1and/dev/sdc1are the two partitions included in our array of independent disks.So our RAID of partitions has been created, but like any device, it does not yet have a filesystem and it hasn’t been mounted.We can look at it again withlsblk:# lsblk -o NAME,SIZE,TYPE NAME SIZE TYPE sdb 931.5G disk └─sdb1 4G part └─md0 8G raid0 sdc 931.5G disk └─sdc1 4G part └─md0 8G raid0CopyNotice how themd0device is the size of the two partitions added together, as we’d expect from RAID 0.3.2. Managing Our RAIDWe also find useful information in/proc/mdstat:# cat /proc/mdstat Personalities : [raid0] md0 : active raid0 sdc1[1] sdb1[0] 1952448512 blocks super 1.2 512k chunks unused devices: noneCopyTo use this new RAID, we need to format it with a filesystem and mount it:# mkfs /dev/md0 mke2fs 1.46.2 (28-Feb-2021) Creating filesystem with 2094592 4k blocks and 524288 inodes Filesystem UUID: 947484b6-05ff-4d34-a0ed-49ee7c5eebd5 Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632 Allocating group tables: done Writing inode tables: done Writing superblocks and filesystem accounting information: done # mount /dev/md0 /mnt/myraid/ # df -h /mnt/myraid Filesystem Size Used Avail Use% Mounted on /dev/md0 7.8G 24K 7.4G 1% /mnt/myraidCopyLike any other filesystem besides ZFS, we would add a line to/etc/fstabto make this mount point permanent.If we want to boot from our RAID device (and we may not, to keep things simple), or otherwise allowmdadmto manage the array during startup or shutdown, we can append our array’s info to an optional/etc/mdadm/mdadm.conffile:# mdadm --detail --scan ARRAY /dev/md1 metadata1.2 spares1 namesalvage:1 UUID0c32834c:e5491814:94a4aa96:32d87024CopyAnd if we want to take down our raid, we can usemdadmagain:# mdadm -S /dev/md0 mdadm: stopped /dev/md0CopyWe can create similar RAIDs with variations of the–leveland–raid-devicesoptions.For example, we could create a 5-disk RAID 5:# mdadm --verbose --create /dev/md1 --level5 --raid-devices5 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1 mdadm: layout defaults to left-symmetric mdadm: layout defaults to left-symmetric mdadm: chunk size defaults to 512K mdadm: size set to 4189184K Continue creating array? y mdadm: Defaulting to version 1.2 metadata mdadm: array /dev/md1 started.CopyThen, we canmkfsandmountour latest RAID.3.3. Failed Drives and Hot SparesWhat would happen to our new RAID 5 if one of the drives failed? Let’s simulate that event withmdadm:# mdadm /dev/md1 -f /dev/sdc1 mdadm: set /dev/sdc1 faulty in /dev/md1CopyNow, what does/proc/mdstattell us? Let’s take a look:# cat /proc/mdstat Personalities : [raid0] [raid6] [raid5] [raid4] md1 : active raid5 sdf1[5] sde1[3] sdd1[2] sdb1[1] sdc1[0](F) 16756736 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/4] [_UUUU] unused devices: noneCopyHere, we see the partition we selected, marked (F) for failed.We can also askmdadmfor more details of our array:# mdadm --detail /dev/md1 /dev/md1: Version : 1.2 Creation Time : Tue Aug 10 14:52:59 2021 Raid Level : raid5 Array Size : 16756736 (15.98 GiB 17.16 GB) Used Dev Size : 4189184 (4.00 GiB 4.29 GB) Raid Devices : 5 Total Devices : 5 Persistence : Superblock is persistent Update Time : Tue Aug 10 14:59:20 2021 State : clean, degraded Active Devices : 4 Working Devices : 4 Failed Devices : 1 Spare Devices : 0 Layout : left-symmetric Chunk Size : 512K Consistency Policy : resync Name : salvage:1 (local to host salvage) UUID : 0c32834c:e5491814:94a4aa96:32d87024 Events : 24 Number Major Minor RaidDevice State - 0 0 0 removed 1 8 34 1 active sync /dev/sdb1 2 8 35 2 active sync /dev/sdd1 3 8 36 3 active sync /dev/sde1 5 8 37 4 active sync /dev/sdf1 0 8 33 - faulty /dev/sdc1CopyOur RAID is still going strong. A user should not be able to tell any difference. But we can see it’s in a “degraded” state, so we need to replace that faulty hard drive.Let’s say we have a replacement for our dead drive. It should be identical to the originals.We can remove our faulty drive and add a new one. We should remember that the/dev/sd*list of devices will sometimes change if the hardware changes, so double-check withlsblk.First, we remove our faulty drive from the array:# mdadm /dev/md1 --remove /dev/sdc1 mdadm: hot removed /dev/sdc1 from /dev/md1CopyNext, we physically replace our drive and add the new one. (This is where hot-swappable drive hardware saves us a lot of time!)We can look at/proc/mdstatto watch the RAID automatically rebuild:# mdadm /dev/md1 --remove /dev/sdc1 mdadm: hot removed /dev/sdc1 from /dev/md1 rootsalvage:~# mdadm /dev/md1 --add /dev/sdc1 mdadm: added /dev/sdc1 # cat /proc/mdstat Personalities : [raid0] [raid6] [raid5] [raid4] md1 : active raid5 sdc1[6] sdf1[5] sde1[3] sdd1[2] sdb1[1] 16756736 blocks super 1.2 level 5, 512k chunk, algorithm 2 [5/4] [_UUUU] [..................] recovery 10.7% (452572/4189184) finish3.4min speed18102K/secCopyIf uptime is really important, we can add a dedicated spare drive to letmdadmautomatically switch over to:# mdadm /dev/md1 --add-spare /dev/sdg1 mdadm: added /dev/sdg1CopyIt might be worth it; we can weigh the time and money involved.Let’s check on our array again:# mdadm --detail /dev/md1 | grep spare 7 8 38 - spare /dev/sdg1CopyFive disks are striped with data and parity. One disk is unused, just waiting to be needed.4. The Logical Volume ManagerMost modern Linux filesystems are no longer created directly on a drive or a partition, but on a logical volume created with the LVM.Briefly, LVM combines Physical Volumes (drives or partitions) into Volume Groups. Volume Groups are pools from which we can create Logical Volumes. We can put filesystems onto these Logical Volumes.RAID comes into it during the creation of Logical Volumes. These may be linear, striped, mirrored, or a more complex parity configuration.We should note that creating a RAID LVM Logical Volume uses Linux kernel RAID (mdraid). If we want the convenience of LVM, being able to expand Volume Groups and resize Logical Volumes, we can have it along with the reliability of simple mdraid.But if LVM sounds like too much added complexity, we can always stick with mdraid on our physical drives.Yet another common option is creating our RAID devices with mdadm and then using them as PVs with LVM.4.1. Telling LVM to Use Our Volumes in a RAIDLVM RAIDs are created at the logical volume level.So that means we need to have first created partitions, used pvcreate to tag them as LVM physical volumes, and used vgcreate to put them into a volume group. In this example, we’ve called the volume groupraid1vg0.The RAID creation step specifies the type of RAID and how many disks to use for mirroring or striping. We don’t need to specify each physical volume. We can let LVM handle all of that:# lvcreate --mirrors 1 --type raid1 -l 100%FREE -n raid01v0 raid1vg0 Logical volume raid01v0 created. # mkfs.ext4 /dev/raid1vg0/raid01v0CopyAs usual, we then format and mount our new RAID volume. If we want a system that handles all of that automatically, we have ZFS.5. Integrated Filesystem RAID with ZFS orbtrfsWe won’t cover the details of next-generation filesystems in this article, but many of the concepts from software RAID and LVM translate over.ZFS uses “vdevs”, virtual devices, much as LVM uses Volume Groups. These vdevs may be physical disks, mirrors, raidz variants (ZFS’s take on RAID 5), or as of OpenZFS 2.1, draid.For example, we can create a RAID 1 mirror zpool:# zpool create -f demo mirror /dev/sdc /dev/sddCopyZFS handles everything else for us, formatting and mounting our new volume pool under/demo.The equivalent in btrfs is:# mkfs.btrfs -L demo -d raid1 /dev/sdc /dev/sddCopyOne major limitation of btrfs is that it does not support RAID5 or RAID 6, at least not reliably. So, we’ll keep that far away from production systems.These next-generation filesystems take care of many of the details of RAID and volume management. In addition, they provide much greater data integrity through block-level checksums.Although they are a whole other topic, we may solve more of our storage problems by investigating ZFS or btrfs.6. Further ReadingThe Linux RAID wiki goes into depth about current issues and best practices.The ArchLinux wiki explains LVM in great detail.The OpenZFS project details how to get started using ZFS on Linux.7. ConclusionWe use RAID for reliability and to limit downtime.In this article, we’ve looked at the building blocks of Linux software RAID (md). We’ve also considered some more complex and advanced additions.There are more details to consider in the day-to-day monitoring and maintenance of our RAID, but this gets us started.