Tips on setting up raid10 on *buntu

Question

I'd like to set up a software raid 10 array (4x2TB) for a workstation using a fresh install, but I'm finding non applicable/conflicting/old resources online. I'd like to get the community's advice on setting up such an array as there appears to be a plethora of possible configurations.

Specifically, I'll be using the workstation for image analysis (i.e., typical file sizes from several hundred MB-several GB), so if there are optimizations that can be made for such purposes that would be great.

Cheers

dobey · Accepted Answer · 2012-12-14T16:22:04.417

I don't remember exactly what resources I followed when setting up my RAID on my server, but I think this article was the main point of information. Some important things:

Use mdadm and not dmraid.
Use /dev/disk/by-id/ paths to point at the disks, instead of /dev/sda etc… It's easier to map them to the physical devices, in case you need to replace a disk or such.
Be patient. At first I thought my RAID would be ready after the 5 hours initial setup time. Then it took another day to rebuild itself and actually be ready. (4x2TB)
Use a separate disk/partition/RAID for primary OS installs. It's easier to manage everything if you keep the large data RAID separate from primary OS and its data. Much easier to recover a small OS disk, than rebuild a huge multi-terabyte array, if something goes bad on the OS side.

gertvdijk · Answer 2 · 2012-12-15T00:10:39.067

With RAID10 in the given situation I see only two variables candidate for optimization:

Chunk size

Set it to something larger than the default of 512KiB to minimize the overhead for linear reads/writes of large files. You should try it on a small partition on your machine to see what gives the best performance, e.g. 1MB, 2MB, 5MB, 10MB...
Near vs Far layout

comparable to RAID1+0 vs RAID0+1. Far is a bit faster as the performance for reading is more like RAID0. Yet a Near layout is the default because it has a slightly higher chance of surviving the unlikely event that all mirrored disks are broken (some probability math here). Some more visual idea of the differens is below, happily stolen from SLES mdadm documentation:

Near looks like
```
sda1 sdb1 sdc1 sdd1
  0    0    1    1
  2    2    3    3
  4    4    5    5
```
Far looks like
```
sda1 sdb1 sdc1 sdd1
  0    1    2    3
  4    5    6    7       
  . . .
  3    0    1    2
  7    4    5    6
```

Update about far vs near redundancy from the discussion in the comments. Suppose sda fails:

       near
sda1 sdb1 sdc1 sdd1
  -    0    1    1
  -    2    3    3
  -    4    5    5

then sdc or sdd can still fail, while in far:

        far
sda1 sdb1 sdc1 sdd1
  -    1    2    3
  -    5    6    7       
  . . .
  -    0    1    2
  -    4    5    6

now only sdc can fail, as a failed sdb drive make block 4 inaccessible and a failed sdd drive will make block 3 inaccessible.

Conclusion: chances of surviving a 2-disk failure are higher when using a near layout. (can someone do the math here for a quantitative number?)

score -3 · Answer 3 · edited Apr 13 '17 at 12:24

Picking up some hotspares in advance would be a good idea. Also taking these notes into account.

Recommended storage scheme for home server? (LVM/JBOD/RAID 5...)

See footnote [1] in above link to see what happens with cheap storage when you need it the most.

This is all a moot point however until you profile how your the target application actually uses storage. You might find that parallelism is possible, so one block can be used for reading results, and one for writing them. This could be further abstracted behind a RAID0 (until the HBA reports QUEUE_FULL) with the results backed up via rsync.

It really depends, saying "I'm doing image analysis" without defining the workload or level of service just isn't enough; even if you did, that level of performance analysis is real work, I know it's something "I" wouldn't do in my spare time. My intentions are to get you thinking about your application to create your own solutions. Spindles are always the slowest part of your system, plan accordingly.

One idea if you wish to do the multi-array approach would be creating two RAID 1's, on separate controllers, and adding those MD devices to a LVM VG for management. Sure a RAID 10 is fast, but it's still one storage queue, now you have two, and with separate controllers there's no HBA queue sharing either.

Performance Notes:

Remember, SW RAID is no different than HW RAID, understand how it works or when it fails you might end up being more at risk as opposed to say spending your energies creating a regular backup strategy (rsync.net) instead. I've lost count of the number of users who've lost everything because they didn't read the manual and actually test the failure modes.

Tips on setting up raid10 on *buntu

3 Answers3