1

I have a software RAID-10 setup on Linux 3.16.6-203.fc20.x86_64, with 1.2 metadata and the default chunk size (512K):

$ cat /proc/mdstat
md0 : active raid10 sdc1[4] sdb1[0] sdd1[2] sde1[3]
      3907023872 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]
      bitmap: 4/30 pages [16KB], 65536KB chunk

unused devices: <none>

The filesystem is ext4, on top of LVM, on top of the RAID-10 volume group.

$ df -k .
Filesystem                     1K-blocks      Used  Available Use% Mounted on
/dev/mapper/vg_raid10-lv_home 2015734504 810039552 1103278568  43% /home

with mount options:

$ mount | grep vg_raid10-lv_home
/dev/mapper/vg_raid10-lv_home on /home type ext4 (rw,relatime,seclabel,stripe=256)

Everything seems fine. SMART indicates all disks are perfectly normal with no reallocated, pending, or offline sectors. Raw synchronous write throughput seems to be reasonably good:

$ dd if=/dev/zero of=tmp.bin bs=1G count=1 oflag=dsync
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB) copied, 7.85743 s, 137 MB/s

However, when writing small 100b (EDIT: as pointed out in the answers, I was writing 512 byte chunks, not 100 byte chunks) to the RAID-array, it is extremely slow (around 84ms per synchronous write):

$ dd if=/dev/zero of=tmp.bin bs=512 count=1000 oflag=dsync
1000+0 records in
1000+0 records out
512000 bytes (512 kB) copied, 84.2859 s, 6.1 kB/s

Is this normal for the RAID-10 configuration that I have?

Improve this question

3 Answers 3

Reset to default
1

RAID10 does not accelerate any form of O_*SYNC, at least not for small files. You're writing 512b at a time, and after each write forcing it out to disk (plus the metadata required to read it back, e.g., file size).

That requires RAID10 writes to at least 2 disks, probably 4 (i.e., all your disks). And all those writes need to be completed before it can return to dd, as that's the guarantee O_DSYNC asked for.

Improve this answer
0

Your dd command gives me 73.7 kB/s - on a SSD. So yes I guess it's normal. Or rather, dd just isn't a good benchmark.

RAID certainly does not do any speedups for small files. Access times still remain the same, and for a small file that's what will take most of the effort for HDDs, getting the read head to the physical address of the file in the first place will take longer than actually reading it.

(You're slowing it down even further if you have 4K sector disks, which are having to do something special to do a write of less than a full sector.)

Improve this answer
-1

You're not doing "100b" (byte? blocks? what was your intention?) chunks, you're doing 512 byte chunks. That is always slow because the RAID is using 512k chunks. That means that for every 512 bytes the RAID system has to read a 512k chunk, update 512 bytes in that chunk, compute its parity, and write the data + parity out to disk. It has to do the update, parity calculation and 512k + 512k parity write every time for each of the 1000 blocks you're writing. The read is only done once as that's kept in the buffer cache, and the entire operation only takes one RAID chunk.

So, yes, that's normal. Don't do that :-)

EDIT: the above would be true for RAID5. For RAID10 the parity isn't calculated, but for each dd block of 512 bytes there is still 2 times a write of 512k, one for each mirror. So you're still writing a total of 1000MB while updating just 512k. Possibly the system is smart enough to only write out the updated sectors, but I strongly suspect that's not what's happening. Also consider the updates to the write intent bitmap.

Improve this answer
4
  • No, it doesn't. This is RAID10, not RAID5 or RAID6. There is no parity. Commented Nov 5, 2014 at 16:03
  • Argh, sorry, you're right. But that only removes the calculation and parity write; there is still 1000 time a write of 512k. Commented Nov 5, 2014 at 16:05
  • No, RAID1, RAID0, and RAID10 do not have to work with the full chunk. Only with the bytes that actually change. Commented Nov 5, 2014 at 16:07
  • raid5/6 don't work with the full chunk either. the chunksize is more about physical distribution; the parity calculation takes smaller units Commented Nov 5, 2014 at 23:38

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.