Supersize io operations in Linux
This time I will tell you something that most of you really don't care about: Max io size.
More specific, how large of an io operation I can issue to a storage system in one go. This might not make much of a difference for most people, but when you are tweaking a system for databases, distributed computing etc, there is a case for being able to tune this properly.
About a year ago I was testing storage performance on different platforms, i.e Solaris and Linux. One of the tests was to measure the throughput and number of iops when ramping one to many processes over io operation sizes (i.e 8,32,64,128,256,512,1024,2048k).
To finalize this blog post, I had to go back to some emails I sent to myself (note book-keeping), so I am lacking some of the screendumps. But the basic setup is the same on both Ubuntu and SUSE using the lpfc-driver (Emulex).
On Linux, I just could not get the box to issue larger io operations than 256k, which I found to be quite disturbing as I really tried hard to bash the system with large iops. No matter how hard I tried (using different tools to generate the IO, and pressing the enter button on my keyboard really, really hard), I just could not get iostat to show IOs larger than 256k. Since I trust iostat, I deduct that my system does not produce larger IOs.
When it comes to reading the output in this post, there are a couple of things to keep in mind so that you don't get confused.
- Even if you start iostat with -k to show kilobytes, the avgrq-sz is shown as the number of 512 byte disk blocks, hence you need to multiply the number seen in iostat with 512 to get the io size
- Some config parameters need to be multiplied with 4kb memory page size (I will get to that later)
To get to my point, I need to show you a couple of things. For example, I can see that dd is issuing 1MB iops like this:
malu@kmg-sandbox-0001:~$ sudo strace dd if=/dev/zero of=/dev/sdc bs=1024k count=10000 write(1, ""..., 1048576) = 1048576 read(0, ""..., 1048576) = 1048576
But, when checking the iostat output, it only shows me 256k iops.
dm-7 0.00 0.00 0.00 1072.00 0.00 274432.00 512.00 3.07 2.85 0.93 99.20
Remember to multiply the 512.00 with 512 to get the io size.
To find out how to tune this, I really bent myself over twice. In the bad old days, pre kernel 2.6, there was MAXPHYS kernel parameter, which was probably not optimal, since it was removed in the 2.6 kernel. Don't ask me too much about the history, but I know it was a pain to get this set correctly, and often a kernel recompile was required, which in turn voided any support.
When investigating this topic, most people I talked to either told me "can't be done", or "don't bother".
The first thing I found out was that there is a tweakable parameter per device, which controls the max physical IO. It is not very convenient to use, and it only affects the LUN layer; how large of an IO can you send to a LUN.
malu@kmg-guran-0001:/sys/block/sda/queue $cat max_hw_sectors_kb 4096
Setting this (echo 128 > max_hw_sectors_kb) change the IO size sent to that device, up to some magic limit (256k) and not the 4MB as seen in the tunable above.
So... I could only limit the max io size to something which was smaller or equal to 256k, regardless of the max_hw_sectors_kb content. It is quite easy to check this, as the system will react immediately when you tune this parameter. Here is an example:
In one terminal window, run the following which will run "dd" reading 1M blocks over and over again:
malu@poc01:/sys/block/sda/queue> while true; do echo "Restarting dd" ; sudo dd iflag=direct if=/dev/sda of=/dev/null bs=1024k; done
In another terminal window, do the following:
malu@poc01:/sys/block/sda/queue> for bs in 8 16 32 64 128 256 512 1024 2048; do date "+%Y%m%d %H:%M:%S"; echo "Setting max_sectors_kb to $bs"; sudo sh -c "echo $bs > max_sectors_kb"; sleep 10; done 20110923 09:05:05 Setting max_sectors_kb to 8 20110923 09:05:15 Setting max_sectors_kb to 16 20110923 09:05:25 Setting max_sectors_kb to 32 20110923 09:05:35 Setting max_sectors_kb to 64 20110923 09:05:45 Setting max_sectors_kb to 128 20110923 09:05:55 Setting max_sectors_kb to 256 20110923 09:06:05 Setting max_sectors_kb to 512 20110923 09:06:15 Setting max_sectors_kb to 1024 20110923 09:06:25 Setting max_sectors_kb to 2048
In a third terminal window you run "iostat -xtc 1" or similar, and you will see the blocksize read from the device changing from 8k to 256k (don't forget multiplying the avgrq-sz by 512 to get the IO size).
This got me very frustrated. I really wanted to be able to tune this properly (to be able to issue larger IOs) and none of my favorite contacts at different vendors could help me out. After quite some googling, I came across a discussion about Lustre, where someone had a similar issue. This directed me to Bug 22850, where I finally found the configuration parameter I needed, to change the size of IOs sent through the driver stack.
Knowing this, I could get the following information from my system:
malu@poc01:/sys/class/scsi_host> cat host*/lpfc_sg_seg_cnt 64 64 64 64
The Emulex driver was limited to 256k IO size, which is easily tuned by changing the _lpfc_sg_segcnt parameter to 256 in /etc/modprobe.conf.local
malu@poc01:/sys/class/scsi_host> cat /etc/modprobe.conf.local options lpfc lpfc_lun_queue_depth=16 options lpfc lpfc_sg_seg_cnt=256 options lpfc lpfc_link_speed=4 malu@poc01:/sys/class/scsi_host> sudo mkinitrd malu@poc01:/sys/class/scsi_host> sudo shutdown -r now ... malu@poc01:/sys/class/scsi_host> cat host*/lpfc_sg_seg_cnt 256 256 256 256
Now the max IO size is 1MB and I am happy again. Try it out yourself!