so, after using linux since early 200x as my main os for work i can say what linux is shit, hard to understand, not user friendly, complicated shit, i found windows much more simple and user friendly, especially now-days, let me explain in details:

installation

Windows: (simple enough, most can handle it without contacting paid support)

here we have two options
1) but pc with windows preinstalled (simplest solution)
- to use this method you need to go to nearby pc/notebook/phone store and just by what you like, that's it, you done installation and may use your windows now.
2) manual install (for advanced users)
- you need to by installation optical media
- you need to put installation optical media to your optical drive
- you need to boot from your optical drive (this is most complicated part, you may even need to go to uefi/bios to achieve this)
- you will boot into graphical shell where you need few time press "next" "accept" or "ok" button near right bottom corner of screen
- installation must be done, if it is not, or machine does not boot into windows you can use installation method 1, or buy new pc,
- that's all. it is as simple as this
3) advanced installation methods
to be more user friendly and simple windows does not support advanced installation methods and does not support unusual configurations such as network boot, boot from encrypted media with external key and all other unneeded complicated things, with exceptions for server editions which users does not need and do not licensed and do not wanted to use.

windows installation is simple enough for normal user, to not overthink yourself you got just two simple options too choose from.

Linux (and other unix systems): (impossible)

with linux you have problems right after a moment of time you decided to install linux, first of all it have a LOT methods of installation, and all is bad and complicated, most does not work at all, many involving undesired brain activity called "learning", can you imagine it ?, so, let us proceed to installation methods explanation
1) buy pc with linux preinstalled (hard)
surprisingly somewhere near 2010+ this options become available too, but it is relatively hard to find pc with linux preinstalled in store, and even if you find one, most probably large part of bought pc's hardware will not work at all or work incorrectly, system configuration and problems resolving will be explained later.
2) manual install using installation media (very hard)
a big variety of installation media exists, it may be optical media, regular block device image for booting from block devices or even network boot with pxe, most normal users does not even understand this heresy typed above.
- you need to choose what you want to install, as explained above it is hard, and even involving undesired brain activities called "thinking", "learning"
- because of linux does not have unified graphical installer which used for every distro and have ability to use non-trivial complicated setups next steps are unpredictable and may vary, also involving undesired brain activities such as "learning", "reading", "thinking" which is unacceptable for normal users
3) redeploying existing installed system to new pc (impossible, myth)
tales and legends heard from time to time about some strange mythical thing like no installing at all but just moving installed system to new pc done by powerful unix adepts and other unholy mythical persons, it's scary just think about it.

i will not explain further as it may permanently damage brain of normal sane user, but even this little part should be enough to understand what linux installation is a pain and does not worth it.

Initial system configuration

Windows: (simple, straightforward, easy to understand)

1) drivers
- in case of manual installation you most probably need to install drivers for your pc, but do not be scared, you do not need to disturb your precious brain for this task. most probably you have some optical disks which come with your pc, you need to do following steps for each of your disks bundled with pc: a) insert disk in your optical drive. b) run every executable file with admistrator privileges, if some windows appears you need to press following buttons "next" "accept" "ok" "reboot", each small operation most probably require reboot or two, it's perfectly ok, if during process you pc hang (freeze) just hit reset button or hold power button of your notebook, it is also perfectly ok for windows to solve problems this way, you may need to repeat it like 10 times and problem should go away. c) if after this operations your system still lacks some drivers or does not behave properly you just need to buy new pc and repeat beginning from installation, it is as simple as that.
- if during previous actions your system start to constantly boot into mode called "BSOD" (blue screen with some strange numbers) it is also perfectly ok, just do installation from the beginning and do driver installation in different order, yes solving problems called "constant BSOD during boot" is as simple as this, just do full reinstall and it will gone, if not, buy new PC.
2) necessary programs
- everything you need is already included, also you can use a lot of services from microsoft online with your microsoft account, so you do not need install everything at all, in soce rare case microsoft does not provide needed service, in this case you can use online service from google with your google account for your task, but in very rare case where something missed, windows 10 introduced windows store where you can by everything, in very rare and very special case if you need some very special software for work, you can just contact software vendors and buy it directly, i suggest buy from vendors like adobe, you can call them from cell phone, yes, it's just as simple as this.

Linux (and other unix systems): (very hard, nearly impossible)

1) boot
often it is impossible to install system at all, and after you do all perfectly fine, your system just does not boot at all, you will need to figure out what wrong all by yourself, which will for sure involve a lot of undesired brain activities like "learning", "thinking", "understanding", you may even need to interact with unix communities where unholly unix adepts will most probably laught at you and not help at all, will say some unholy meaningless words like "man", "bootloader", "grub", "efi" and a lot of other nonsense, belive me, it's meaningless words impossible to understand and they just making fun of you, they may even try to force you read large complicated technical articles, documents and manuals, but reading does not help you at all, they just making fun of you again, also you can't use windows way to solve all and every problem called "reset button" or "full reinstall", here it does not solve anything, so if you are not lucky you will stuck here, kill a lot of time before you finally understand what you just need to go windows way.
2) boot?
if in very rare case after you turn on you pc something is happening, often instead of your desktop you see some nonsense on screen, a lot of text fast runing through screan and after few seconds just empty screen with some text ending with "login:" it's absolutely meaningless and nothing you can do about it, you should try to install different linux version and pray to have desktop after this hard and comlicated install process or you can try contact unix community, but as mentioned earlier they will just make fun of you with sensles words and large compplicated sensless texts, so better just go windows way before it's too late.
3) boot??
it is exists a very tiny possibility what after installation your desktop will appear.
4) necessary programs
but wait, what is this ?, where to input your microsoft account login and password ?, where is my start menu ?, all is worng, you do not have your programs, you can't check what drivers installed, drivers from optical discs does not installing at all, you can't use your adobe photoshop bought for a large price, it's just not installing at all, you do not have your favorite windows media player here or even internet explorer or microsoft edge, you can't connect to network, all your precious software downloaded from torrents does not installs at all, and not only pirated, but paid software too, so now you are realized what linux is shit yes ?, just go windows way already. you do not need more pain.

Troubleshooting

Windows: (simple, straightforward, easy to understand)

1) performance problems (simple)
- if you have problems with performance, reboot you pc (this may be needed up to 10 or more times, it's perfectly ok), yes, it's that simple
- if step above does not solve your issue, do full system reinstall, it should resolve your issue, yes. it's as simple as this.
- if step above does not solve your issue, buy new pc, yes, just as simple,
2) stability problems (simple, straightforward, easy to understand)
- solutions are exactly same as for performance problems, yes, it is that simple, one set of solutions for every kind of problem to keep users mind free.

Linux (and other unix systems): (impossible)
1) performance problems (impossible)
- if you have performance problems with linux nor reboot nor reinstall does nothing, just does not help, you can try buy new pc but it will also most probably not help
- in linux used hard complicated approach to solving any kind of problems, you will be force to do a lot of undesired brain activites such as "reading", "learning", "understanding", "thinking"m this is a real hell belive me, in worst cases you may even need to talk with linux developers directly, you will be forced to understand how system works and how problemstic parts of system works, you will be forced to spend your precious time which is dedicated for porno, games and slacking on lot of undesired brain activities instead, and when you think problem is finally solved, another problem will appears, and you will see no end for this, ever, in worst cases you will be forced to learn programming and scripting languages instead of drinking beer with your friends, this is a hell with no way out, belive me and go windows way !
2) stability problems (impossible)
- same as above, hitting reset and syste, reinstalls does not helping at all.
- same as above

Conclusion:

soo....., just a friendly advice, before you ruined your life like me, just turn around and run, run..., run..., run windows way.
if something goes wrong you can always hit reset button few times to solve any kind of problems
in case problem not solved you can always reinstall windows
if reinstalling also not solved problem, you can always buy new pc with windows preinstalled.
do not ruin your life, it's time to reinstall windows !

small linux block devices performance test

#linux #filesystems #xfs #btrfs #ext4 #lvm #raid #mdadm #luks #cryptsetup

linux io test on different storage and diferent filesystems with fio:

NOTES:
tests done on different hw configurations, on devices wrapped by mdadm raid0, cryptsetup luks crypto and lvm, parameters, crypto and raw io performance will be shown for each test.

machine 1:

storage:
2 x Samsung SSD 860 EVO 500GB
3 x WDC WD40PURZ-85TTDY0

storage layout:

ssd array: 2 x ssd mdadm raid 0 + cryptsetup luks with aes-xts-plain64 + lvm
hdd array: 3 x hdd mdadm raid 0 + cryptsetup luks with camellia-xts-plain64 + lvm

raw linear read io performance with hdparm -Tt (to understand bus limitations):

ssd array:

 Timing cached reads: 22102 MB in 2.00 seconds = 11064.06 MB/sec
 Timing buffered disk reads: 3108 MB in 3.00 seconds = 1035.40 MB/sec

hdd array:

 Timing cached reads: 23258 MB in 2.00 seconds = 11643.60 MB/sec
 Timing buffered disk reads: 1522 MB in 3.00 seconds = 506.96 MB/sec

crypto performance:

[h1]Tests are approximate using memory only (no storage IO).[/h1]

[h1]Algorithm | Key | Encryption | Decryption[/h1]

twofish-xts 512b 353.1 MiB/s 357.0 MiB/s

[h1]Tests are approximate using memory only (no storage IO).[/h1]

[h1]Algorithm | Key | Encryption | Decryption[/h1]

serpent-xts 512b 612.3 MiB/s 609.4 MiB/s

[h1]Tests are approximate using memory only (no storage IO).[/h1]

[h1]Algorithm | Key | Encryption | Decryption[/h1]

aes-xts 512b 2681.4 MiB/s 2723.0 MiB/s

[h1]Tests are approximate using memory only (no storage IO).[/h1]

[h1]Algorithm | Key | Encryption | Decryption[/h1]

camellia-xts 512b 740.9 MiB/s 755.7 MiB/s

[h1]Tests are approximate using memory only (no storage IO).[/h1]

PBKDF2-ripemd160 200722 iterations per second for 512-bit key

[h1]Tests are approximate using memory only (no storage IO).[/h1]

PBKDF2-sha256 586451 iterations per second for 512-bit key

[h1]Tests are approximate using memory only (no storage IO).[/h1]

PBKDF2-sha512 781353 iterations per second for 512-bit key

[h1]Tests are approximate using memory only (no storage IO).[/h1]

PBKDF2-whirlpool 369737 iterations per second for 512-bit key

xfs filesystem tests:

fio-rand-RW.fio:

ssd array:

NOTE: filesystem BLOCK_SIZE=”512”

file1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.18
Starting 4 processes
file1: Laying out IO file (1 file / 10240MiB)
Jobs: 4 (f=4): [m(4)][100.0%][r=125MiB/s,w=83.3MiB/s][r=31.0k,w=21.3k IOPS][eta 00m:00s]
file1: (groupid=0, jobs=1): err= 0: pid=3927: Wed Apr 15 01:25:51 2020
 read: IOPS=14.2k, BW=55.4MiB/s (58.1MB/s)(48.7GiB/900001msec)
 slat (nsec): min=611, max=15747M, avg=30942.93, stdev=10093578.28
 clat (nsec): min=1002, max=22729M, avg=647744.82, stdev=56483253.23
 lat (usec): min=24, max=22729k, avg=678.75, stdev=57420.76
 clat percentiles (usec):
 | 1.00th=[ 53], 5.00th=[ 62], 10.00th=[ 67], 20.00th=[ 73],
 | 30.00th=[ 80], 40.00th=[ 91], 50.00th=[ 198], 60.00th=[ 265],
 | 70.00th=[ 404], 80.00th=[ 611], 90.00th=[ 1057], 95.00th=[ 1647],
 | 99.00th=[ 3097], 99.50th=[ 3687], 99.90th=[ 5145], 99.95th=[ 5866],
 | 99.99th=[10290]
 bw ( KiB/s): min= 8, max=542264, per=36.81%, avg=83508.11, stdev=106473.54, samples=1222
 iops : min= 2, max=135566, avg=20877.03, stdev=26618.40, samples=1222
 write: IOPS=9453, BW=36.9MiB/s (38.7MB/s)(32.5GiB/900001msec); 0 zone resets
 slat (nsec): min=1012, max=17920M, avg=51399.19, stdev=13365178.34
 clat (usec): min=24, max=22729k, avg=615.97, stdev=53329.84
 lat (usec): min=26, max=22729k, avg=667.44, stdev=55364.63
 clat percentiles (usec):
 | 1.00th=[ 52], 5.00th=[ 62], 10.00th=[ 67], 20.00th=[ 73],
 | 30.00th=[ 80], 40.00th=[ 89], 50.00th=[ 186], 60.00th=[ 245],
 | 70.00th=[ 379], 80.00th=[ 578], 90.00th=[ 1012], 95.00th=[ 1582],
 | 99.00th=[ 3032], 99.50th=[ 3621], 99.90th=[ 5014], 99.95th=[ 5735],
 | 99.99th=[ 9372]
 bw ( KiB/s): min= 7, max=364976, per=36.96%, avg=55910.94, stdev=71077.22, samples=1217
 iops : min= 1, max=91244, avg=13977.74, stdev=17769.31, samples=1217
 lat (usec) : 2=0.01%, 50=0.70%, 100=44.30%, 250=14.47%, 500=16.34%
 lat (usec) : 750=8.82%, 1000=4.80%
 lat (msec) : 2=7.22%, 4=3.01%, 10=0.32%, 20=0.01%, 50=0.01%
 lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.01%
 cpu : usr=2.22%, sys=17.10%, ctx=7748150, majf=0, minf=23
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=12759994,8508306,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=3928: Wed Apr 15 01:25:51 2020
 read: IOPS=14.2k, BW=55.4MiB/s (58.1MB/s)(48.7GiB/900001msec)
 slat (nsec): min=632, max=14055M, avg=28707.68, stdev=8367525.63
 clat (nsec): min=1262, max=22585M, avg=641342.80, stdev=54006164.38
 lat (usec): min=25, max=22585k, avg=670.12, stdev=54889.64
 clat percentiles (usec):
 | 1.00th=[ 52], 5.00th=[ 62], 10.00th=[ 67], 20.00th=[ 74],
 | 30.00th=[ 81], 40.00th=[ 91], 50.00th=[ 198], 60.00th=[ 269],
 | 70.00th=[ 404], 80.00th=[ 611], 90.00th=[ 1057], 95.00th=[ 1647],
 | 99.00th=[ 3097], 99.50th=[ 3687], 99.90th=[ 5145], 99.95th=[ 5866],
 | 99.99th=[10028]
 bw ( KiB/s): min= 8, max=534096, per=36.83%, avg=83552.06, stdev=106239.96, samples=1221
 iops : min= 2, max=133524, avg=20888.01, stdev=26559.98, samples=1221
 write: IOPS=9459, BW=36.0MiB/s (38.7MB/s)(32.5GiB/900001msec); 0 zone resets
 slat (nsec): min=1002, max=17920M, avg=54725.10, stdev=15045267.29
 clat (usec): min=24, max=22585k, avg=625.45, stdev=55917.05
 lat (usec): min=26, max=22585k, avg=680.25, stdev=58145.95
 clat percentiles (usec):
 | 1.00th=[ 51], 5.00th=[ 62], 10.00th=[ 67], 20.00th=[ 73],
 | 30.00th=[ 80], 40.00th=[ 89], 50.00th=[ 186], 60.00th=[ 245],
 | 70.00th=[ 379], 80.00th=[ 578], 90.00th=[ 1012], 95.00th=[ 1582],
 | 99.00th=[ 3032], 99.50th=[ 3589], 99.90th=[ 5014], 99.95th=[ 5735],
 | 99.99th=[ 9765]
 bw ( KiB/s): min= 7, max=359128, per=37.01%, avg=55994.86, stdev=71016.78, samples=1216
 iops : min= 1, max=89782, avg=13998.72, stdev=17754.19, samples=1216
 lat (usec) : 2=0.01%, 50=0.85%, 100=44.08%, 250=14.51%, 500=16.41%
 lat (usec) : 750=8.79%, 1000=4.79%
 lat (msec) : 2=7.23%, 4=3.02%, 10=0.32%, 20=0.01%, 50=0.01%
 lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.01%
 cpu : usr=2.24%, sys=17.08%, ctx=7664942, majf=0, minf=24
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=12756139,8513933,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=3929: Wed Apr 15 01:25:51 2020
 read: IOPS=14.2k, BW=55.4MiB/s (58.1MB/s)(48.7GiB/900001msec)
 slat (nsec): min=641, max=17920M, avg=30274.08, stdev=10110966.15
 clat (usec): min=22, max=22670k, avg=649.54, stdev=55193.99
 lat (usec): min=23, max=22670k, avg=679.88, stdev=56204.10
 clat percentiles (usec):
 | 1.00th=[ 51], 5.00th=[ 62], 10.00th=[ 67], 20.00th=[ 74],
 | 30.00th=[ 81], 40.00th=[ 91], 50.00th=[ 198], 60.00th=[ 265],
 | 70.00th=[ 404], 80.00th=[ 611], 90.00th=[ 1057], 95.00th=[ 1647],
 | 99.00th=[ 3097], 99.50th=[ 3687], 99.90th=[ 5145], 99.95th=[ 5932],
 | 99.99th=[10159]
 bw ( KiB/s): min= 8, max=537344, per=36.82%, avg=83527.72, stdev=106442.07, samples=1222
 iops : min= 2, max=134336, avg=20881.93, stdev=26610.52, samples=1222
 write: IOPS=9457, BW=36.9MiB/s (38.7MB/s)(32.5GiB/900001msec); 0 zone resets
 slat (nsec): min=991, max=15747M, avg=52380.86, stdev=13341407.56
 clat (nsec): min=1192, max=22670M, avg=612775.08, stdev=55147764.50
 lat (usec): min=23, max=22670k, avg=665.23, stdev=57123.82
 clat percentiles (usec):
 | 1.00th=[ 49], 5.00th=[ 61], 10.00th=[ 67], 20.00th=[ 73],
 | 30.00th=[ 80], 40.00th=[ 90], 50.00th=[ 186], 60.00th=[ 245],
 | 70.00th=[ 375], 80.00th=[ 578], 90.00th=[ 1012], 95.00th=[ 1582],
 | 99.00th=[ 3032], 99.50th=[ 3589], 99.90th=[ 5014], 99.95th=[ 5800],
 | 99.99th=[ 9503]
 bw ( KiB/s): min= 8, max=358760, per=37.03%, avg=56023.15, stdev=71031.30, samples=1215
 iops : min= 2, max=89690, avg=14005.79, stdev=17757.82, samples=1215
 lat (usec) : 2=0.01%, 50=1.01%, 100=43.83%, 250=14.60%, 500=16.42%
 lat (usec) : 750=8.80%, 1000=4.79%
 lat (msec) : 2=7.21%, 4=3.02%, 10=0.32%, 20=0.01%, 50=0.01%
 lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.01%
 cpu : usr=2.25%, sys=17.08%, ctx=7573034, majf=0, minf=23
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=12763389,8511685,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=3930: Wed Apr 15 01:25:51 2020
 read: IOPS=14.2k, BW=55.4MiB/s (58.1MB/s)(48.7GiB/900001msec)
 slat (nsec): min=602, max=15251M, avg=26931.70, stdev=8058751.14
 clat (usec): min=24, max=22701k, avg=647.36, stdev=56304.54
 lat (usec): min=25, max=22701k, avg=674.36, stdev=56896.75
 clat percentiles (usec):
 | 1.00th=[ 54], 5.00th=[ 62], 10.00th=[ 67], 20.00th=[ 73],
 | 30.00th=[ 80], 40.00th=[ 91], 50.00th=[ 198], 60.00th=[ 269],
 | 70.00th=[ 404], 80.00th=[ 611], 90.00th=[ 1057], 95.00th=[ 1647],
 | 99.00th=[ 3097], 99.50th=[ 3654], 99.90th=[ 5080], 99.95th=[ 5866],
 | 99.99th=[10290]
 bw ( KiB/s): min= 8, max=538712, per=36.89%, avg=83694.31, stdev=106111.60, samples=1220
 iops : min= 2, max=134678, avg=20923.57, stdev=26527.89, samples=1220
 write: IOPS=9449, BW=36.9MiB/s (38.7MB/s)(32.4GiB/900001msec); 0 zone resets
 slat (nsec): min=982, max=17920M, avg=57355.87, stdev=15301847.63
 clat (nsec): min=1393, max=22701M, avg=616232.01, stdev=53793093.23
 lat (usec): min=26, max=22701k, avg=673.66, stdev=56413.37
 clat percentiles (usec):
 | 1.00th=[ 53], 5.00th=[ 62], 10.00th=[ 67], 20.00th=[ 73],
 | 30.00th=[ 80], 40.00th=[ 89], 50.00th=[ 188], 60.00th=[ 245],
 | 70.00th=[ 375], 80.00th=[ 578], 90.00th=[ 1004], 95.00th=[ 1582],
 | 99.00th=[ 2999], 99.50th=[ 3589], 99.90th=[ 5014], 99.95th=[ 5735],
 | 99.99th=[10028]
 bw ( KiB/s): min= 8, max=358920, per=36.94%, avg=55889.27, stdev=70681.64, samples=1217
 iops : min= 2, max=89730, avg=13972.32, stdev=17670.41, samples=1217
 lat (usec) : 2=0.01%, 50=0.60%, 100=44.09%, 250=14.70%, 500=16.51%
 lat (usec) : 750=8.80%, 1000=4.77%
 lat (msec) : 2=7.21%, 4=3.00%, 10=0.31%, 20=0.01%, 50=0.01%
 lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.01%
 cpu : usr=2.22%, sys=17.14%, ctx=7555022, majf=0, minf=23
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=12767350,8504713,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
 READ: bw=222MiB/s (232MB/s), 55.4MiB/s-55.4MiB/s (58.1MB/s-58.1MB/s), io=195GiB (209GB), run=900001-900001msec
 WRITE: bw=148MiB/s (155MB/s), 36.9MiB/s-36.0MiB/s (38.7MB/s-38.7MB/s), io=130GiB (139GB), run=900001-900001msec

Disk stats (read/write):
 dm-3: ios=4275603/17609796, merge=0/0, ticks=1415389/1013248346, in_queue=1014664460, util=98.65%, aggrios=4277523/17609907, aggrmerge=0/0, aggrticks=1411929/1013681205, aggrin_queue=1015093858, aggrutil=98.65%
 dm-0: ios=4277523/17609907, merge=0/0, ticks=1411929/1013681205, in_queue=1015093858, util=98.65%, aggrios=4278851/17673756, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
 md128: ios=4278851/17673756, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=2139263/6387625, aggrmerge=160/2449587, aggrticks=294633/673499, aggrin_queue=32241, aggrutil=97.61%
 sdd: ios=2139623/6390350, merge=75/2451332, ticks=299397/894150, in_queue=43498, util=91.78%
 sda: ios=2138904/6384901, merge=245/2447843, ticks=289869/452848, in_queue=20984, util=97.61%

hdd array:

file1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.18
Starting 4 processes
file1: Laying out IO file (1 file / 10240MiB)
Jobs: 4 (f=4): [m(4)][100.0%][r=534KiB/s,w=403KiB/s][r=133,w=100 IOPS][eta 00m:00s]
file1: (groupid=0, jobs=1): err= 0: pid=6427: Wed Apr 15 01:45:56 2020
 read: IOPS=40, BW=163KiB/s (167kB/s)(143MiB/900027msec)
 slat (nsec): min=1212, max=878192k, avg=16018904.58, stdev=21699428.23
 clat (msec): min=32, max=2822, avg=221.78, stdev=153.22
 lat (msec): min=40, max=2862, avg=237.80, stdev=161.48
 clat percentiles (msec):
 | 1.00th=[ 89], 5.00th=[ 113], 10.00th=[ 126], 20.00th=[ 142],
 | 30.00th=[ 155], 40.00th=[ 165], 50.00th=[ 178], 60.00th=[ 192],
 | 70.00th=[ 211], 80.00th=[ 257], 90.00th=[ 384], 95.00th=[ 481],
 | 99.00th=[ 860], 99.50th=[ 1083], 99.90th=[ 1687], 99.95th=[ 2005],
 | 99.99th=[ 2366]
 bw ( KiB/s): min= 8, max= 328, per=25.35%, avg=163.77, stdev=73.33, samples=1789
 iops : min= 2, max= 82, avg=40.92, stdev=18.35, samples=1789
 write: IOPS=27, BW=109KiB/s (111kB/s)(95.4MiB/900027msec); 0 zone resets
 slat (nsec): min=1673, max=797254k, avg=12810828.53, stdev=20531401.74
 clat (nsec): min=1874, max=2770.1M, avg=219834449.76, stdev=158340065.25
 lat (msec): min=34, max=2870, avg=232.65, stdev=165.75
 clat percentiles (msec):
 | 1.00th=[ 85], 5.00th=[ 108], 10.00th=[ 123], 20.00th=[ 138],
 | 30.00th=[ 150], 40.00th=[ 163], 50.00th=[ 174], 60.00th=[ 188],
 | 70.00th=[ 209], 80.00th=[ 257], 90.00th=[ 380], 95.00th=[ 477],
 | 99.00th=[ 902], 99.50th=[ 1167], 99.90th=[ 1670], 99.95th=[ 2039],
 | 99.99th=[ 2500]
 bw ( KiB/s): min= 7, max= 288, per=25.33%, avg=109.69, stdev=53.25, samples=1780
 iops : min= 1, max= 72, avg=27.40, stdev=13.32, samples=1780
 lat (usec) : 2=0.01%
 lat (msec) : 50=0.03%, 100=2.56%, 250=76.40%, 500=16.74%, 750=2.84%
 lat (msec) : 1000=0.74%, 2000=0.64%, >=2000=0.05%
 cpu : usr=0.02%, sys=0.70%, ctx=77997, majf=0, minf=19
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=36642,24424,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=6428: Wed Apr 15 01:45:56 2020
 read: IOPS=39, BW=159KiB/s (163kB/s)(140MiB/900027msec)
 slat (nsec): min=1242, max=753470k, avg=16238498.10, stdev=22017584.22
 clat (msec): min=50, max=2980, avg=226.92, stdev=161.24
 lat (msec): min=58, max=3103, avg=243.15, stdev=169.55
 clat percentiles (msec):
 | 1.00th=[ 93], 5.00th=[ 114], 10.00th=[ 128], 20.00th=[ 144],
 | 30.00th=[ 157], 40.00th=[ 167], 50.00th=[ 180], 60.00th=[ 194],
 | 70.00th=[ 215], 80.00th=[ 264], 90.00th=[ 388], 95.00th=[ 493],
 | 99.00th=[ 936], 99.50th=[ 1167], 99.90th=[ 1720], 99.95th=[ 2056],
 | 99.99th=[ 2802]
 bw ( KiB/s): min= 7, max= 336, per=24.79%, avg=160.17, stdev=72.25, samples=1790
 iops : min= 1, max= 84, avg=40.02, stdev=18.07, samples=1790
 write: IOPS=26, BW=106KiB/s (109kB/s)(93.6MiB/900027msec); 0 zone resets
 slat (nsec): min=1603, max=576628k, avg=13256187.30, stdev=20718121.87
 clat (nsec): min=1543, max=3015.3M, avg=223824877.06, stdev=163028368.52
 lat (msec): min=34, max=3088, avg=237.08, stdev=170.93
 clat percentiles (msec):
 | 1.00th=[ 89], 5.00th=[ 112], 10.00th=[ 125], 20.00th=[ 140],
 | 30.00th=[ 153], 40.00th=[ 165], 50.00th=[ 178], 60.00th=[ 190],
 | 70.00th=[ 211], 80.00th=[ 262], 90.00th=[ 384], 95.00th=[ 485],
 | 99.00th=[ 944], 99.50th=[ 1200], 99.90th=[ 1770], 99.95th=[ 2265],
 | 99.99th=[ 2668]
 bw ( KiB/s): min= 7, max= 264, per=24.85%, avg=107.59, stdev=53.57, samples=1780
 iops : min= 1, max= 66, avg=26.87, stdev=13.40, samples=1780
 lat (usec) : 2=0.01%
 lat (msec) : 50=0.01%, 100=2.08%, 250=76.35%, 500=16.82%, 750=3.20%
 lat (msec) : 1000=0.66%, 2000=0.83%, >=2000=0.06%
 cpu : usr=0.02%, sys=0.60%, ctx=77652, majf=0, minf=19
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=35859,23955,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=6429: Wed Apr 15 01:45:56 2020
 read: IOPS=40, BW=163KiB/s (167kB/s)(143MiB/900027msec)
 slat (nsec): min=1443, max=766424k, avg=15892940.95, stdev=21352833.36
 clat (nsec): min=1884, max=2955.3M, avg=221958326.00, stdev=162678407.80
 lat (msec): min=44, max=3029, avg=237.85, stdev=171.07
 clat percentiles (msec):
 | 1.00th=[ 91], 5.00th=[ 113], 10.00th=[ 125], 20.00th=[ 140],
 | 30.00th=[ 153], 40.00th=[ 165], 50.00th=[ 176], 60.00th=[ 190],
 | 70.00th=[ 209], 80.00th=[ 255], 90.00th=[ 380], 95.00th=[ 485],
 | 99.00th=[ 911], 99.50th=[ 1167], 99.90th=[ 1854], 99.95th=[ 2232],
 | 99.99th=[ 2534]
 bw ( KiB/s): min= 8, max= 384, per=25.40%, avg=164.08, stdev=74.91, samples=1789
 iops : min= 2, max= 96, avg=40.99, stdev=18.74, samples=1789
 write: IOPS=27, BW=109KiB/s (112kB/s)(95.9MiB/900027msec); 0 zone resets
 slat (nsec): min=1583, max=589310k, avg=12886125.28, stdev=20885914.54
 clat (msec): min=31, max=2931, avg=217.75, stdev=155.35
 lat (msec): min=41, max=3035, avg=230.64, stdev=162.83
 clat percentiles (msec):
 | 1.00th=[ 87], 5.00th=[ 109], 10.00th=[ 122], 20.00th=[ 138],
 | 30.00th=[ 148], 40.00th=[ 161], 50.00th=[ 174], 60.00th=[ 186],
 | 70.00th=[ 207], 80.00th=[ 255], 90.00th=[ 380], 95.00th=[ 481],
 | 99.00th=[ 869], 99.50th=[ 1099], 99.90th=[ 1636], 99.95th=[ 2005],
 | 99.99th=[ 2903]
 bw ( KiB/s): min= 7, max= 304, per=25.57%, avg=110.72, stdev=53.36, samples=1773
 iops : min= 1, max= 76, avg=27.65, stdev=13.35, samples=1773
 lat (usec) : 2=0.01%
 lat (msec) : 50=0.02%, 100=2.46%, 250=77.00%, 500=16.15%, 750=2.86%
 lat (msec) : 1000=0.75%, 2000=0.68%, >=2000=0.07%
 cpu : usr=0.03%, sys=0.59%, ctx=77951, majf=0, minf=18
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=36713,24552,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=6430: Wed Apr 15 01:45:56 2020
 read: IOPS=40, BW=162KiB/s (165kB/s)(142MiB/900027msec)
 slat (nsec): min=1322, max=824107k, avg=15921847.02, stdev=21301902.36
 clat (usec): min=2, max=2898.1k, avg=223385.40, stdev=158901.08
 lat (msec): min=27, max=3079, avg=239.31, stdev=167.29
 clat percentiles (msec):
 | 1.00th=[ 92], 5.00th=[ 113], 10.00th=[ 126], 20.00th=[ 142],
 | 30.00th=[ 155], 40.00th=[ 165], 50.00th=[ 178], 60.00th=[ 192],
 | 70.00th=[ 211], 80.00th=[ 259], 90.00th=[ 388], 95.00th=[ 485],
 | 99.00th=[ 894], 99.50th=[ 1133], 99.90th=[ 1770], 99.95th=[ 2072],
 | 99.99th=[ 2836]
 bw ( KiB/s): min= 7, max= 344, per=25.16%, avg=162.53, stdev=73.59, samples=1788
 iops : min= 1, max= 86, avg=40.61, stdev=18.41, samples=1788
 write: IOPS=27, BW=109KiB/s (112kB/s)(95.8MiB/900027msec); 0 zone resets
 slat (nsec): min=1583, max=656166k, avg=13099102.70, stdev=21441180.89
 clat (msec): min=29, max=2542, avg=219.31, stdev=158.69
 lat (msec): min=32, max=2556, avg=232.41, stdev=165.99
 clat percentiles (msec):
 | 1.00th=[ 88], 5.00th=[ 110], 10.00th=[ 122], 20.00th=[ 138],
 | 30.00th=[ 150], 40.00th=[ 161], 50.00th=[ 174], 60.00th=[ 188],
 | 70.00th=[ 207], 80.00th=[ 255], 90.00th=[ 384], 95.00th=[ 481],
 | 99.00th=[ 919], 99.50th=[ 1133], 99.90th=[ 1720], 99.95th=[ 2039],
 | 99.99th=[ 2433]
 bw ( KiB/s): min= 7, max= 304, per=25.52%, avg=110.50, stdev=54.44, samples=1774
 iops : min= 1, max= 76, avg=27.60, stdev=13.62, samples=1774
 lat (usec) : 4=0.01%
 lat (msec) : 50=0.03%, 100=2.30%, 250=76.78%, 500=16.50%, 750=2.89%
 lat (msec) : 1000=0.74%, 2000=0.69%, >=2000=0.07%
 cpu : usr=0.02%, sys=0.63%, ctx=77963, majf=0, minf=18
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=36345,24519,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
 READ: bw=647KiB/s (662kB/s), 159KiB/s-163KiB/s (163kB/s-167kB/s), io=569MiB (596MB), run=900027-900027msec
 WRITE: bw=433KiB/s (443kB/s), 106KiB/s-109KiB/s (109kB/s-112kB/s), io=381MiB (399MB), run=900027-900027msec

Disk stats (read/write):
 dm-2: ios=147608/115656, merge=0/0, ticks=1852342/56226343, in_queue=58078685, util=21.65%, aggrios=147616/115661, aggrmerge=0/0, aggrticks=1852266/56226297, aggrin_queue=58078563, aggrutil=21.66%
 dm-1: ios=147616/115661, merge=0/0, ticks=1852266/56226297, in_queue=58078563, util=21.66%, aggrios=147617/115710, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
 md127: ios=147617/115710, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=49206/38650, aggrmerge=0/17, aggrticks=615920/1354928, aggrin_queue=1927653, aggrutil=12.05%
 sdb: ios=49620/38693, merge=0/23, ticks=675653/1576403, in_queue=2208415, util=12.05%
 sde: ios=49050/38640, merge=0/17, ticks=590031/1274911, in_queue=1821844, util=12.01%
 sdc: ios=48949/38617, merge=0/12, ticks=582078/1213470, in_queue=1752702, util=11.99%

ssd-test.fio:

ssd array:

NOTE: filesystem BLOCK_SIZE=”512”

seq-read: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
rand-read: (g=1): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
seq-write: (g=2): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
rand-write: (g=3): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
fio-3.18
Starting 4 processes
seq-read: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [_(3),w(1)][97.1%][w=144MiB/s][w=36.0k IOPS][eta 00m:07s] 
seq-read: (groupid=0, jobs=1): err= 0: pid=493789: Wed Apr 15 09:36:28 2020
 read: IOPS=47.3k, BW=185MiB/s (194MB/s)(10.0GiB/55432msec)
 slat (nsec): min=1734, max=7910.4k, avg=6939.26, stdev=6107.82
 clat (usec): min=18, max=18890, avg=77.00, stdev=58.03
 lat (usec): min=39, max=18894, avg=84.06, stdev=57.46
 clat percentiles (usec):
 | 1.00th=[ 46], 5.00th=[ 55], 10.00th=[ 57], 20.00th=[ 60],
 | 30.00th=[ 61], 40.00th=[ 63], 50.00th=[ 73], 60.00th=[ 87],
 | 70.00th=[ 91], 80.00th=[ 93], 90.00th=[ 95], 95.00th=[ 96],
 | 99.00th=[ 176], 99.50th=[ 212], 99.90th=[ 330], 99.95th=[ 379],
 | 99.99th=[ 2245]
 bw ( KiB/s): min=165624, max=198600, per=100.00%, avg=189438.94, stdev=5253.06, samples=110
 iops : min=41406, max=49650, avg=47359.73, stdev=1313.24, samples=110
 lat (usec) : 20=0.01%, 50=2.23%, 100=95.33%, 250=2.25%, 500=0.16%
 lat (usec) : 750=0.01%, 1000=0.01%
 lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
 cpu : usr=5.21%, sys=33.67%, ctx=1870277, majf=0, minf=12
 IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=2621440,0,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=4
rand-read: (groupid=1, jobs=1): err= 0: pid=493941: Wed Apr 15 09:36:28 2020
 read: IOPS=21.4k, BW=83.7MiB/s (87.8MB/s)(5021MiB/60001msec)
 slat (nsec): min=1993, max=698901, avg=5083.29, stdev=2238.84
 clat (usec): min=10, max=5835, avg=180.83, stdev=107.98
 lat (usec): min=64, max=5841, avg=186.03, stdev=107.39
 clat percentiles (usec):
 | 1.00th=[ 85], 5.00th=[ 87], 10.00th=[ 89], 20.00th=[ 92],
 | 30.00th=[ 96], 40.00th=[ 113], 50.00th=[ 133], 60.00th=[ 178],
 | 70.00th=[ 249], 80.00th=[ 265], 90.00th=[ 343], 95.00th=[ 363],
 | 99.00th=[ 457], 99.50th=[ 519], 99.90th=[ 676], 99.95th=[ 881],
 | 99.99th=[ 2343]
 bw ( KiB/s): min=75832, max=90616, per=100.00%, avg=85828.91, stdev=2325.89, samples=119
 iops : min=18958, max=22654, avg=21457.24, stdev=581.46, samples=119
 lat (usec) : 20=0.01%, 100=35.33%, 250=34.90%, 500=29.15%, 750=0.54%
 lat (usec) : 1000=0.03%
 lat (msec) : 2=0.03%, 4=0.01%, 10=0.01%
 cpu : usr=2.97%, sys=13.06%, ctx=1000451, majf=0, minf=11
 IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=1285489,0,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=4
seq-write: (groupid=2, jobs=1): err= 0: pid=494174: Wed Apr 15 09:36:28 2020
 write: IOPS=41.5k, BW=162MiB/s (170MB/s)(9724MiB/60001msec); 0 zone resets
 slat (nsec): min=1302, max=505648, avg=3797.37, stdev=1015.23
 clat (usec): min=9, max=85537, avg=91.98, stdev=119.30
 lat (usec): min=42, max=85543, avg=95.89, stdev=119.29
 clat percentiles (usec):
 | 1.00th=[ 58], 5.00th=[ 72], 10.00th=[ 76], 20.00th=[ 78],
 | 30.00th=[ 79], 40.00th=[ 80], 50.00th=[ 87], 60.00th=[ 96],
 | 70.00th=[ 100], 80.00th=[ 104], 90.00th=[ 108], 95.00th=[ 111],
 | 99.00th=[ 204], 99.50th=[ 239], 99.90th=[ 379], 99.95th=[ 553],
 | 99.99th=[ 1237]
 bw ( KiB/s): min=139312, max=174680, per=100.00%, avg=166184.06, stdev=5209.64, samples=119
 iops : min=34828, max=43670, avg=41546.03, stdev=1302.41, samples=119
 lat (usec) : 10=0.01%, 20=0.01%, 50=0.23%, 100=69.39%, 250=29.98%
 lat (usec) : 500=0.34%, 750=0.03%, 1000=0.01%
 lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
 lat (msec) : 100=0.01%
 cpu : usr=5.76%, sys=19.09%, ctx=1840937, majf=0, minf=13
 IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=0,2489393,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=4
rand-write: (groupid=3, jobs=1): err= 0: pid=494628: Wed Apr 15 09:36:28 2020
 write: IOPS=39.1k, BW=153MiB/s (160MB/s)(9155MiB/60001msec); 0 zone resets
 slat (nsec): min=1373, max=1069.0k, avg=3878.94, stdev=1213.13
 clat (usec): min=10, max=647688, avg=97.77, stdev=994.90
 lat (usec): min=41, max=647745, avg=101.76, stdev=994.94
 clat percentiles (usec):
 | 1.00th=[ 45], 5.00th=[ 47], 10.00th=[ 49], 20.00th=[ 52],
 | 30.00th=[ 56], 40.00th=[ 60], 50.00th=[ 70], 60.00th=[ 86],
 | 70.00th=[ 115], 80.00th=[ 129], 90.00th=[ 161], 95.00th=[ 180],
 | 99.00th=[ 281], 99.50th=[ 363], 99.90th=[ 1860], 99.95th=[ 2900],
 | 99.99th=[ 7308]
 bw ( KiB/s): min=58584, max=185840, per=100.00%, avg=157849.32, stdev=21750.44, samples=118
 iops : min=14646, max=46460, avg=39462.38, stdev=5437.63, samples=118
 lat (usec) : 20=0.01%, 50=15.62%, 100=49.24%, 250=33.66%, 500=1.20%
 lat (usec) : 750=0.09%, 1000=0.04%
 lat (msec) : 2=0.06%, 4=0.06%, 10=0.03%, 20=0.01%, 50=0.01%
 lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
 cpu : usr=6.20%, sys=18.91%, ctx=2049357, majf=0, minf=13
 IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=0,2343796,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
 READ: bw=185MiB/s (194MB/s), 185MiB/s-185MiB/s (194MB/s-194MB/s), io=10.0GiB (10.7GB), run=55432-55432msec

Run status group 1 (all jobs):
 READ: bw=83.7MiB/s (87.8MB/s), 83.7MiB/s-83.7MiB/s (87.8MB/s-87.8MB/s), io=5021MiB (5265MB), run=60001-60001msec

Run status group 2 (all jobs):
 WRITE: bw=162MiB/s (170MB/s), 162MiB/s-162MiB/s (170MB/s-170MB/s), io=9724MiB (10.2GB), run=60001-60001msec

Run status group 3 (all jobs):
 WRITE: bw=153MiB/s (160MB/s), 153MiB/s-153MiB/s (160MB/s-160MB/s), io=9155MiB (9600MB), run=60001-60001msec

Disk stats (read/write):
 dm-3: ios=3920958/4851512, merge=0/0, ticks=445073/1856767, in_queue=2302040, util=99.14%, aggrios=3920967/4854312, aggrmerge=0/0, aggrticks=442211/1947582, aggrin_queue=2389993, aggrutil=99.23%
 dm-0: ios=3920967/4854312, merge=0/0, ticks=442211/1947582, in_queue=2389993, util=99.23%, aggrios=3923477/4857404, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
 md128: ios=3923477/4857404, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=1638145/2116515, aggrmerge=323580/311709, aggrticks=181144/169117, aggrin_queue=11661, aggrutil=87.44%
 sdd: ios=1962398/2416430, merge=341/10761, ticks=141224/141199, in_queue=13519, util=80.41%
 sda: ios=1313892/1816601, merge=646819/612657, ticks=221064/197036, in_queue=9804, util=87.44%

fio-seq-RW.fio:

ssd array:

NOTE: filesystem BLOCK_SIZE=”512”

file1: (g=0): rw=rw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=16
...
fio-3.18
Starting 4 processes
file1: Laying out IO file (1 file / 10240MiB)
Jobs: 4 (f=4): [M(4)][100.0%][eta 00m:00s] 
file1: (groupid=0, jobs=1): err= 0: pid=42891: Thu Apr 16 22:22:20 2020
 read: IOPS=1814, BW=454MiB/s (476MB/s)(400GiB/902747msec)
 slat (usec): min=13, max=2717.6k, avg=184.83, stdev=5610.28
 clat (usec): min=10, max=9105.8k, avg=5027.03, stdev=103760.79
 lat (usec): min=501, max=9106.0k, avg=5212.05, stdev=103913.63
 clat percentiles (usec):
 | 1.00th=[ 1106], 5.00th=[ 1418], 10.00th=[ 1614],
 | 20.00th=[ 1876], 30.00th=[ 2114], 40.00th=[ 2343],
 | 50.00th=[ 2573], 60.00th=[ 2900], 70.00th=[ 3359],
 | 80.00th=[ 4113], 90.00th=[ 5473], 95.00th=[ 6915],
 | 99.00th=[ 11207], 99.50th=[ 14222], 99.90th=[ 25297],
 | 99.95th=[ 40109], 99.99th=[5737808]
 bw ( KiB/s): min= 1024, max=1362944, per=36.93%, avg=686251.06, stdev=248132.23, samples=1222
 iops : min= 4, max= 5324, avg=2680.66, stdev=969.26, samples=1222
 write: IOPS=1209, BW=302MiB/s (317MB/s)(267GiB/902747msec); 0 zone resets
 slat (usec): min=26, max=3839.1k, avg=279.70, stdev=3882.85
 clat (usec): min=481, max=9105.6k, avg=4859.48, stdev=100268.29
 lat (usec): min=540, max=9105.6k, avg=5139.40, stdev=100344.72
 clat percentiles (usec):
 | 1.00th=[ 1090], 5.00th=[ 1401], 10.00th=[ 1598],
 | 20.00th=[ 1844], 30.00th=[ 2073], 40.00th=[ 2278],
 | 50.00th=[ 2540], 60.00th=[ 2835], 70.00th=[ 3261],
 | 80.00th=[ 3949], 90.00th=[ 5342], 95.00th=[ 6718],
 | 99.00th=[ 10945], 99.50th=[ 13829], 99.90th=[ 24511],
 | 99.95th=[ 34341], 99.99th=[5268046]
 bw ( KiB/s): min= 505, max=936960, per=36.89%, avg=456976.76, stdev=166758.28, samples=1223
 iops : min= 1, max= 3660, avg=1785.06, stdev=651.40, samples=1223
 lat (usec) : 20=0.01%, 500=0.01%, 750=0.13%, 1000=0.38%
 lat (msec) : 2=25.57%, 4=53.44%, 10=19.08%, 20=1.21%, 50=0.14%
 lat (msec) : 100=0.01%, 250=0.01%, 750=0.01%, 1000=0.01%, 2000=0.01%
 lat (msec) : >=2000=0.03%
 cpu : usr=1.10%, sys=51.43%, ctx=14324594, majf=0, minf=21
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=1638401,1091922,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=42892: Thu Apr 16 22:22:20 2020
 read: IOPS=1814, BW=454MiB/s (476MB/s)(400GiB/902746msec)
 slat (usec): min=11, max=3839.3k, avg=184.51, stdev=6310.31
 clat (usec): min=8, max=8994.8k, avg=5096.20, stdev=105688.68
 lat (usec): min=570, max=8994.9k, avg=5280.91, stdev=105880.31
 clat percentiles (usec):
 | 1.00th=[ 1123], 5.00th=[ 1434], 10.00th=[ 1614],
 | 20.00th=[ 1876], 30.00th=[ 2114], 40.00th=[ 2311],
 | 50.00th=[ 2573], 60.00th=[ 2900], 70.00th=[ 3326],
 | 80.00th=[ 4080], 90.00th=[ 5473], 95.00th=[ 6849],
 | 99.00th=[ 11207], 99.50th=[ 14222], 99.90th=[ 25297],
 | 99.95th=[ 40109], 99.99th=[5536482]
 bw ( KiB/s): min= 512, max=1246720, per=37.08%, avg=689074.87, stdev=243035.95, samples=1217
 iops : min= 2, max= 4870, avg=2691.69, stdev=949.36, samples=1217
 write: IOPS=1209, BW=302MiB/s (317MB/s)(267GiB/902746msec); 0 zone resets
 slat (usec): min=24, max=1319.9k, avg=278.72, stdev=1498.99
 clat (usec): min=536, max=8994.7k, avg=4753.99, stdev=98651.40
 lat (usec): min=599, max=8994.8k, avg=5032.93, stdev=98661.83
 clat percentiles (usec):
 | 1.00th=[ 1123], 5.00th=[ 1418], 10.00th=[ 1598],
 | 20.00th=[ 1860], 30.00th=[ 2073], 40.00th=[ 2278],
 | 50.00th=[ 2540], 60.00th=[ 2835], 70.00th=[ 3228],
 | 80.00th=[ 3916], 90.00th=[ 5276], 95.00th=[ 6718],
 | 99.00th=[ 10945], 99.50th=[ 13829], 99.90th=[ 24511],
 | 99.95th=[ 33162], 99.99th=[5469373]
 bw ( KiB/s): min= 505, max=813056, per=37.09%, avg=459402.89, stdev=162443.85, samples=1217
 iops : min= 1, max= 3176, avg=1794.53, stdev=634.55, samples=1217
 lat (usec) : 10=0.01%, 750=0.04%, 1000=0.39%
 lat (msec) : 2=25.53%, 4=53.81%, 10=18.85%, 20=1.17%, 50=0.16%
 lat (msec) : 100=0.01%, 250=0.01%, 750=0.01%, 1000=0.01%, 2000=0.01%
 lat (msec) : >=2000=0.03%
 cpu : usr=1.09%, sys=51.99%, ctx=14033638, majf=0, minf=21
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=1638401,1092319,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=42893: Thu Apr 16 22:22:20 2020
 read: IOPS=1814, BW=454MiB/s (476MB/s)(400GiB/902745msec)
 slat (usec): min=13, max=2572.8k, avg=178.75, stdev=4798.26
 clat (usec): min=9, max=8953.0k, avg=4957.37, stdev=100759.70
 lat (usec): min=424, max=8953.0k, avg=5136.31, stdev=100874.82
 clat percentiles (usec):
 | 1.00th=[ 1106], 5.00th=[ 1418], 10.00th=[ 1614],
 | 20.00th=[ 1876], 30.00th=[ 2089], 40.00th=[ 2311],
 | 50.00th=[ 2573], 60.00th=[ 2900], 70.00th=[ 3359],
 | 80.00th=[ 4113], 90.00th=[ 5473], 95.00th=[ 6849],
 | 99.00th=[ 11207], 99.50th=[ 14222], 99.90th=[ 25560],
 | 99.95th=[ 38536], 99.99th=[5402264]
 bw ( KiB/s): min= 512, max=1317376, per=36.90%, avg=685699.53, stdev=248396.06, samples=1223
 iops : min= 2, max= 5146, avg=2678.50, stdev=970.29, samples=1223
 write: IOPS=1211, BW=303MiB/s (317MB/s)(267GiB/902745msec); 0 zone resets
 slat (usec): min=22, max=3839.1k, avg=287.84, stdev=5264.77
 clat (usec): min=415, max=8952.5k, avg=4957.54, stdev=103987.50
 lat (usec): min=483, max=8952.6k, avg=5245.60, stdev=104124.31
 clat percentiles (usec):
 | 1.00th=[ 1090], 5.00th=[ 1401], 10.00th=[ 1598],
 | 20.00th=[ 1844], 30.00th=[ 2073], 40.00th=[ 2278],
 | 50.00th=[ 2540], 60.00th=[ 2835], 70.00th=[ 3261],
 | 80.00th=[ 3949], 90.00th=[ 5342], 95.00th=[ 6718],
 | 99.00th=[ 10814], 99.50th=[ 13698], 99.90th=[ 25035],
 | 99.95th=[ 36963], 99.99th=[5469373]
 bw ( KiB/s): min= 512, max=848896, per=36.97%, avg=457954.36, stdev=164997.43, samples=1222
 iops : min= 2, max= 3316, avg=1788.87, stdev=644.52, samples=1222
 lat (usec) : 10=0.01%, 500=0.01%, 750=0.22%, 1000=0.39%
 lat (msec) : 2=25.59%, 4=53.40%, 10=19.04%, 20=1.15%, 50=0.16%
 lat (msec) : 100=0.01%, 250=0.01%, 750=0.01%, 1000=0.01%, 2000=0.01%
 lat (msec) : >=2000=0.03%
 cpu : usr=1.09%, sys=51.41%, ctx=14245479, majf=0, minf=21
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=1638401,1093352,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=42894: Thu Apr 16 22:22:20 2020
 read: IOPS=1814, BW=454MiB/s (476MB/s)(400GiB/902746msec)
 slat (usec): min=12, max=3839.2k, avg=203.77, stdev=6385.25
 clat (usec): min=9, max=9027.8k, avg=5012.82, stdev=103448.02
 lat (usec): min=501, max=9027.0k, avg=5216.78, stdev=103648.60
 clat percentiles (usec):
 | 1.00th=[ 996], 5.00th=[ 1418], 10.00th=[ 1614],
 | 20.00th=[ 1893], 30.00th=[ 2114], 40.00th=[ 2376],
 | 50.00th=[ 2638], 60.00th=[ 2933], 70.00th=[ 3392],
 | 80.00th=[ 4113], 90.00th=[ 5473], 95.00th=[ 6849],
 | 99.00th=[ 11207], 99.50th=[ 14222], 99.90th=[ 25560],
 | 99.95th=[ 39060], 99.99th=[5402264]
 bw ( KiB/s): min= 512, max=1742848, per=36.90%, avg=685701.93, stdev=248867.68, samples=1223
 iops : min= 2, max= 6808, avg=2678.51, stdev=972.14, samples=1223
 write: IOPS=1208, BW=302MiB/s (317MB/s)(266GiB/902746msec); 0 zone resets
 slat (usec): min=20, max=953698, avg=253.24, stdev=1060.72
 clat (usec): min=462, max=9027.6k, avg=4886.78, stdev=100816.16
 lat (usec): min=506, max=9027.9k, avg=5140.24, stdev=100821.59
 clat percentiles (usec):
 | 1.00th=[ 988], 5.00th=[ 1385], 10.00th=[ 1598],
 | 20.00th=[ 1860], 30.00th=[ 2089], 40.00th=[ 2311],
 | 50.00th=[ 2573], 60.00th=[ 2868], 70.00th=[ 3294],
 | 80.00th=[ 3949], 90.00th=[ 5276], 95.00th=[ 6652],
 | 99.00th=[ 10814], 99.50th=[ 13829], 99.90th=[ 24511],
 | 99.95th=[ 36963], 99.99th=[5268046]
 bw ( KiB/s): min= 505, max=1180160, per=36.85%, avg=456424.24, stdev=166213.07, samples=1223
 iops : min= 1, max= 4610, avg=1782.90, stdev=649.27, samples=1223
 lat (usec) : 10=0.01%, 500=0.01%, 750=0.59%, 1000=0.45%
 lat (msec) : 2=23.97%, 4=54.39%, 10=19.25%, 20=1.17%, 50=0.14%
 lat (msec) : 100=0.01%, 250=0.01%, 750=0.01%, 1000=0.01%, 2000=0.01%
 lat (msec) : >=2000=0.03%
 cpu : usr=1.12%, sys=52.21%, ctx=14216879, majf=0, minf=23
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=1638401,1090585,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
 READ: bw=1815MiB/s (1903MB/s), 454MiB/s-454MiB/s (476MB/s-476MB/s), io=1600GiB (1718GB), run=902745-902747msec
 WRITE: bw=1210MiB/s (1268MB/s), 302MiB/s-303MiB/s (317MB/s-317MB/s), io=1066GiB (1145GB), run=902745-902747msec

Disk stats (read/write):
 dm-3: ios=3862919/26552569, merge=0/0, ticks=12686654/1044094996, in_queue=1056784029, util=76.85%, aggrios=3862919/26620232, aggrmerge=0/0, aggrticks=12688443/1281930380, aggrin_queue=1294621201, aggrutil=79.59%
 dm-0: ios=3862919/26620232, merge=0/0, ticks=12688443/1281930380, in_queue=1294621201, util=79.59%, aggrios=4339895/26927913, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
 md128: ios=4339895/26927913, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=2048871/356161, aggrmerge=101818/13108098, aggrticks=6068873/720584, aggrin_queue=5628966, aggrutil=70.02%
 sdd: ios=2105652/369456, merge=57410/13091934, ticks=3621523/872452, in_queue=3320778, util=67.30%
 sda: ios=1992090/342866, merge=146227/13124263, ticks=8516224/568716, in_queue=7937154, util=70.02%

btrfs filesystem tests:

fio-rand-RW.fio:

ssd array:

file1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.18
Starting 4 processes
file1: Laying out IO file (1 file / 10240MiB)
Jobs: 4 (f=4): [m(4)][100.0%][r=174MiB/s,w=117MiB/s][r=44.5k,w=29.9k IOPS][eta 00m:00s] 
file1: (groupid=0, jobs=1): err= 0: pid=488554: Wed Apr 15 09:15:29 2020
 read: IOPS=6112, BW=23.9MiB/s (25.0MB/s)(20.0GiB/900103msec)
 slat (nsec): min=571, max=1947.5M, avg=67475.51, stdev=2205656.85
 clat (usec): min=22, max=3821.9k, avg=1499.71, stdev=30697.76
 lat (usec): min=23, max=3821.9k, avg=1567.27, stdev=30783.88
 clat percentiles (usec):
 | 1.00th=[ 46], 5.00th=[ 61], 10.00th=[ 75],
 | 20.00th=[ 99], 30.00th=[ 127], 40.00th=[ 172],
 | 50.00th=[ 245], 60.00th=[ 396], 70.00th=[ 627],
 | 80.00th=[ 963], 90.00th=[ 1516], 95.00th=[ 2278],
 | 99.00th=[ 5473], 99.50th=[ 7308], 99.90th=[ 37487],
 | 99.95th=[ 968885], 99.99th=[1367344]
 bw ( KiB/s): min= 16, max=151848, per=35.00%, avg=34250.51, stdev=31158.93, samples=1285
 iops : min= 4, max=37962, avg=8562.63, stdev=7789.73, samples=1285
 write: IOPS=4075, BW=15.9MiB/s (16.7MB/s)(13.0GiB/900103msec); 0 zone resets
 slat (usec): min=2, max=1557.9k, avg=140.13, stdev=11835.59
 clat (nsec): min=1853, max=3821.8M, avg=1432078.82, stdev=29974401.79
 lat (usec): min=26, max=3821.8k, avg=1572.28, stdev=32228.56
 clat percentiles (usec):
 | 1.00th=[ 46], 5.00th=[ 61], 10.00th=[ 75],
 | 20.00th=[ 100], 30.00th=[ 128], 40.00th=[ 172],
 | 50.00th=[ 245], 60.00th=[ 396], 70.00th=[ 627],
 | 80.00th=[ 963], 90.00th=[ 1516], 95.00th=[ 2278],
 | 99.00th=[ 5473], 99.50th=[ 7242], 99.90th=[ 29754],
 | 99.95th=[ 918553], 99.99th=[1350566]
 bw ( KiB/s): min= 24, max=101080, per=35.05%, avg=22854.64, stdev=20765.75, samples=1284
 iops : min= 6, max=25270, avg=5713.66, stdev=5191.44, samples=1284
 lat (usec) : 2=0.01%, 50=1.89%, 100=18.49%, 250=30.01%, 500=14.57%
 lat (usec) : 750=9.24%, 1000=6.71%
 lat (msec) : 2=12.98%, 4=4.01%, 10=1.84%, 20=0.13%, 50=0.03%
 lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.02%
 lat (msec) : 2000=0.05%, >=2000=0.01%
 cpu : usr=0.95%, sys=7.46%, ctx=1312493, majf=0, minf=22
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=5501661,3668295,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=488555: Wed Apr 15 09:15:29 2020
 read: IOPS=6112, BW=23.9MiB/s (25.0MB/s)(20.0GiB/900104msec)
 slat (nsec): min=561, max=1465.5M, avg=67391.92, stdev=2092061.09
 clat (usec): min=21, max=3823.5k, avg=1510.35, stdev=31053.86
 lat (usec): min=22, max=3823.5k, avg=1577.82, stdev=31126.44
 clat percentiles (usec):
 | 1.00th=[ 46], 5.00th=[ 61], 10.00th=[ 74],
 | 20.00th=[ 98], 30.00th=[ 126], 40.00th=[ 169],
 | 50.00th=[ 245], 60.00th=[ 396], 70.00th=[ 627],
 | 80.00th=[ 963], 90.00th=[ 1516], 95.00th=[ 2278],
 | 99.00th=[ 5538], 99.50th=[ 7373], 99.90th=[ 36439],
 | 99.95th=[ 977273], 99.99th=[1367344]
 bw ( KiB/s): min= 16, max=148960, per=34.95%, avg=34200.94, stdev=31141.73, samples=1287
 iops : min= 4, max=37240, avg=8550.23, stdev=7785.43, samples=1287
 write: IOPS=4076, BW=15.9MiB/s (16.7MB/s)(13.0GiB/900104msec); 0 zone resets
 slat (usec): min=2, max=1947.7k, avg=140.52, stdev=11867.77
 clat (usec): min=4, max=3823.5k, avg=1415.34, stdev=29048.97
 lat (usec): min=27, max=3823.5k, avg=1555.92, stdev=31388.25
 clat percentiles (usec):
 | 1.00th=[ 45], 5.00th=[ 61], 10.00th=[ 75],
 | 20.00th=[ 99], 30.00th=[ 128], 40.00th=[ 172],
 | 50.00th=[ 247], 60.00th=[ 396], 70.00th=[ 627],
 | 80.00th=[ 963], 90.00th=[ 1516], 95.00th=[ 2311],
 | 99.00th=[ 5538], 99.50th=[ 7373], 99.90th=[ 29230],
 | 99.95th=[ 910164], 99.99th=[1350566]
 bw ( KiB/s): min= 8, max=100416, per=35.00%, avg=22827.97, stdev=20768.03, samples=1286
 iops : min= 2, max=25104, avg=5706.99, stdev=5192.01, samples=1286
 lat (usec) : 10=0.01%, 50=1.97%, 100=18.55%, 250=29.86%, 500=14.57%
 lat (usec) : 750=9.25%, 1000=6.73%
 lat (msec) : 2=12.94%, 4=4.03%, 10=1.84%, 20=0.14%, 50=0.03%
 lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.02%
 lat (msec) : 2000=0.05%, >=2000=0.01%
 cpu : usr=0.97%, sys=7.30%, ctx=1311783, majf=0, minf=23
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=5502258,3669714,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=488556: Wed Apr 15 09:15:29 2020
 read: IOPS=6116, BW=23.9MiB/s (25.1MB/s)(21.0GiB/900102msec)
 slat (nsec): min=541, max=1557.7M, avg=68480.10, stdev=2406561.16
 clat (usec): min=21, max=3848.9k, avg=1499.25, stdev=30707.96
 lat (usec): min=22, max=3848.9k, avg=1567.81, stdev=30804.46
 clat percentiles (usec):
 | 1.00th=[ 45], 5.00th=[ 61], 10.00th=[ 74],
 | 20.00th=[ 98], 30.00th=[ 126], 40.00th=[ 169],
 | 50.00th=[ 245], 60.00th=[ 396], 70.00th=[ 627],
 | 80.00th=[ 963], 90.00th=[ 1516], 95.00th=[ 2278],
 | 99.00th=[ 5538], 99.50th=[ 7373], 99.90th=[ 38011],
 | 99.95th=[ 968885], 99.99th=[1367344]
 bw ( KiB/s): min= 32, max=153216, per=34.98%, avg=34222.23, stdev=31198.49, samples=1287
 iops : min= 8, max=38304, avg=8555.56, stdev=7799.62, samples=1287
 write: IOPS=4075, BW=15.9MiB/s (16.7MB/s)(13.0GiB/900102msec); 0 zone resets
 slat (usec): min=2, max=1947.5k, avg=138.80, stdev=11780.17
 clat (usec): min=8, max=3848.7k, avg=1431.08, stdev=29641.13
 lat (usec): min=26, max=3848.8k, avg=1569.95, stdev=31905.49
 clat percentiles (usec):
 | 1.00th=[ 45], 5.00th=[ 61], 10.00th=[ 75],
 | 20.00th=[ 99], 30.00th=[ 127], 40.00th=[ 172],
 | 50.00th=[ 245], 60.00th=[ 396], 70.00th=[ 627],
 | 80.00th=[ 963], 90.00th=[ 1516], 95.00th=[ 2278],
 | 99.00th=[ 5538], 99.50th=[ 7308], 99.90th=[ 28705],
 | 99.95th=[ 926942], 99.99th=[1350566]
 bw ( KiB/s): min= 32, max=101216, per=34.96%, avg=22801.94, stdev=20801.58, samples=1287
 iops : min= 8, max=25304, avg=5700.48, stdev=5200.39, samples=1287
 lat (usec) : 10=0.01%, 50=2.01%, 100=18.63%, 250=29.77%, 500=14.52%
 lat (usec) : 750=9.25%, 1000=6.72%
 lat (msec) : 2=13.00%, 4=3.98%, 10=1.85%, 20=0.14%, 50=0.03%
 lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.02%
 lat (msec) : 2000=0.05%, >=2000=0.01%
 cpu : usr=0.95%, sys=7.35%, ctx=1308964, majf=0, minf=23
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=5505680,3668390,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=488557: Wed Apr 15 09:15:29 2020
 read: IOPS=6120, BW=23.9MiB/s (25.1MB/s)(21.0GiB/900102msec)
 slat (nsec): min=531, max=1520.5M, avg=68615.13, stdev=2477276.87
 clat (usec): min=22, max=3761.5k, avg=1504.46, stdev=30983.41
 lat (usec): min=23, max=3761.5k, avg=1573.15, stdev=31084.50
 clat percentiles (usec):
 | 1.00th=[ 45], 5.00th=[ 61], 10.00th=[ 74],
 | 20.00th=[ 98], 30.00th=[ 126], 40.00th=[ 169],
 | 50.00th=[ 245], 60.00th=[ 396], 70.00th=[ 627],
 | 80.00th=[ 963], 90.00th=[ 1516], 95.00th=[ 2278],
 | 99.00th=[ 5538], 99.50th=[ 7373], 99.90th=[ 37487],
 | 99.95th=[ 968885], 99.99th=[1367344]
 bw ( KiB/s): min= 40, max=152888, per=35.08%, avg=34321.05, stdev=31303.41, samples=1284
 iops : min= 10, max=38222, avg=8580.26, stdev=7825.85, samples=1284
 write: IOPS=4075, BW=15.9MiB/s (16.7MB/s)(13.0GiB/900102msec); 0 zone resets
 slat (nsec): min=1953, max=1947.7M, avg=137847.28, stdev=11731710.62
 clat (usec): min=3, max=3761.5k, avg=1421.93, stdev=29600.36
 lat (usec): min=26, max=3761.5k, avg=1559.84, stdev=31849.23
 clat percentiles (usec):
 | 1.00th=[ 45], 5.00th=[ 61], 10.00th=[ 74],
 | 20.00th=[ 99], 30.00th=[ 127], 40.00th=[ 172],
 | 50.00th=[ 245], 60.00th=[ 396], 70.00th=[ 627],
 | 80.00th=[ 963], 90.00th=[ 1516], 95.00th=[ 2278],
 | 99.00th=[ 5473], 99.50th=[ 7308], 99.90th=[ 28181],
 | 99.95th=[ 910164], 99.99th=[1350566]
 bw ( KiB/s): min= 16, max=100128, per=35.05%, avg=22856.83, stdev=20864.27, samples=1284
 iops : min= 4, max=25032, avg=5714.21, stdev=5216.07, samples=1284
 lat (usec) : 4=0.01%, 50=2.04%, 100=18.75%, 250=29.65%, 500=14.54%
 lat (usec) : 750=9.23%, 1000=6.73%
 lat (msec) : 2=12.96%, 4=4.00%, 10=1.85%, 20=0.13%, 50=0.03%
 lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.02%
 lat (msec) : 2000=0.05%, >=2000=0.01%
 cpu : usr=0.94%, sys=7.63%, ctx=1325019, majf=0, minf=22
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=5508710,3668641,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
 READ: bw=95.6MiB/s (100MB/s), 23.9MiB/s-23.9MiB/s (25.0MB/s-25.1MB/s), io=83.0GiB (90.2GB), run=900102-900104msec
 WRITE: bw=63.7MiB/s (66.8MB/s), 15.9MiB/s-15.9MiB/s (16.7MB/s-16.7MB/s), io=55.0GiB (60.1GB), run=900102-900104msec

NOTE: deleting 10gb test file tooke like one minute or more on btrfs.

ssd-test.fio:

ssd array:

seq-read: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
rand-read: (g=1): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
seq-write: (g=2): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
rand-write: (g=3): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
fio-3.18
Starting 4 processes
seq-read: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [_(3),w(1)][95.6%][w=145MiB/s][w=37.2k IOPS][eta 00m:11s] 
seq-read: (groupid=0, jobs=1): err= 0: pid=491399: Wed Apr 15 09:23:03 2020
 read: IOPS=47.3k, BW=185MiB/s (194MB/s)(10.0GiB/55385msec)
 slat (usec): min=2, max=340, avg=10.66, stdev= 4.89
 clat (usec): min=21, max=15591, avg=73.22, stdev=41.49
 lat (usec): min=43, max=15630, avg=84.02, stdev=39.98
 clat percentiles (usec):
 | 1.00th=[ 44], 5.00th=[ 49], 10.00th=[ 52], 20.00th=[ 54],
 | 30.00th=[ 55], 40.00th=[ 59], 50.00th=[ 74], 60.00th=[ 84],
 | 70.00th=[ 86], 80.00th=[ 89], 90.00th=[ 93], 95.00th=[ 96],
 | 99.00th=[ 178], 99.50th=[ 212], 99.90th=[ 326], 99.95th=[ 375],
 | 99.99th=[ 1795]
 bw ( KiB/s): min=171528, max=198360, per=100.00%, avg=189631.37, stdev=5075.54, samples=110
 iops : min=42882, max=49590, avg=47407.82, stdev=1268.90, samples=110
 lat (usec) : 50=6.37%, 100=90.79%, 250=2.65%, 500=0.16%, 750=0.01%
 lat (usec) : 1000=0.01%
 lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
 cpu : usr=5.30%, sys=49.00%, ctx=971079, majf=0, minf=15
 IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=2621440,0,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=4
rand-read: (groupid=1, jobs=1): err= 0: pid=491498: Wed Apr 15 09:23:03 2020
 read: IOPS=21.3k, BW=83.1MiB/s (87.2MB/s)(4989MiB/60001msec)
 slat (usec): min=3, max=767, avg= 8.89, stdev= 2.77
 clat (usec): min=14, max=30037, avg=178.14, stdev=121.64
 lat (usec): min=72, max=30045, avg=187.17, stdev=121.07
 clat percentiles (usec):
 | 1.00th=[ 88], 5.00th=[ 91], 10.00th=[ 93], 20.00th=[ 96],
 | 30.00th=[ 103], 40.00th=[ 120], 50.00th=[ 137], 60.00th=[ 172],
 | 70.00th=[ 231], 80.00th=[ 258], 90.00th=[ 326], 95.00th=[ 351],
 | 99.00th=[ 445], 99.50th=[ 498], 99.90th=[ 660], 99.95th=[ 840],
 | 99.99th=[ 2442]
 bw ( KiB/s): min=76015, max=88776, per=100.00%, avg=85235.61, stdev=2436.96, samples=119
 iops : min=19003, max=22194, avg=21308.89, stdev=609.28, samples=119
 lat (usec) : 20=0.01%, 50=0.01%, 100=27.27%, 250=47.88%, 500=24.35%
 lat (usec) : 750=0.43%, 1000=0.03%
 lat (msec) : 2=0.02%, 4=0.02%, 10=0.01%, 20=0.01%, 50=0.01%
 cpu : usr=3.30%, sys=20.80%, ctx=1025418, majf=0, minf=14
 IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=1277191,0,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=4
seq-write: (groupid=2, jobs=1): err= 0: pid=491822: Wed Apr 15 09:23:03 2020
 write: IOPS=39.7k, BW=155MiB/s (162MB/s)(9295MiB/60001msec); 0 zone resets
 slat (usec): min=4, max=296, avg=11.64, stdev= 2.44
 clat (usec): min=27, max=102728, avg=88.55, stdev=222.10
 lat (usec): min=48, max=102736, avg=100.32, stdev=222.12
 clat percentiles (usec):
 | 1.00th=[ 57], 5.00th=[ 64], 10.00th=[ 67], 20.00th=[ 70],
 | 30.00th=[ 71], 40.00th=[ 73], 50.00th=[ 84], 60.00th=[ 96],
 | 70.00th=[ 99], 80.00th=[ 102], 90.00th=[ 105], 95.00th=[ 110],
 | 99.00th=[ 204], 99.50th=[ 239], 99.90th=[ 379], 99.95th=[ 586],
 | 99.99th=[ 2900]
 bw ( KiB/s): min=76064, max=170440, per=100.00%, avg=158811.59, stdev=9809.61, samples=119
 iops : min=19016, max=42610, avg=39702.91, stdev=2452.40, samples=119
 lat (usec) : 50=0.23%, 100=74.24%, 250=25.14%, 500=0.33%, 750=0.02%
 lat (usec) : 1000=0.01%
 lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
 lat (msec) : 100=0.01%, 250=0.01%
 cpu : usr=6.06%, sys=48.71%, ctx=1685703, majf=0, minf=14
 IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=0,2379610,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=4
rand-write: (groupid=3, jobs=1): err= 0: pid=492071: Wed Apr 15 09:23:03 2020
 write: IOPS=36.8k, BW=144MiB/s (151MB/s)(8626MiB/60002msec); 0 zone resets
 slat (usec): min=5, max=160439, avg=14.43, stdev=232.09
 clat (usec): min=26, max=160506, avg=93.44, stdev=486.35
 lat (usec): min=46, max=179366, avg=108.00, stdev=541.71
 clat percentiles (usec):
 | 1.00th=[ 46], 5.00th=[ 50], 10.00th=[ 54], 20.00th=[ 61],
 | 30.00th=[ 67], 40.00th=[ 70], 50.00th=[ 74], 60.00th=[ 94],
 | 70.00th=[ 98], 80.00th=[ 103], 90.00th=[ 125], 95.00th=[ 163],
 | 99.00th=[ 249], 99.50th=[ 302], 99.90th=[ 1045], 99.95th=[ 2024],
 | 99.99th=[12387]
 bw ( KiB/s): min=34888, max=163664, per=100.00%, avg=147452.25, stdev=19099.00, samples=119
 iops : min= 8722, max=40916, avg=36863.08, stdev=4774.77, samples=119
 lat (usec) : 50=5.95%, 100=68.34%, 250=24.72%, 500=0.84%, 750=0.04%
 lat (usec) : 1000=0.02%
 lat (msec) : 2=0.05%, 4=0.03%, 10=0.01%, 20=0.01%, 50=0.01%
 lat (msec) : 100=0.01%, 250=0.01%
 cpu : usr=6.21%, sys=51.58%, ctx=1384080, majf=0, minf=12
 IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=0,2208289,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
 READ: bw=185MiB/s (194MB/s), 185MiB/s-185MiB/s (194MB/s-194MB/s), io=10.0GiB (10.7GB), run=55385-55385msec

Run status group 1 (all jobs):
 READ: bw=83.1MiB/s (87.2MB/s), 83.1MiB/s-83.1MiB/s (87.2MB/s-87.2MB/s), io=4989MiB (5231MB), run=60001-60001msec

Run status group 2 (all jobs):
 WRITE: bw=155MiB/s (162MB/s), 155MiB/s-155MiB/s (162MB/s-162MB/s), io=9295MiB (9747MB), run=60001-60001msec

Run status group 3 (all jobs):
 WRITE: bw=144MiB/s (151MB/s), 144MiB/s-144MiB/s (151MB/s-151MB/s), io=8626MiB (9045MB), run=60002-60002msec

fio-seq-RW.fio:

ssd array:

file1: (g=0): rw=rw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=16
...
fio-3.18
Starting 4 processes
file1: Laying out IO file (1 file / 10240MiB)
Jobs: 3 (f=3): [M(3),_(1)][100.0%][r=2813KiB/s,w=1278KiB/s][r=10,w=4 IOPS][eta 00m:00s] 
file1: (groupid=0, jobs=1): err= 0: pid=46321: Thu Apr 16 23:02:16 2020
 read: IOPS=1908, BW=477MiB/s (500MB/s)(420GiB/901390msec)
 slat (usec): min=10, max=3292.9k, avg=204.17, stdev=6557.26
 clat (usec): min=390, max=9498.3k, avg=4851.96, stdev=113854.72
 lat (usec): min=418, max=9498.4k, avg=5056.25, stdev=114423.64
 clat percentiles (usec):
 | 1.00th=[ 685], 5.00th=[ 857], 10.00th=[ 955],
 | 20.00th=[ 1074], 30.00th=[ 1188], 40.00th=[ 1287],
 | 50.00th=[ 1418], 60.00th=[ 1631], 70.00th=[ 2180],
 | 80.00th=[ 3032], 90.00th=[ 4752], 95.00th=[ 7832],
 | 99.00th=[ 17171], 99.50th=[ 20055], 99.90th=[ 59507],
 | 99.95th=[ 918553], 99.99th=[5737808]
 bw ( KiB/s): min= 512, max=2663424, per=41.53%, avg=811549.54, stdev=498872.78, samples=1085
 iops : min= 2, max=10404, avg=3170.11, stdev=1948.72, samples=1085
 write: IOPS=1272, BW=318MiB/s (333MB/s)(280GiB/901390msec); 0 zone resets
 slat (usec): min=17, max=7010.7k, avg=246.03, stdev=18576.42
 clat (usec): min=7, max=9498.2k, avg=4513.35, stdev=108187.37
 lat (usec): min=433, max=9498.3k, avg=4759.52, stdev=109821.97
 clat percentiles (usec):
 | 1.00th=[ 668], 5.00th=[ 848], 10.00th=[ 938],
 | 20.00th=[ 1074], 30.00th=[ 1172], 40.00th=[ 1270],
 | 50.00th=[ 1385], 60.00th=[ 1565], 70.00th=[ 1991],
 | 80.00th=[ 2835], 90.00th=[ 4490], 95.00th=[ 7504],
 | 99.00th=[ 16909], 99.50th=[ 19792], 99.90th=[ 50594],
 | 99.95th=[ 350225], 99.99th=[5603591]
 bw ( KiB/s): min= 512, max=1752064, per=41.84%, avg=544907.86, stdev=330632.85, samples=1077
 iops : min= 2, max= 6844, avg=2128.55, stdev=1291.53, samples=1077
 lat (usec) : 10=0.01%, 500=0.02%, 750=2.13%, 1000=11.57%
 lat (msec) : 2=54.87%, 4=18.91%, 10=9.22%, 20=2.79%, 50=0.38%
 lat (msec) : 100=0.03%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.04%
 cpu : usr=0.89%, sys=52.00%, ctx=11350080, majf=0, minf=19
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=1720322,1146597,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=46322: Thu Apr 16 23:02:16 2020
 read: IOPS=1908, BW=477MiB/s (500MB/s)(420GiB/901390msec)
 slat (usec): min=10, max=3293.5k, avg=200.37, stdev=6405.68
 clat (usec): min=323, max=9485.2k, avg=4939.08, stdev=116035.25
 lat (usec): min=344, max=9485.2k, avg=5139.56, stdev=116579.45
 clat percentiles (usec):
 | 1.00th=[ 652], 5.00th=[ 865], 10.00th=[ 963],
 | 20.00th=[ 1090], 30.00th=[ 1188], 40.00th=[ 1303],
 | 50.00th=[ 1434], 60.00th=[ 1631], 70.00th=[ 2147],
 | 80.00th=[ 2999], 90.00th=[ 4621], 95.00th=[ 7635],
 | 99.00th=[ 17171], 99.50th=[ 20055], 99.90th=[ 58983],
 | 99.95th=[1216349], 99.99th=[5737808]
 bw ( KiB/s): min= 512, max=2759680, per=42.15%, avg=823612.52, stdev=494622.50, samples=1069
 iops : min= 2, max=10780, avg=3217.23, stdev=1932.12, samples=1069
 write: IOPS=1272, BW=318MiB/s (334MB/s)(280GiB/901390msec); 0 zone resets
 slat (usec): min=17, max=7010.9k, avg=246.65, stdev=18480.59
 clat (usec): min=8, max=9484.8k, avg=4382.28, stdev=105404.90
 lat (usec): min=356, max=9484.9k, avg=4629.06, stdev=107053.86
 clat percentiles (usec):
 | 1.00th=[ 644], 5.00th=[ 848], 10.00th=[ 955],
 | 20.00th=[ 1074], 30.00th=[ 1172], 40.00th=[ 1270],
 | 50.00th=[ 1401], 60.00th=[ 1565], 70.00th=[ 1958],
 | 80.00th=[ 2802], 90.00th=[ 4424], 95.00th=[ 7439],
 | 99.00th=[ 16909], 99.50th=[ 19530], 99.90th=[ 45876],
 | 99.95th=[ 299893], 99.99th=[5670700]
 bw ( KiB/s): min= 512, max=1901056, per=42.32%, avg=551189.49, stdev=329238.16, samples=1065
 iops : min= 2, max= 7426, avg=2153.08, stdev=1286.09, samples=1065
 lat (usec) : 10=0.01%, 500=0.12%, 750=2.12%, 1000=10.72%
 lat (msec) : 2=56.17%, 4=18.62%, 10=9.04%, 20=2.73%, 50=0.38%
 lat (msec) : 100=0.03%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.04%
 cpu : usr=0.87%, sys=52.48%, ctx=11428364, majf=0, minf=21
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=1720153,1146882,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=46323: Thu Apr 16 23:02:16 2020
 read: IOPS=1907, BW=477MiB/s (500MB/s)(420GiB/901390msec)
 slat (usec): min=10, max=3293.0k, avg=199.17, stdev=5943.95
 clat (usec): min=426, max=9500.2k, avg=4796.43, stdev=112188.75
 lat (usec): min=453, max=9500.2k, avg=4995.71, stdev=112701.60
 clat percentiles (usec):
 | 1.00th=[ 685], 5.00th=[ 865], 10.00th=[ 955],
 | 20.00th=[ 1090], 30.00th=[ 1188], 40.00th=[ 1287],
 | 50.00th=[ 1418], 60.00th=[ 1631], 70.00th=[ 2147],
 | 80.00th=[ 3032], 90.00th=[ 4686], 95.00th=[ 7767],
 | 99.00th=[ 16909], 99.50th=[ 19792], 99.90th=[ 56361],
 | 99.95th=[ 759170], 99.99th=[5737808]
 bw ( KiB/s): min= 512, max=2656256, per=41.86%, avg=817886.86, stdev=490288.18, samples=1076
 iops : min= 2, max=10376, avg=3194.87, stdev=1915.19, samples=1076
 write: IOPS=1272, BW=318MiB/s (334MB/s)(280GiB/901390msec); 0 zone resets
 slat (usec): min=19, max=7010.9k, avg=252.68, stdev=18856.09
 clat (usec): min=8, max=9500.1k, avg=4599.49, stdev=112152.04
 lat (usec): min=466, max=9500.2k, avg=4852.30, stdev=113800.28
 clat percentiles (usec):
 | 1.00th=[ 676], 5.00th=[ 848], 10.00th=[ 947],
 | 20.00th=[ 1074], 30.00th=[ 1172], 40.00th=[ 1270],
 | 50.00th=[ 1385], 60.00th=[ 1565], 70.00th=[ 1975],
 | 80.00th=[ 2835], 90.00th=[ 4490], 95.00th=[ 7439],
 | 99.00th=[ 16909], 99.50th=[ 19792], 99.90th=[ 50594],
 | 99.95th=[ 434111], 99.99th=[5737808]
 bw ( KiB/s): min= 512, max=1813504, per=42.28%, avg=550665.49, stdev=324374.68, samples=1066
 iops : min= 2, max= 7084, avg=2151.03, stdev=1267.09, samples=1066
 lat (usec) : 10=0.01%, 500=0.01%, 750=1.98%, 1000=11.29%
 lat (msec) : 2=55.58%, 4=18.75%, 10=9.13%, 20=2.79%, 50=0.37%
 lat (msec) : 100=0.03%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.04%
 cpu : usr=0.87%, sys=52.21%, ctx=11309634, majf=0, minf=22
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=1719377,1146884,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=46324: Thu Apr 16 23:02:16 2020
 read: IOPS=1910, BW=478MiB/s (501MB/s)(420GiB/900276msec)
 slat (usec): min=10, max=3292.0k, avg=208.68, stdev=7003.33
 clat (usec): min=7, max=9491.9k, avg=4823.59, stdev=112823.41
 lat (usec): min=456, max=9491.0k, avg=5032.38, stdev=113451.79
 clat percentiles (usec):
 | 1.00th=[ 668], 5.00th=[ 857], 10.00th=[ 955],
 | 20.00th=[ 1090], 30.00th=[ 1188], 40.00th=[ 1303],
 | 50.00th=[ 1434], 60.00th=[ 1680], 70.00th=[ 2245],
 | 80.00th=[ 3064], 90.00th=[ 4752], 95.00th=[ 7832],
 | 99.00th=[ 17171], 99.50th=[ 20055], 99.90th=[ 58983],
 | 99.95th=[ 692061], 99.99th=[5872026]
 bw ( KiB/s): min= 512, max=2155008, per=41.53%, avg=811541.40, stdev=486199.95, samples=1085
 iops : min= 2, max= 8418, avg=3170.08, stdev=1899.22, samples=1085
 write: IOPS=1272, BW=318MiB/s (333MB/s)(280GiB/900276msec); 0 zone resets
 slat (usec): min=19, max=7010.8k, avg=246.79, stdev=18222.38
 clat (usec): min=406, max=9491.6k, avg=4546.45, stdev=108194.79
 lat (usec): min=462, max=9491.7k, avg=4793.38, stdev=109812.04
 clat percentiles (usec):
 | 1.00th=[ 660], 5.00th=[ 848], 10.00th=[ 947],
 | 20.00th=[ 1074], 30.00th=[ 1172], 40.00th=[ 1287],
 | 50.00th=[ 1401], 60.00th=[ 1598], 70.00th=[ 2057],
 | 80.00th=[ 2900], 90.00th=[ 4555], 95.00th=[ 7504],
 | 99.00th=[ 16909], 99.50th=[ 19792], 99.90th=[ 52691],
 | 99.95th=[ 358613], 99.99th=[5737808]
 bw ( KiB/s): min= 512, max=1421824, per=41.56%, avg=541277.40, stdev=323462.89, samples=1083
 iops : min= 2, max= 5554, avg=2114.36, stdev=1263.52, samples=1083
 lat (usec) : 10=0.01%, 500=0.02%, 750=2.22%, 1000=10.98%
 lat (msec) : 2=54.39%, 4=19.69%, 10=9.43%, 20=2.78%, 50=0.38%
 lat (msec) : 100=0.03%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.04%
 cpu : usr=0.87%, sys=51.56%, ctx=10983262, majf=0, minf=22
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=1720324,1145310,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
 READ: bw=1908MiB/s (2001MB/s), 477MiB/s-478MiB/s (500MB/s-501MB/s), io=1680GiB (1804GB), run=900276-901390msec
 WRITE: bw=1272MiB/s (1334MB/s), 318MiB/s-318MiB/s (333MB/s-334MB/s), io=1120GiB (1202GB), run=900276-901390msec

ext4 filesystem tests:

fio-rand-RW.fio:

ssd array:

file1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.18
Starting 4 processes
file1: Laying out IO file (1 file / 10240MiB)
Jobs: 4 (f=4): [m(4)][100.0%][eta 00m:00s] 
file1: (groupid=0, jobs=1): err= 0: pid=4187: Thu Apr 16 20:47:28 2020
 read: IOPS=26.8k, BW=105MiB/s (110MB/s)(92.2GiB/902250msec)
 slat (nsec): min=521, max=22937M, avg=24337.34, stdev=12851101.99
 clat (usec): min=7, max=25704k, avg=336.13, stdev=58337.67
 lat (usec): min=22, max=25704k, avg=360.52, stdev=59985.21
 clat percentiles (usec):
 | 1.00th=[ 26], 5.00th=[ 29], 10.00th=[ 31], 20.00th=[ 35],
 | 30.00th=[ 38], 40.00th=[ 42], 50.00th=[ 46], 60.00th=[ 52],
 | 70.00th=[ 72], 80.00th=[ 178], 90.00th=[ 351], 95.00th=[ 570],
 | 99.00th=[ 1385], 99.50th=[ 1860], 99.90th=[ 2868], 99.95th=[ 3359],
 | 99.99th=[ 5735]
 bw ( KiB/s): min= 8, max=850920, per=54.82%, avg=234830.66, stdev=237231.04, samples=823
 iops : min= 2, max=212730, avg=58707.67, stdev=59307.77, samples=823
 write: IOPS=17.8k, BW=69.7MiB/s (73.1MB/s)(61.4GiB/902250msec); 0 zone resets
 slat (nsec): min=751, max=19321M, avg=12303.51, stdev=10578386.16
 clat (usec): min=20, max=25704k, avg=336.73, stdev=60134.65
 lat (usec): min=22, max=25704k, avg=349.08, stdev=61123.80
 clat percentiles (usec):
 | 1.00th=[ 26], 5.00th=[ 29], 10.00th=[ 31], 20.00th=[ 35],
 | 30.00th=[ 39], 40.00th=[ 42], 50.00th=[ 46], 60.00th=[ 52],
 | 70.00th=[ 72], 80.00th=[ 178], 90.00th=[ 351], 95.00th=[ 570],
 | 99.00th=[ 1385], 99.50th=[ 1860], 99.90th=[ 2868], 99.95th=[ 3392],
 | 99.99th=[ 5800]
 bw ( KiB/s): min= 8, max=564768, per=54.54%, avg=155777.15, stdev=158137.96, samples=827
 iops : min= 2, max=141192, avg=38944.29, stdev=39534.49, samples=827
 lat (usec) : 10=0.01%, 50=57.99%, 100=12.64%, 250=14.02%, 500=9.16%
 lat (usec) : 750=2.97%, 1000=1.30%
 lat (msec) : 2=1.50%, 4=0.37%, 10=0.03%, 20=0.01%, 50=0.01%
 lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.01%
 cpu : usr=3.06%, sys=18.16%, ctx=6486230, majf=0, minf=24
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=24158675,16103789,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=4188: Thu Apr 16 20:47:28 2020
 read: IOPS=26.8k, BW=105MiB/s (110MB/s)(92.1GiB/902250msec)
 slat (nsec): min=511, max=19324M, avg=24262.33, stdev=11823631.98
 clat (usec): min=7, max=25701k, avg=331.05, stdev=56646.11
 lat (usec): min=22, max=25701k, avg=355.36, stdev=58246.64
 clat percentiles (usec):
 | 1.00th=[ 26], 5.00th=[ 29], 10.00th=[ 31], 20.00th=[ 35],
 | 30.00th=[ 38], 40.00th=[ 42], 50.00th=[ 46], 60.00th=[ 52],
 | 70.00th=[ 72], 80.00th=[ 178], 90.00th=[ 355], 95.00th=[ 570],
 | 99.00th=[ 1385], 99.50th=[ 1860], 99.90th=[ 2835], 99.95th=[ 3326],
 | 99.99th=[ 5735]
 bw ( KiB/s): min= 8, max=845840, per=54.28%, avg=232522.77, stdev=236563.19, samples=831
 iops : min= 2, max=211460, avg=58130.68, stdev=59140.78, samples=831
 write: IOPS=17.9k, BW=69.7MiB/s (73.1MB/s)(61.5GiB/902250msec); 0 zone resets
 slat (nsec): min=751, max=18028M, avg=12467.36, stdev=10905218.24
 clat (usec): min=20, max=25701k, avg=344.32, stdev=59589.00
 lat (usec): min=23, max=25701k, avg=356.83, stdev=60581.24
 clat percentiles (usec):
 | 1.00th=[ 26], 5.00th=[ 29], 10.00th=[ 31], 20.00th=[ 35],
 | 30.00th=[ 39], 40.00th=[ 42], 50.00th=[ 46], 60.00th=[ 52],
 | 70.00th=[ 72], 80.00th=[ 178], 90.00th=[ 351], 95.00th=[ 570],
 | 99.00th=[ 1385], 99.50th=[ 1860], 99.90th=[ 2868], 99.95th=[ 3359],
 | 99.99th=[ 5735]
 bw ( KiB/s): min= 8, max=564408, per=54.43%, avg=155455.95, stdev=157786.99, samples=829
 iops : min= 2, max=141104, avg=38863.98, stdev=39446.74, samples=829
 lat (usec) : 10=0.01%, 50=57.83%, 100=12.83%, 250=13.96%, 500=9.19%
 lat (usec) : 750=2.98%, 1000=1.29%
 lat (msec) : 2=1.50%, 4=0.37%, 10=0.03%, 20=0.01%, 50=0.01%
 lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.01%
 cpu : usr=3.09%, sys=18.11%, ctx=6347526, majf=0, minf=25
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=24153782,16109439,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=4189: Thu Apr 16 20:47:28 2020
 read: IOPS=26.8k, BW=105MiB/s (110MB/s)(92.1GiB/902250msec)
 slat (nsec): min=511, max=19321M, avg=22579.21, stdev=10573514.31
 clat (usec): min=7, max=25718k, avg=333.62, stdev=57753.62
 lat (usec): min=22, max=25718k, avg=356.25, stdev=59144.99
 clat percentiles (usec):
 | 1.00th=[ 26], 5.00th=[ 29], 10.00th=[ 31], 20.00th=[ 35],
 | 30.00th=[ 38], 40.00th=[ 42], 50.00th=[ 46], 60.00th=[ 51],
 | 70.00th=[ 71], 80.00th=[ 178], 90.00th=[ 351], 95.00th=[ 578],
 | 99.00th=[ 1385], 99.50th=[ 1860], 99.90th=[ 2868], 99.95th=[ 3392],
 | 99.99th=[ 5866]
 bw ( KiB/s): min= 8, max=850224, per=54.47%, avg=233345.08, stdev=237852.27, samples=828
 iops : min= 2, max=212556, avg=58336.27, stdev=59463.07, samples=828
 write: IOPS=17.9k, BW=69.7MiB/s (73.1MB/s)(61.4GiB/902250msec); 0 zone resets
 slat (nsec): min=742, max=18031M, avg=14999.88, stdev=12288734.93
 clat (usec): min=21, max=25718k, avg=340.53, stdev=58896.28
 lat (usec): min=22, max=25718k, avg=355.58, stdev=60247.68
 clat percentiles (usec):
 | 1.00th=[ 26], 5.00th=[ 29], 10.00th=[ 31], 20.00th=[ 35],
 | 30.00th=[ 39], 40.00th=[ 42], 50.00th=[ 46], 60.00th=[ 52],
 | 70.00th=[ 71], 80.00th=[ 178], 90.00th=[ 351], 95.00th=[ 570],
 | 99.00th=[ 1385], 99.50th=[ 1860], 99.90th=[ 2868], 99.95th=[ 3359],
 | 99.99th=[ 5800]
 bw ( KiB/s): min= 8, max=569056, per=54.36%, avg=155259.24, stdev=158673.45, samples=830
 iops : min= 2, max=142264, avg=38814.81, stdev=39668.36, samples=830
 lat (usec) : 10=0.01%, 50=58.15%, 100=12.55%, 250=13.97%, 500=9.15%
 lat (usec) : 750=2.97%, 1000=1.30%
 lat (msec) : 2=1.50%, 4=0.38%, 10=0.03%, 20=0.01%, 50=0.01%
 lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.01%
 cpu : usr=3.08%, sys=18.10%, ctx=6328985, majf=0, minf=24
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=24151688,16108475,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=4190: Thu Apr 16 20:47:28 2020
 read: IOPS=26.8k, BW=105MiB/s (110MB/s)(92.2GiB/902250msec)
 slat (nsec): min=511, max=20650M, avg=23949.60, stdev=11590584.38
 clat (usec): min=8, max=25693k, avg=338.63, stdev=58590.52
 lat (usec): min=21, max=25693k, avg=362.64, stdev=60059.25
 clat percentiles (usec):
 | 1.00th=[ 26], 5.00th=[ 29], 10.00th=[ 31], 20.00th=[ 35],
 | 30.00th=[ 38], 40.00th=[ 42], 50.00th=[ 46], 60.00th=[ 51],
 | 70.00th=[ 71], 80.00th=[ 178], 90.00th=[ 351], 95.00th=[ 570],
 | 99.00th=[ 1385], 99.50th=[ 1860], 99.90th=[ 2868], 99.95th=[ 3359],
 | 99.99th=[ 5735]
 bw ( KiB/s): min= 8, max=855224, per=54.49%, avg=233415.42, stdev=237744.80, samples=828
 iops : min= 2, max=213806, avg=58353.86, stdev=59436.20, samples=828
 write: IOPS=17.8k, BW=69.7MiB/s (73.1MB/s)(61.4GiB/902250msec); 0 zone resets
 slat (nsec): min=741, max=19324M, avg=12901.57, stdev=11251956.87
 clat (usec): min=20, max=25692k, avg=332.98, stdev=58218.59
 lat (usec): min=22, max=25692k, avg=345.93, stdev=59474.78
 clat percentiles (usec):
 | 1.00th=[ 26], 5.00th=[ 29], 10.00th=[ 31], 20.00th=[ 35],
 | 30.00th=[ 38], 40.00th=[ 42], 50.00th=[ 46], 60.00th=[ 52],
 | 70.00th=[ 72], 80.00th=[ 178], 90.00th=[ 351], 95.00th=[ 570],
 | 99.00th=[ 1385], 99.50th=[ 1860], 99.90th=[ 2868], 99.95th=[ 3392],
 | 99.99th=[ 5866]
 bw ( KiB/s): min= 8, max=572496, per=54.67%, avg=156138.08, stdev=158425.86, samples=825
 iops : min= 2, max=143124, avg=39034.52, stdev=39606.47, samples=825
 lat (usec) : 10=0.01%, 50=58.27%, 100=12.37%, 250=14.02%, 500=9.17%
 lat (usec) : 750=2.98%, 1000=1.29%
 lat (msec) : 2=1.50%, 4=0.38%, 10=0.03%, 20=0.01%, 50=0.01%
 lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.01%
 cpu : usr=2.99%, sys=18.21%, ctx=6419172, majf=0, minf=23
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=24158978,16102061,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
 READ: bw=418MiB/s (439MB/s), 105MiB/s-105MiB/s (110MB/s-110MB/s), io=369GiB (396GB), run=902250-902250msec
 WRITE: bw=279MiB/s (292MB/s), 69.7MiB/s-69.7MiB/s (73.1MB/s-73.1MB/s), io=246GiB (264GB), run=902250-902250msec

Disk stats (read/write):
 dm-4: ios=5993954/14497630, merge=0/0, ticks=1634288/1037657607, in_queue=1039291895, util=97.78%, aggrios=5998052/14574477, aggrmerge=0/0, aggrticks=2123171/1462952472, aggrin_queue=1465076704, aggrutil=97.99%
 dm-0: ios=5998052/14574477, merge=0/0, ticks=2123171/1462952472, in_queue=1465076704, util=97.99%, aggrios=5998766/14562384, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
 md128: ios=5998766/14562384, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=2999228/5929551, aggrmerge=147/1352125, aggrticks=524887/1056422, aggrin_queue=47791, aggrutil=97.33%
 sdd: ios=2997639/5937272, merge=110/1346600, ticks=387298/1487985, in_queue=87156, util=93.40%
 sda: ios=3000817/5921831, merge=185/1357651, ticks=662477/624859, in_queue=8426, util=97.33%

ssd-test.fio:

ssd array:

seq-read: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
rand-read: (g=1): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
seq-write: (g=2): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
rand-write: (g=3): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
fio-3.18
Starting 4 processes
seq-read: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [_(3),w(1)][99.2%][w=169MiB/s][w=43.3k IOPS][eta 00m:02s] 
seq-read: (groupid=0, jobs=1): err= 0: pid=25992: Thu Apr 16 20:59:39 2020
 read: IOPS=47.4k, BW=185MiB/s (194MB/s)(10.0GiB/55281msec)
 slat (nsec): min=1683, max=301896, avg=6826.06, stdev=3563.61
 clat (usec): min=16, max=7686, avg=76.89, stdev=41.30
 lat (usec): min=39, max=7693, avg=83.83, stdev=40.16
 clat percentiles (usec):
 | 1.00th=[ 46], 5.00th=[ 55], 10.00th=[ 57], 20.00th=[ 59],
 | 30.00th=[ 61], 40.00th=[ 63], 50.00th=[ 75], 60.00th=[ 87],
 | 70.00th=[ 90], 80.00th=[ 92], 90.00th=[ 95], 95.00th=[ 97],
 | 99.00th=[ 182], 99.50th=[ 206], 99.90th=[ 289], 99.95th=[ 355],
 | 99.99th=[ 1582]
 bw ( KiB/s): min=162408, max=199288, per=100.00%, avg=189975.23, stdev=5129.09, samples=110
 iops : min=40602, max=49822, avg=47493.80, stdev=1282.28, samples=110
 lat (usec) : 20=0.01%, 50=2.25%, 100=94.94%, 250=2.66%, 500=0.13%
 lat (usec) : 750=0.01%, 1000=0.01%
 lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%
 cpu : usr=5.25%, sys=33.18%, ctx=1879641, majf=0, minf=12
 IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=2621440,0,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=4
rand-read: (groupid=1, jobs=1): err= 0: pid=26072: Thu Apr 16 20:59:39 2020
 read: IOPS=21.1k, BW=82.6MiB/s (86.6MB/s)(4953MiB/60001msec)
 slat (usec): min=2, max=339, avg= 5.07, stdev= 1.96
 clat (usec): min=23, max=28131, avg=183.42, stdev=131.33
 lat (usec): min=64, max=28139, avg=188.62, stdev=130.84
 clat percentiles (usec):
 | 1.00th=[ 85], 5.00th=[ 88], 10.00th=[ 90], 20.00th=[ 93],
 | 30.00th=[ 96], 40.00th=[ 112], 50.00th=[ 135], 60.00th=[ 180],
 | 70.00th=[ 253], 80.00th=[ 269], 90.00th=[ 347], 95.00th=[ 367],
 | 99.00th=[ 461], 99.50th=[ 519], 99.90th=[ 660], 99.95th=[ 865],
 | 99.99th=[ 2474]
 bw ( KiB/s): min=74800, max=89048, per=100.00%, avg=84649.53, stdev=2408.84, samples=119
 iops : min=18700, max=22262, avg=21162.38, stdev=602.20, samples=119
 lat (usec) : 50=0.01%, 100=35.20%, 250=33.60%, 500=30.60%, 750=0.53%
 lat (usec) : 1000=0.02%
 lat (msec) : 2=0.02%, 4=0.02%, 10=0.01%, 20=0.01%, 50=0.01%
 cpu : usr=3.11%, sys=12.85%, ctx=978765, majf=0, minf=12
 IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=1268052,0,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=4
seq-write: (groupid=2, jobs=1): err= 0: pid=26145: Thu Apr 16 20:59:39 2020
 write: IOPS=42.8k, BW=167MiB/s (175MB/s)(9.79GiB/60003msec); 0 zone resets
 slat (nsec): min=1543, max=10207k, avg=4043.19, stdev=6419.44
 clat (usec): min=35, max=82562, avg=88.87, stdev=139.06
 lat (usec): min=44, max=82565, avg=93.03, stdev=139.21
 clat percentiles (usec):
 | 1.00th=[ 55], 5.00th=[ 74], 10.00th=[ 76], 20.00th=[ 78],
 | 30.00th=[ 79], 40.00th=[ 80], 50.00th=[ 84], 60.00th=[ 95],
 | 70.00th=[ 96], 80.00th=[ 97], 90.00th=[ 100], 95.00th=[ 102],
 | 99.00th=[ 178], 99.50th=[ 212], 99.90th=[ 302], 99.95th=[ 359],
 | 99.99th=[ 832]
 bw ( KiB/s): min=145760, max=176824, per=100.00%, avg=171344.39, stdev=4452.16, samples=119
 iops : min=36440, max=44206, avg=42836.09, stdev=1113.04, samples=119
 lat (usec) : 50=0.31%, 100=91.40%, 250=8.09%, 500=0.17%, 750=0.01%
 lat (usec) : 1000=0.01%
 lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
 lat (msec) : 100=0.01%
 cpu : usr=5.71%, sys=20.67%, ctx=1905968, majf=0, minf=14
 IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=0,2566351,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=4
rand-write: (groupid=3, jobs=1): err= 0: pid=26269: Thu Apr 16 20:59:39 2020
 write: IOPS=42.1k, BW=165MiB/s (173MB/s)(9871MiB/60001msec); 0 zone resets
 slat (nsec): min=1653, max=154630, avg=4258.08, stdev=690.88
 clat (usec): min=14, max=69914, avg=89.99, stdev=138.70
 lat (usec): min=39, max=69919, avg=94.37, stdev=138.71
 clat percentiles (usec):
 | 1.00th=[ 44], 5.00th=[ 46], 10.00th=[ 47], 20.00th=[ 50],
 | 30.00th=[ 53], 40.00th=[ 58], 50.00th=[ 67], 60.00th=[ 85],
 | 70.00th=[ 115], 80.00th=[ 127], 90.00th=[ 161], 95.00th=[ 174],
 | 99.00th=[ 260], 99.50th=[ 302], 99.90th=[ 420], 99.95th=[ 515],
 | 99.99th=[ 2311]
 bw ( KiB/s): min=136816, max=189304, per=100.00%, avg=168739.86, stdev=11097.33, samples=119
 iops : min=34204, max=47326, avg=42184.97, stdev=2774.35, samples=119
 lat (usec) : 20=0.01%, 50=22.63%, 100=42.21%, 250=33.94%, 500=1.16%
 lat (usec) : 750=0.03%, 1000=0.01%
 lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
 lat (msec) : 100=0.01%
 cpu : usr=6.15%, sys=22.35%, ctx=2177658, majf=0, minf=12
 IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=0,2526915,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
 READ: bw=185MiB/s (194MB/s), 185MiB/s-185MiB/s (194MB/s-194MB/s), io=10.0GiB (10.7GB), run=55281-55281msec

Run status group 1 (all jobs):
 READ: bw=82.6MiB/s (86.6MB/s), 82.6MiB/s-82.6MiB/s (86.6MB/s-86.6MB/s), io=4953MiB (5194MB), run=60001-60001msec

Run status group 2 (all jobs):
 WRITE: bw=167MiB/s (175MB/s), 167MiB/s-167MiB/s (175MB/s-175MB/s), io=9.79GiB (10.5GB), run=60003-60003msec

Run status group 3 (all jobs):
 WRITE: bw=165MiB/s (173MB/s), 165MiB/s-165MiB/s (173MB/s-173MB/s), io=9871MiB (10.3GB), run=60001-60001msec

Disk stats (read/write):
 dm-4: ios=3889492/5087822, merge=0/0, ticks=435735/446565, in_queue=882300, util=99.40%, aggrios=3890468/5113230, aggrmerge=0/0, aggrticks=433045/1974911, aggrin_queue=2408127, aggrutil=99.55%
 dm-0: ios=3890468/5113230, merge=0/0, ticks=433045/1974911, in_queue=2408127, util=99.55%, aggrios=3890470/5113458, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
 md128: ios=3890470/5113458, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=1621878/2239322, aggrmerge=323357/317244, aggrticks=175205/174238, aggrin_queue=1788, aggrutil=87.64%
 sdd: ios=1945635/2553523, merge=123/2765, ticks=134406/145088, in_queue=3176, util=81.04%
 sda: ios=1298121/1925121, merge=646591/631723, ticks=216004/203388, in_queue=400, util=87.64%

fio-seq-RW.fio:

ssd array:

file1: (g=0): rw=rw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=16
...
fio-3.18
Starting 4 processes
file1: Laying out IO file (1 file / 10240MiB)
Jobs: 4 (f=4): [M(4)][100.0%][r=4876MiB/s,w=3196MiB/s][r=19.5k,w=12.8k IOPS][eta 00m:00s]
file1: (groupid=0, jobs=1): err= 0: pid=31629: Thu Apr 16 22:02:18 2020
 read: IOPS=1755, BW=439MiB/s (460MB/s)(386GiB/900001msec)
 slat (usec): min=10, max=6001.5k, avg=243.42, stdev=13604.69
 clat (usec): min=239, max=10819k, avg=5185.92, stdev=109885.26
 lat (usec): min=281, max=10819k, avg=5429.46, stdev=111139.70
 clat percentiles (usec):
 | 1.00th=[ 889], 5.00th=[ 1221], 10.00th=[ 1418],
 | 20.00th=[ 1680], 30.00th=[ 1893], 40.00th=[ 2114],
 | 50.00th=[ 2376], 60.00th=[ 2737], 70.00th=[ 3261],
 | 80.00th=[ 4015], 90.00th=[ 5276], 95.00th=[ 6783],
 | 99.00th=[ 10945], 99.50th=[ 13304], 99.90th=[ 36963],
 | 99.95th=[ 792724], 99.99th=[7281312]
 bw ( KiB/s): min= 512, max=1683968, per=38.57%, avg=693499.08, stdev=320473.55, samples=1165
 iops : min= 2, max= 6578, avg=2708.97, stdev=1251.85, samples=1165
 write: IOPS=1170, BW=293MiB/s (307MB/s)(257GiB/900001msec); 0 zone resets
 slat (usec): min=24, max=4547.7k, avg=272.29, stdev=10969.43
 clat (nsec): min=1943, max=10818M, avg=5038702.95, stdev=110500052.27
 lat (usec): min=238, max=10818k, avg=5311.12, stdev=111106.80
 clat percentiles (usec):
 | 1.00th=[ 865], 5.00th=[ 1188], 10.00th=[ 1385],
 | 20.00th=[ 1631], 30.00th=[ 1844], 40.00th=[ 2040],
 | 50.00th=[ 2278], 60.00th=[ 2540], 70.00th=[ 2999],
 | 80.00th=[ 3720], 90.00th=[ 4948], 95.00th=[ 6521],
 | 99.00th=[ 10683], 99.50th=[ 13042], 99.90th=[ 36963],
 | 99.95th=[ 650118], 99.99th=[7281312]
 bw ( KiB/s): min= 512, max=1103872, per=38.91%, avg=466225.56, stdev=210492.39, samples=1155
 iops : min= 2, max= 4312, avg=1821.18, stdev=822.24, samples=1155
 lat (usec) : 2=0.01%, 250=0.01%, 500=0.01%, 750=0.36%, 1000=1.58%
 lat (msec) : 2=34.04%, 4=45.13%, 10=17.43%, 20=1.27%, 50=0.10%
 lat (msec) : 100=0.02%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.04%
 cpu : usr=0.82%, sys=56.44%, ctx=7701664, majf=0, minf=21
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=1580114,1053170,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=31630: Thu Apr 16 22:02:18 2020
 read: IOPS=1755, BW=439MiB/s (460MB/s)(386GiB/900001msec)
 slat (usec): min=11, max=6001.3k, avg=236.29, stdev=12943.27
 clat (usec): min=418, max=9439.3k, avg=5315.54, stdev=111617.09
 lat (usec): min=435, max=9439.3k, avg=5551.96, stdev=112738.52
 clat percentiles (usec):
 | 1.00th=[ 906], 5.00th=[ 1237], 10.00th=[ 1434],
 | 20.00th=[ 1713], 30.00th=[ 1926], 40.00th=[ 2147],
 | 50.00th=[ 2409], 60.00th=[ 2737], 70.00th=[ 3228],
 | 80.00th=[ 3982], 90.00th=[ 5211], 95.00th=[ 6783],
 | 99.00th=[ 10945], 99.50th=[ 13173], 99.90th=[ 41157],
 | 99.95th=[1132463], 99.99th=[7214203]
 bw ( KiB/s): min= 512, max=1685504, per=38.81%, avg=697806.97, stdev=310838.43, samples=1158
 iops : min= 2, max= 6584, avg=2725.80, stdev=1214.22, samples=1158
 write: IOPS=1170, BW=293MiB/s (307MB/s)(257GiB/900001msec); 0 zone resets
 slat (usec): min=27, max=4547.6k, avg=283.11, stdev=11936.31
 clat (nsec): min=1353, max=9438.9M, avg=4841527.39, stdev=107041842.36
 lat (usec): min=278, max=9438.0k, avg=5124.77, stdev=107745.97
 clat percentiles (usec):
 | 1.00th=[ 889], 5.00th=[ 1221], 10.00th=[ 1401],
 | 20.00th=[ 1663], 30.00th=[ 1860], 40.00th=[ 2057],
 | 50.00th=[ 2278], 60.00th=[ 2573], 70.00th=[ 2999],
 | 80.00th=[ 3687], 90.00th=[ 4883], 95.00th=[ 6456],
 | 99.00th=[ 10552], 99.50th=[ 12780], 99.90th=[ 34341],
 | 99.95th=[ 170918], 99.99th=[7214203]
 bw ( KiB/s): min= 512, max=1175552, per=38.88%, avg=465971.38, stdev=207316.31, samples=1156
 iops : min= 2, max= 4592, avg=1820.19, stdev=809.83, samples=1156
 lat (usec) : 2=0.01%, 500=0.01%, 750=0.31%, 1000=1.47%
 lat (msec) : 2=33.17%, 4=46.48%, 10=17.13%, 20=1.25%, 50=0.11%
 lat (msec) : 100=0.02%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.04%
 cpu : usr=0.81%, sys=57.17%, ctx=7700112, majf=0, minf=24
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=1580380,1053470,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=31631: Thu Apr 16 22:02:18 2020
 read: IOPS=1755, BW=439MiB/s (460MB/s)(386GiB/900001msec)
 slat (usec): min=10, max=6001.6k, avg=231.26, stdev=12228.15
 clat (nsec): min=1082, max=9452.4M, avg=5205366.88, stdev=109985047.72
 lat (usec): min=36, max=9452.4k, avg=5436.74, stdev=111086.16
 clat percentiles (usec):
 | 1.00th=[ 914], 5.00th=[ 1237], 10.00th=[ 1434],
 | 20.00th=[ 1696], 30.00th=[ 1909], 40.00th=[ 2147],
 | 50.00th=[ 2376], 60.00th=[ 2737], 70.00th=[ 3261],
 | 80.00th=[ 4015], 90.00th=[ 5276], 95.00th=[ 6783],
 | 99.00th=[ 10945], 99.50th=[ 13042], 99.90th=[ 37487],
 | 99.95th=[ 884999], 99.99th=[7281312]
 bw ( KiB/s): min= 512, max=1386496, per=38.87%, avg=698891.65, stdev=310544.20, samples=1156
 iops : min= 2, max= 5416, avg=2730.03, stdev=1213.07, samples=1156
 write: IOPS=1171, BW=293MiB/s (307MB/s)(257GiB/900001msec); 0 zone resets
 slat (usec): min=29, max=4547.4k, avg=290.22, stdev=12847.83
 clat (usec): min=303, max=9451.9k, avg=5004.77, stdev=108932.38
 lat (usec): min=534, max=9452.1k, avg=5295.11, stdev=109724.17
 clat percentiles (usec):
 | 1.00th=[ 889], 5.00th=[ 1205], 10.00th=[ 1385],
 | 20.00th=[ 1647], 30.00th=[ 1844], 40.00th=[ 2057],
 | 50.00th=[ 2278], 60.00th=[ 2540], 70.00th=[ 2999],
 | 80.00th=[ 3687], 90.00th=[ 4948], 95.00th=[ 6456],
 | 99.00th=[ 10683], 99.50th=[ 12780], 99.90th=[ 36439],
 | 99.95th=[ 734004], 99.99th=[7147095]
 bw ( KiB/s): min= 512, max=912384, per=38.91%, avg=466288.45, stdev=207371.91, samples=1156
 iops : min= 2, max= 3564, avg=1821.43, stdev=810.05, samples=1156
 lat (usec) : 2=0.01%, 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%
 lat (usec) : 750=0.33%, 1000=1.42%
 lat (msec) : 2=33.79%, 4=45.70%, 10=17.30%, 20=1.27%, 50=0.10%
 lat (msec) : 100=0.02%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.04%
 cpu : usr=0.80%, sys=56.71%, ctx=7578493, majf=0, minf=24
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=1580092,1054191,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=31632: Thu Apr 16 22:02:18 2020
 read: IOPS=1755, BW=439MiB/s (460MB/s)(386GiB/900001msec)
 slat (usec): min=10, max=7776.5k, avg=238.59, stdev=12921.86
 clat (nsec): min=1242, max=9439.3M, avg=5241348.12, stdev=109996869.99
 lat (usec): min=74, max=9439.4k, avg=5480.06, stdev=111197.22
 clat percentiles (usec):
 | 1.00th=[ 898], 5.00th=[ 1237], 10.00th=[ 1434],
 | 20.00th=[ 1713], 30.00th=[ 1926], 40.00th=[ 2147],
 | 50.00th=[ 2409], 60.00th=[ 2769], 70.00th=[ 3326],
 | 80.00th=[ 4047], 90.00th=[ 5276], 95.00th=[ 6849],
 | 99.00th=[ 10945], 99.50th=[ 13173], 99.90th=[ 33162],
 | 99.95th=[ 884999], 99.99th=[7281312]
 bw ( KiB/s): min= 512, max=1449984, per=38.38%, avg=689968.99, stdev=307834.81, samples=1171
 iops : min= 2, max= 5664, avg=2695.18, stdev=1202.49, samples=1171
 write: IOPS=1169, BW=292MiB/s (306MB/s)(257GiB/900001msec); 0 zone resets
 slat (usec): min=29, max=4367.0k, avg=285.30, stdev=11879.82
 clat (usec): min=75, max=9415.3k, avg=4960.03, stdev=107558.96
 lat (usec): min=313, max=9415.4k, avg=5245.46, stdev=108213.90
 clat percentiles (usec):
 | 1.00th=[ 881], 5.00th=[ 1205], 10.00th=[ 1401],
 | 20.00th=[ 1663], 30.00th=[ 1876], 40.00th=[ 2073],
 | 50.00th=[ 2311], 60.00th=[ 2606], 70.00th=[ 3064],
 | 80.00th=[ 3752], 90.00th=[ 5014], 95.00th=[ 6587],
 | 99.00th=[ 10683], 99.50th=[ 12911], 99.90th=[ 32900],
 | 99.95th=[ 450888], 99.99th=[7147095]
 bw ( KiB/s): min= 512, max=935424, per=38.67%, avg=463388.66, stdev=201307.47, samples=1161
 iops : min= 2, max= 3654, avg=1810.10, stdev=786.36, samples=1161
 lat (usec) : 2=0.01%, 100=0.01%, 500=0.01%, 750=0.38%, 1000=1.43%
 lat (msec) : 2=32.60%, 4=46.20%, 10=17.92%, 20=1.29%, 50=0.10%
 lat (msec) : 100=0.02%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.04%
 cpu : usr=0.80%, sys=56.16%, ctx=7520294, majf=0, minf=25
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=1580181,1052150,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
 READ: bw=1756MiB/s (1841MB/s), 439MiB/s-439MiB/s (460MB/s-460MB/s), io=1543GiB (1657GB), run=900001-900001msec
 WRITE: bw=1170MiB/s (1227MB/s), 292MiB/s-293MiB/s (306MB/s-307MB/s), io=1029GiB (1104GB), run=900001-900001msec

Disk stats (read/write):
 dm-4: ios=3563816/5044708, merge=0/0, ticks=13307671/18446744071674149170, in_queue=18446744071687456841, util=70.15%, aggrios=3565147/5185081, aggrmerge=0/0, aggrticks=13426178/18446744071890657751, aggrin_queue=18446744071904087582, aggrutil=73.91%
 dm-0: ios=3565147/5185081, merge=0/0, ticks=13426178/18446744071890657751, in_queue=18446744071904087582, util=73.91%, aggrios=4097296/5789839, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
 md128: ios=4097296/5789839, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=1864869/331459, aggrmerge=135781/2563833, aggrticks=4842566/833110, aggrin_queue=4628710, aggrutil=65.40%
 sdd: ios=1933063/340559, merge=81504/2554693, ticks=2903166/1168077, in_queue=3010995, util=64.10%
 sda: ios=1796675/322360, merge=190058/2572974, ticks=6781967/498143, in_queue=6246426, util=65.40%

machine 2:

storage: WDC WDS500G2B0A-00SM50 (ssd)

storage layout:
cryptsetup luks with camellia-xts-plain64 + lvm

raw linear read io performance with hdparm -Tt (to understand bus limitations):

/dev/sda:

Timing cached reads: 5738 MB in 2.00 seconds = 2870.00 MB/sec
 Timing buffered disk reads: 226 MB in 3.02 seconds = 74.74 MB/sec

for some reason very low speed here, probably because of external sata3 card
02:00.0 SATA controller: ASMedia Technology Inc. Device 0622 (rev 01)
smartctl show 6gb link speed here:
SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
anyway, let’s continue…

crypto performance:

[h1]Tests are approximate using memory only (no storage IO).[/h1]

[h1]Algorithm | Key | Encryption | Decryption[/h1]

twofish-xts 512b 181.5 MiB/s 186.3 MiB/s

[h1]Tests are approximate using memory only (no storage IO).[/h1]

[h1]Algorithm | Key | Encryption | Decryption[/h1]

serpent-xts 512b 170.6 MiB/s 179.2 MiB/s

[h1]Tests are approximate using memory only (no storage IO).[/h1]

[h1]Algorithm | Key | Encryption | Decryption[/h1]

aes-xts 512b 122.8 MiB/s 125.9 MiB/s

[h1]Tests are approximate using memory only (no storage IO).[/h1]

[h1]Algorithm | Key | Encryption | Decryption[/h1]

camellia-xts 512b 167.3 MiB/s 173.4 MiB/s

[h1]Tests are approximate using memory only (no storage IO).[/h1]

PBKDF2-ripemd160 129007 iterations per second for 512-bit key

[h1]Tests are approximate using memory only (no storage IO).[/h1]

PBKDF2-sha256 220659 iterations per second for 512-bit key

[h1]Tests are approximate using memory only (no storage IO).[/h1]

PBKDF2-sha512 332670 iterations per second for 512-bit key

[h1]Tests are approximate using memory only (no storage IO).[/h1]

PBKDF2-whirlpool 199197 iterations per second for 512-bit key

obvously camellia is not the best choice here for crypto, probably i should switch to twofish or serpent in future

xfs filesystem tests:

fio-rand-RW.fio:

file1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.18
Starting 4 processes
file1: Laying out IO file (1 file / 10240MiB)
Jobs: 4 (f=4): [m(4)][100.0%][r=32.4MiB/s,w=22.3MiB/s][r=8306,w=5708 IOPS][eta 00m:00s]
file1: (groupid=0, jobs=1): err= 0: pid=621474: Wed Apr 15 02:06:48 2020
 read: IOPS=1935, BW=7741KiB/s (7926kB/s)(6803MiB/900001msec)
 slat (nsec): min=1891, max=5893.4M, avg=300566.15, stdev=10037311.50
 clat (usec): min=66, max=6080.1k, avg=4678.04, stdev=40385.49
 lat (usec): min=69, max=6080.1k, avg=4979.19, stdev=42253.34
 clat percentiles (usec):
 | 1.00th=[ 685], 5.00th=[ 1352], 10.00th=[ 1778],
 | 20.00th=[ 2278], 30.00th=[ 2638], 40.00th=[ 2999],
 | 50.00th=[ 3392], 60.00th=[ 3851], 70.00th=[ 4359],
 | 80.00th=[ 5145], 90.00th=[ 6521], 95.00th=[ 8455],
 | 99.00th=[ 22676], 99.50th=[ 30016], 99.90th=[ 49021],
 | 99.95th=[ 64226], 99.99th=[2734687]
 bw ( KiB/s): min= 7, max=15936, per=27.09%, avg=8355.01, stdev=3468.68, samples=1666
 iops : min= 1, max= 3984, avg=2088.62, stdev=867.19, samples=1666
 write: IOPS=1289, BW=5159KiB/s (5283kB/s)(4534MiB/900001msec); 0 zone resets
 slat (usec): min=2, max=3014.6k, avg=312.79, stdev=6120.86
 clat (usec): min=8, max=6053.9k, avg=4614.70, stdev=41890.22
 lat (usec): min=70, max=6080.6k, avg=4928.04, stdev=42745.15
 clat percentiles (usec):
 | 1.00th=[ 660], 5.00th=[ 1303], 10.00th=[ 1713],
 | 20.00th=[ 2212], 30.00th=[ 2573], 40.00th=[ 2966],
 | 50.00th=[ 3326], 60.00th=[ 3785], 70.00th=[ 4293],
 | 80.00th=[ 5080], 90.00th=[ 6390], 95.00th=[ 8356],
 | 99.00th=[ 21627], 99.50th=[ 29230], 99.90th=[ 47973],
 | 99.95th=[ 63701], 99.99th=[2902459]
 bw ( KiB/s): min= 7, max=10672, per=27.23%, avg=5598.61, stdev=2288.35, samples=1657
 iops : min= 1, max= 2668, avg=1399.51, stdev=572.10, samples=1657
 lat (usec) : 10=0.01%, 100=0.02%, 250=0.06%, 500=0.38%, 750=0.85%
 lat (usec) : 1000=1.21%
 lat (msec) : 2=12.05%, 4=49.09%, 10=32.74%, 20=2.38%, 50=1.13%
 lat (msec) : 100=0.06%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.01%
 cpu : usr=1.59%, sys=8.53%, ctx=2707503, majf=2, minf=20
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=1741637,1160773,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=621475: Wed Apr 15 02:06:48 2020
 read: IOPS=1922, BW=7690KiB/s (7874kB/s)(6759MiB/900001msec)
 slat (nsec): min=1916, max=5612.2M, avg=294667.69, stdev=8843280.94
 clat (usec): min=5, max=6000.1k, avg=4705.97, stdev=40069.66
 lat (usec): min=69, max=6000.1k, avg=5001.22, stdev=41468.73
 clat percentiles (usec):
 | 1.00th=[ 693], 5.00th=[ 1385], 10.00th=[ 1795],
 | 20.00th=[ 2278], 30.00th=[ 2671], 40.00th=[ 3032],
 | 50.00th=[ 3425], 60.00th=[ 3851], 70.00th=[ 4424],
 | 80.00th=[ 5145], 90.00th=[ 6587], 95.00th=[ 8586],
 | 99.00th=[ 22676], 99.50th=[ 30016], 99.90th=[ 49021],
 | 99.95th=[ 65274], 99.99th=[2835350]
 bw ( KiB/s): min= 7, max=14352, per=26.88%, avg=8290.15, stdev=3431.62, samples=1668
 iops : min= 1, max= 3588, avg=2072.39, stdev=857.92, samples=1668
 write: IOPS=1281, BW=5127KiB/s (5250kB/s)(4506MiB/900001msec); 0 zone resets
 slat (usec): min=2, max=4509.8k, avg=327.40, stdev=8306.99
 clat (usec): min=66, max=6000.1k, avg=4647.82, stdev=42006.68
 lat (usec): min=69, max=6000.1k, avg=4975.78, stdev=43569.84
 clat percentiles (usec):
 | 1.00th=[ 668], 5.00th=[ 1336], 10.00th=[ 1729],
 | 20.00th=[ 2212], 30.00th=[ 2606], 40.00th=[ 2966],
 | 50.00th=[ 3359], 60.00th=[ 3818], 70.00th=[ 4359],
 | 80.00th=[ 5145], 90.00th=[ 6521], 95.00th=[ 8586],
 | 99.00th=[ 21890], 99.50th=[ 29230], 99.90th=[ 47973],
 | 99.95th=[ 63177], 99.99th=[2701132]
 bw ( KiB/s): min= 7, max= 9776, per=27.05%, avg=5560.95, stdev=2246.97, samples=1658
 iops : min= 1, max= 2444, avg=1390.10, stdev=561.75, samples=1658
 lat (usec) : 10=0.01%, 100=0.02%, 250=0.07%, 500=0.36%, 750=0.80%
 lat (usec) : 1000=1.17%
 lat (msec) : 2=11.68%, 4=49.07%, 10=33.14%, 20=2.45%, 50=1.14%
 lat (msec) : 100=0.06%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.01%
 cpu : usr=1.59%, sys=8.40%, ctx=2711432, majf=1, minf=23
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=1730177,1153643,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=621476: Wed Apr 15 02:06:48 2020
 read: IOPS=1925, BW=7701KiB/s (7886kB/s)(6769MiB/900001msec)
 slat (nsec): min=1835, max=3014.6M, avg=293882.01, stdev=7975456.23
 clat (usec): min=16, max=6081.5k, avg=4701.02, stdev=40545.51
 lat (usec): min=65, max=6081.5k, avg=4995.47, stdev=41788.33
 clat percentiles (usec):
 | 1.00th=[ 676], 5.00th=[ 1385], 10.00th=[ 1795],
 | 20.00th=[ 2278], 30.00th=[ 2671], 40.00th=[ 3032],
 | 50.00th=[ 3425], 60.00th=[ 3851], 70.00th=[ 4424],
 | 80.00th=[ 5145], 90.00th=[ 6521], 95.00th=[ 8586],
 | 99.00th=[ 22676], 99.50th=[ 30278], 99.90th=[ 50070],
 | 99.95th=[ 68682], 99.99th=[2868904]
 bw ( KiB/s): min= 7, max=14626, per=26.97%, avg=8317.50, stdev=3430.16, samples=1665
 iops : min= 1, max= 3656, avg=2079.23, stdev=857.57, samples=1665
 write: IOPS=1284, BW=5137KiB/s (5260kB/s)(4515MiB/900001msec); 0 zone resets
 slat (usec): min=2, max=5770.0k, avg=327.47, stdev=9706.72
 clat (usec): min=62, max=6081.3k, avg=4635.87, stdev=41925.59
 lat (usec): min=66, max=6081.6k, avg=4963.87, stdev=43686.92
 clat percentiles (usec):
 | 1.00th=[ 644], 5.00th=[ 1319], 10.00th=[ 1729],
 | 20.00th=[ 2212], 30.00th=[ 2606], 40.00th=[ 2966],
 | 50.00th=[ 3359], 60.00th=[ 3785], 70.00th=[ 4359],
 | 80.00th=[ 5080], 90.00th=[ 6456], 95.00th=[ 8455],
 | 99.00th=[ 21890], 99.50th=[ 29230], 99.90th=[ 47449],
 | 99.95th=[ 63177], 99.99th=[2835350]
 bw ( KiB/s): min= 8, max= 9720, per=27.12%, avg=5574.94, stdev=2265.45, samples=1657
 iops : min= 2, max= 2430, avg=1393.59, stdev=566.38, samples=1657
 lat (usec) : 20=0.01%, 100=0.02%, 250=0.07%, 500=0.38%, 750=0.86%
 lat (usec) : 1000=1.17%
 lat (msec) : 2=11.63%, 4=49.28%, 10=32.89%, 20=2.47%, 50=1.14%
 lat (msec) : 100=0.06%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.01%
 cpu : usr=1.60%, sys=8.41%, ctx=2726842, majf=0, minf=24
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=1732741,1155858,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=621477: Wed Apr 15 02:06:48 2020
 read: IOPS=1928, BW=7714KiB/s (7900kB/s)(6780MiB/900001msec)
 slat (nsec): min=1882, max=4509.8M, avg=295620.68, stdev=8875316.44
 clat (usec): min=2, max=6079.9k, avg=4726.13, stdev=43107.65
 lat (usec): min=65, max=6079.9k, avg=5022.33, stdev=44581.37
 clat percentiles (usec):
 | 1.00th=[ 693], 5.00th=[ 1385], 10.00th=[ 1795],
 | 20.00th=[ 2278], 30.00th=[ 2671], 40.00th=[ 3032],
 | 50.00th=[ 3425], 60.00th=[ 3851], 70.00th=[ 4424],
 | 80.00th=[ 5145], 90.00th=[ 6521], 95.00th=[ 8586],
 | 99.00th=[ 22414], 99.50th=[ 29754], 99.90th=[ 47973],
 | 99.95th=[ 64226], 99.99th=[2969568]
 bw ( KiB/s): min= 7, max=14872, per=27.04%, avg=8341.89, stdev=3408.53, samples=1663
 iops : min= 1, max= 3718, avg=2085.33, stdev=852.15, samples=1663
 write: IOPS=1283, BW=5136KiB/s (5259kB/s)(4514MiB/900001msec); 0 zone resets
 slat (usec): min=2, max=5611.0k, avg=324.11, stdev=8194.12
 clat (usec): min=62, max=6063.8k, avg=4587.10, stdev=38392.47
 lat (usec): min=66, max=6080.2k, avg=4911.76, stdev=39836.92
 clat percentiles (usec):
 | 1.00th=[ 668], 5.00th=[ 1336], 10.00th=[ 1729],
 | 20.00th=[ 2212], 30.00th=[ 2606], 40.00th=[ 2966],
 | 50.00th=[ 3359], 60.00th=[ 3785], 70.00th=[ 4359],
 | 80.00th=[ 5080], 90.00th=[ 6456], 95.00th=[ 8455],
 | 99.00th=[ 21890], 99.50th=[ 29230], 99.90th=[ 46400],
 | 99.95th=[ 61080], 99.99th=[1870660]
 bw ( KiB/s): min= 7, max= 9768, per=27.08%, avg=5566.76, stdev=2257.52, samples=1659
 iops : min= 1, max= 2442, avg=1391.55, stdev=564.40, samples=1659
 lat (usec) : 4=0.01%, 100=0.02%, 250=0.06%, 500=0.36%, 750=0.82%
 lat (usec) : 1000=1.18%
 lat (msec) : 2=11.73%, 4=49.18%, 10=32.94%, 20=2.47%, 50=1.15%
 lat (msec) : 100=0.05%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.01%
 cpu : usr=1.53%, sys=8.46%, ctx=2720607, majf=0, minf=24
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=1735751,1155553,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
 READ: bw=30.1MiB/s (31.6MB/s), 7690KiB/s-7741KiB/s (7874kB/s-7926kB/s), io=26.5GiB (28.4GB), run=900001-900001msec
 WRITE: bw=20.1MiB/s (21.1MB/s), 5127KiB/s-5159KiB/s (5250kB/s-5283kB/s), io=17.6GiB (18.9GB), run=900001-900001msec

Disk stats (read/write):
 dm-1: ios=3685533/4166016, merge=0/0, ticks=1286849/119447523, in_queue=120737260, util=99.40%, aggrios=3685677/4166016, aggrmerge=0/0, aggrticks=1277080/119441923, aggrin_queue=120721887, aggrutil=99.40%
 dm-0: ios=3685677/4166016, merge=0/0, ticks=1277080/119441923, in_queue=120721887, util=99.40%, aggrios=3685677/4160531, aggrmerge=0/5487, aggrticks=797504/1490474, aggrin_queue=73134, aggrutil=99.38%
 sda: ios=3685677/4160531, merge=0/5487, ticks=797504/1490474, in_queue=73134, util=99.38%

machine 3:

storage:

WDC WDS500G2B0A-00SM50 (ssd)
2 x WDC WD10EZRX-00L4HB0

storage layout:

ssd: cryptsetup luks with aes-xts-plain64 + lvm
hdd array: 2x hdd mdadm raid0 + cryptsetup luks with camellia-xts-plain64 + lvm

raw linear read io performance with hdparm -Tt (to understand bus limitations):

ssd:

 Timing cached reads: 8664 MB in 2.00 seconds = 4335.58 MB/sec
 Timing buffered disk reads: 1248 MB in 3.00 seconds = 415.96 MB/sec

hdd array:

 Timing cached reads: 8692 MB in 2.00 seconds = 4349.16 MB/sec
 Timing buffered disk reads: 892 MB in 3.00 seconds = 297.28 MB/sec

crypto performance:

[h1]Tests are approximate using memory only (no storage IO).[/h1]

[h1]Algorithm | Key | Encryption | Decryption[/h1]

twofish-xts 512b 219.6 MiB/s 225.9 MiB/s

[h1]Tests are approximate using memory only (no storage IO).[/h1]

[h1]Algorithm | Key | Encryption | Decryption[/h1]

serpent-xts 512b 226.4 MiB/s 233.5 MiB/s

[h1]Tests are approximate using memory only (no storage IO).[/h1]

[h1]Algorithm | Key | Encryption | Decryption[/h1]

aes-xts 512b 1198.2 MiB/s 1102.4 MiB/s

[h1]Tests are approximate using memory only (no storage IO).[/h1]

[h1]Algorithm | Key | Encryption | Decryption[/h1]

camellia-xts 512b 259.6 MiB/s 267.7 MiB/s

[h1]Tests are approximate using memory only (no storage IO).[/h1]

PBKDF2-ripemd160 168473 iterations per second for 512-bit key

[h1]Tests are approximate using memory only (no storage IO).[/h1]

PBKDF2-sha256 230355 iterations per second for 512-bit key

[h1]Tests are approximate using memory only (no storage IO).[/h1]

PBKDF2-sha512 392431 iterations per second for 512-bit key

[h1]Tests are approximate using memory only (no storage IO).[/h1]

PBKDF2-whirlpool 215934 iterations per second for 512-bit key

xfs filesystem tests:

fio-rand-RW.fio:

ssd:

file1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.18
Starting 4 processes
file1: Laying out IO file (1 file / 10240MiB)
Jobs: 4 (f=4): [m(4)][100.0%][eta 00m:00s] 
file1: (groupid=0, jobs=1): err= 0: pid=115788: Wed Apr 15 02:42:20 2020
 read: IOPS=4398, BW=17.2MiB/s (18.0MB/s)(15.2GiB/904453msec)
 slat (nsec): min=1713, max=19778M, avg=106136.70, stdev=11322677.73
 clat (usec): min=52, max=27685k, avg=2068.36, stdev=71398.30
 lat (usec): min=54, max=27685k, avg=2174.71, stdev=72597.41
 clat percentiles (usec):
 | 1.00th=[ 83], 5.00th=[ 100], 10.00th=[ 110], 20.00th=[ 165],
 | 30.00th=[ 469], 40.00th=[ 775], 50.00th=[ 1090], 60.00th=[ 1467],
 | 70.00th=[ 1909], 80.00th=[ 2474], 90.00th=[ 3392], 95.00th=[ 4359],
 | 99.00th=[ 7570], 99.50th=[ 10683], 99.90th=[ 29230], 99.95th=[ 41157],
 | 99.99th=[187696]
 bw ( KiB/s): min= 8, max=340064, per=31.70%, avg=22272.22, stdev=33531.22, samples=1428
 iops : min= 2, max=85016, avg=5567.98, stdev=8382.81, samples=1428
 write: IOPS=2932, BW=11.5MiB/s (12.0MB/s)(10.1GiB/904453msec); 0 zone resets
 slat (usec): min=2, max=12792k, avg=169.85, stdev=19676.64
 clat (usec): min=4, max=27685k, avg=2015.24, stdev=70796.95
 lat (usec): min=55, max=27685k, avg=2185.32, stdev=74041.06
 clat percentiles (usec):
 | 1.00th=[ 82], 5.00th=[ 99], 10.00th=[ 110], 20.00th=[ 159],
 | 30.00th=[ 449], 40.00th=[ 734], 50.00th=[ 1045], 60.00th=[ 1401],
 | 70.00th=[ 1844], 80.00th=[ 2409], 90.00th=[ 3326], 95.00th=[ 4293],
 | 99.00th=[ 7439], 99.50th=[ 10290], 99.90th=[ 27657], 99.95th=[ 39584],
 | 99.99th=[160433]
 bw ( KiB/s): min= 8, max=228088, per=31.78%, avg=14883.21, stdev=22374.69, samples=1425
 iops : min= 2, max=57022, avg=3720.72, stdev=5593.66, samples=1425
 lat (usec) : 10=0.01%, 100=5.29%, 250=18.63%, 500=7.73%, 750=8.09%
 lat (usec) : 1000=7.88%
 lat (msec) : 2=24.60%, 4=21.55%, 10=5.69%, 20=0.34%, 50=0.17%
 lat (msec) : 100=0.02%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.01%
 cpu : usr=2.18%, sys=13.19%, ctx=3611418, majf=0, minf=23
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=3977929,2652645,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=115789: Wed Apr 15 02:42:20 2020
 read: IOPS=4387, BW=17.1MiB/s (17.0MB/s)(15.1GiB/904453msec)
 slat (nsec): min=1492, max=9319.9M, avg=114899.83, stdev=10620345.06
 clat (usec): min=9, max=27835k, avg=2066.00, stdev=71693.91
 lat (usec): min=56, max=27835k, avg=2181.12, stdev=72886.66
 clat percentiles (usec):
 | 1.00th=[ 83], 5.00th=[ 100], 10.00th=[ 110], 20.00th=[ 163],
 | 30.00th=[ 465], 40.00th=[ 766], 50.00th=[ 1090], 60.00th=[ 1467],
 | 70.00th=[ 1926], 80.00th=[ 2507], 90.00th=[ 3425], 95.00th=[ 4359],
 | 99.00th=[ 7570], 99.50th=[ 10683], 99.90th=[ 29230], 99.95th=[ 40109],
 | 99.99th=[147850]
 bw ( KiB/s): min= 8, max=347336, per=31.64%, avg=22231.52, stdev=33886.02, samples=1427
 iops : min= 2, max=86834, avg=5557.80, stdev=8471.50, samples=1427
 write: IOPS=2925, BW=11.4MiB/s (11.0MB/s)(10.1GiB/904453msec); 0 zone resets
 slat (usec): min=2, max=19745k, avg=157.48, stdev=20211.37
 clat (usec): min=54, max=27835k, avg=2031.82, stdev=73292.55
 lat (usec): min=57, max=27835k, avg=2189.53, stdev=76492.99
 clat percentiles (usec):
 | 1.00th=[ 83], 5.00th=[ 99], 10.00th=[ 109], 20.00th=[ 159],
 | 30.00th=[ 441], 40.00th=[ 734], 50.00th=[ 1045], 60.00th=[ 1418],
 | 70.00th=[ 1860], 80.00th=[ 2442], 90.00th=[ 3359], 95.00th=[ 4293],
 | 99.00th=[ 7439], 99.50th=[ 10290], 99.90th=[ 28181], 99.95th=[ 39060],
 | 99.99th=[137364]
 bw ( KiB/s): min= 8, max=231312, per=31.85%, avg=14918.22, stdev=22628.46, samples=1418
 iops : min= 2, max=57828, avg=3729.47, stdev=5657.11, samples=1418
 lat (usec) : 10=0.01%, 100=5.33%, 250=18.74%, 500=7.82%, 750=8.01%
 lat (usec) : 1000=7.79%
 lat (msec) : 2=24.35%, 4=21.61%, 10=5.80%, 20=0.35%, 50=0.17%
 lat (msec) : 100=0.02%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.01%
 cpu : usr=2.16%, sys=13.21%, ctx=3592523, majf=0, minf=23
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=3967874,2645804,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=115790: Wed Apr 15 02:42:20 2020
 read: IOPS=4393, BW=17.2MiB/s (17.0MB/s)(15.2GiB/904453msec)
 slat (nsec): min=1681, max=12792M, avg=116265.18, stdev=13332344.36
 clat (usec): min=53, max=28423k, avg=2089.26, stdev=73339.30
 lat (usec): min=56, max=28423k, avg=2205.74, stdev=74861.83
 clat percentiles (usec):
 | 1.00th=[ 87], 5.00th=[ 100], 10.00th=[ 110], 20.00th=[ 172],
 | 30.00th=[ 469], 40.00th=[ 775], 50.00th=[ 1090], 60.00th=[ 1467],
 | 70.00th=[ 1909], 80.00th=[ 2474], 90.00th=[ 3392], 95.00th=[ 4359],
 | 99.00th=[ 7570], 99.50th=[ 10552], 99.90th=[ 28967], 99.95th=[ 39584],
 | 99.99th=[198181]
 bw ( KiB/s): min= 8, max=342864, per=31.75%, avg=22311.96, stdev=33541.39, samples=1424
 iops : min= 2, max=85716, avg=5577.91, stdev=8385.34, samples=1424
 write: IOPS=2928, BW=11.4MiB/s (11.0MB/s)(10.1GiB/904453msec); 0 zone resets
 slat (usec): min=2, max=19745k, avg=154.84, stdev=17365.04
 clat (usec): min=10, max=28423k, avg=1990.12, stdev=69995.95
 lat (usec): min=56, max=28423k, avg=2145.20, stdev=72725.24
 clat percentiles (usec):
 | 1.00th=[ 86], 5.00th=[ 100], 10.00th=[ 110], 20.00th=[ 163],
 | 30.00th=[ 445], 40.00th=[ 734], 50.00th=[ 1045], 60.00th=[ 1401],
 | 70.00th=[ 1844], 80.00th=[ 2409], 90.00th=[ 3326], 95.00th=[ 4293],
 | 99.00th=[ 7439], 99.50th=[ 10290], 99.90th=[ 28181], 99.95th=[ 39060],
 | 99.99th=[141558]
 bw ( KiB/s): min= 8, max=229368, per=31.84%, avg=14913.15, stdev=22414.86, samples=1420
 iops : min= 2, max=57342, avg=3728.20, stdev=5603.70, samples=1420
 lat (usec) : 20=0.01%, 100=5.01%, 250=18.72%, 500=7.96%, 750=8.05%
 lat (usec) : 1000=7.88%
 lat (msec) : 2=24.63%, 4=21.49%, 10=5.71%, 20=0.35%, 50=0.17%
 lat (msec) : 100=0.02%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.01%
 cpu : usr=2.15%, sys=13.25%, ctx=3649760, majf=0, minf=23
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=3973815,2648617,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=115791: Wed Apr 15 02:42:20 2020
 read: IOPS=4387, BW=17.1MiB/s (17.0MB/s)(15.1GiB/904454msec)
 slat (nsec): min=1743, max=19778M, avg=121148.63, stdev=16032951.75
 clat (usec): min=53, max=27687k, avg=2051.97, stdev=66316.94
 lat (usec): min=56, max=27687k, avg=2173.34, stdev=68621.44
 clat percentiles (usec):
 | 1.00th=[ 80], 5.00th=[ 99], 10.00th=[ 109], 20.00th=[ 165],
 | 30.00th=[ 469], 40.00th=[ 775], 50.00th=[ 1106], 60.00th=[ 1483],
 | 70.00th=[ 1926], 80.00th=[ 2507], 90.00th=[ 3425], 95.00th=[ 4359],
 | 99.00th=[ 7570], 99.50th=[ 10683], 99.90th=[ 28443], 99.95th=[ 39584],
 | 99.99th=[198181]
 bw ( KiB/s): min= 8, max=349384, per=31.64%, avg=22234.08, stdev=33880.89, samples=1427
 iops : min= 2, max=87346, avg=5558.44, stdev=8470.21, samples=1427
 write: IOPS=2921, BW=11.4MiB/s (11.0MB/s)(10.1GiB/904454msec); 0 zone resets
 slat (usec): min=2, max=12792k, avg=148.36, stdev=13818.41
 clat (usec): min=10, max=27686k, avg=2055.01, stdev=77333.17
 lat (usec): min=56, max=27686k, avg=2203.60, stdev=78954.75
 clat percentiles (usec):
 | 1.00th=[ 79], 5.00th=[ 99], 10.00th=[ 109], 20.00th=[ 159],
 | 30.00th=[ 449], 40.00th=[ 742], 50.00th=[ 1057], 60.00th=[ 1418],
 | 70.00th=[ 1860], 80.00th=[ 2442], 90.00th=[ 3359], 95.00th=[ 4293],
 | 99.00th=[ 7439], 99.50th=[ 10290], 99.90th=[ 27919], 99.95th=[ 39060],
 | 99.99th=[198181]
 bw ( KiB/s): min= 8, max=232600, per=31.68%, avg=14837.29, stdev=22596.83, samples=1424
 iops : min= 2, max=58150, avg=3709.24, stdev=5649.20, samples=1424
 lat (usec) : 20=0.01%, 100=5.68%, 250=18.35%, 500=7.66%, 750=7.96%
 lat (usec) : 1000=7.82%
 lat (msec) : 2=24.51%, 4=21.71%, 10=5.78%, 20=0.35%, 50=0.16%
 lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.01%
 cpu : usr=2.18%, sys=13.17%, ctx=3600493, majf=0, minf=23
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=3968340,2642604,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
 READ: bw=68.6MiB/s (71.0MB/s), 17.1MiB/s-17.2MiB/s (17.0MB/s-18.0MB/s), io=60.6GiB (65.1GB), run=904453-904454msec
 WRITE: bw=45.7MiB/s (47.0MB/s), 11.4MiB/s-11.5MiB/s (11.0MB/s-12.0MB/s), io=40.4GiB (43.4GB), run=904453-904454msec

Disk stats (read/write):
 dm-2: ios=3315692/8009253, merge=0/0, ticks=1040390/18446744071668371254, in_queue=18446744071669412014, util=98.70%, aggrios=3315692/8009253, aggrmerge=0/0, aggrticks=1036954/18446744071689086446, aggrin_queue=18446744071690123769, aggrutil=98.70%
 dm-1: ios=3315692/8009253, merge=0/0, ticks=1036954/18446744071689086446, in_queue=18446744071690123769, util=98.70%, aggrios=3315692/7323541, aggrmerge=0/447753, aggrticks=464244/1102433, aggrin_queue=46733, aggrutil=98.65%
 sdc: ios=3315692/7323541, merge=0/447753, ticks=464244/1102433, in_queue=46733, util=98.65%

hdd array:

file1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.18
Starting 4 processes
file1: Laying out IO file (1 file / 10240MiB)
Jobs: 3 (f=3): [m(1),_(1),m(2)][50.1%][r=47KiB/s,w=15KiB/s][r=11,w=3 IOPS][eta 14m:59s]
file1: (groupid=0, jobs=1): err= 0: pid=116099: Wed Apr 15 03:01:43 2020
 read: IOPS=35, BW=142KiB/s (145kB/s)(125MiB/901171msec)
 slat (usec): min=4, max=2393.1k, avg=19205.65, stdev=52749.56
 clat (msec): min=68, max=3719, avg=255.72, stdev=245.40
 lat (msec): min=78, max=3735, avg=274.93, stdev=255.14
 clat percentiles (msec):
 | 1.00th=[ 131], 5.00th=[ 159], 10.00th=[ 174], 20.00th=[ 192],
 | 30.00th=[ 205], 40.00th=[ 218], 50.00th=[ 230], 60.00th=[ 243],
 | 70.00th=[ 255], 80.00th=[ 271], 90.00th=[ 300], 95.00th=[ 326],
 | 99.00th=[ 651], 99.50th=[ 2836], 99.90th=[ 3205], 99.95th=[ 3473],
 | 99.99th=[ 3708]
 bw ( KiB/s): min= 7, max= 247, per=26.52%, avg=149.84, stdev=39.31, samples=1694
 iops : min= 1, max= 61, avg=36.78, stdev= 9.82, samples=1694
 write: IOPS=23, BW=94.3KiB/s (96.5kB/s)(82.0MiB/901171msec); 0 zone resets
 slat (usec): min=3, max=2581.7k, avg=13522.18, stdev=44734.86
 clat (usec): min=11, max=3695.6k, avg=251779.96, stdev=245770.99
 lat (msec): min=71, max=3708, avg=265.30, stdev=252.69
 clat percentiles (msec):
 | 1.00th=[ 128], 5.00th=[ 155], 10.00th=[ 169], 20.00th=[ 188],
 | 30.00th=[ 201], 40.00th=[ 213], 50.00th=[ 226], 60.00th=[ 239],
 | 70.00th=[ 251], 80.00th=[ 268], 90.00th=[ 296], 95.00th=[ 326],
 | 99.00th=[ 651], 99.50th=[ 2836], 99.90th=[ 3239], 99.95th=[ 3540],
 | 99.99th=[ 3641]
 bw ( KiB/s): min= 7, max= 215, per=26.32%, avg=99.49, stdev=33.86, samples=1692
 iops : min= 1, max= 53, avg=24.20, stdev= 8.47, samples=1692
 lat (usec) : 20=0.01%
 lat (msec) : 100=0.12%, 250=67.49%, 500=30.99%, 750=0.45%, 1000=0.03%
 lat (msec) : 2000=0.18%, >=2000=0.72%
 cpu : usr=0.09%, sys=0.87%, ctx=68032, majf=0, minf=18
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=31933,21241,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=116100: Wed Apr 15 03:01:43 2020
 read: IOPS=35, BW=141KiB/s (145kB/s)(124MiB/900753msec)
 slat (usec): min=3, max=2189.7k, avg=19250.42, stdev=50318.66
 clat (usec): min=12, max=3576.5k, avg=256838.86, stdev=250357.23
 lat (msec): min=76, max=3637, avg=276.09, stdev=259.82
 clat percentiles (msec):
 | 1.00th=[ 132], 5.00th=[ 159], 10.00th=[ 174], 20.00th=[ 192],
 | 30.00th=[ 205], 40.00th=[ 218], 50.00th=[ 228], 60.00th=[ 241],
 | 70.00th=[ 255], 80.00th=[ 271], 90.00th=[ 300], 95.00th=[ 330],
 | 99.00th=[ 793], 99.50th=[ 2869], 99.90th=[ 3205], 99.95th=[ 3473],
 | 99.99th=[ 3540]
 bw ( KiB/s): min= 7, max= 255, per=26.30%, avg=148.62, stdev=39.97, samples=1700
 iops : min= 1, max= 63, avg=36.48, stdev= 9.99, samples=1700
 write: IOPS=23, BW=94.3KiB/s (96.5kB/s)(82.9MiB/900753msec); 0 zone resets
 slat (usec): min=3, max=2186.3k, avg=13572.91, stdev=44869.81
 clat (msec): min=71, max=3604, avg=251.70, stdev=233.77
 lat (msec): min=78, max=3604, avg=265.28, stdev=241.19
 clat percentiles (msec):
 | 1.00th=[ 129], 5.00th=[ 157], 10.00th=[ 171], 20.00th=[ 190],
 | 30.00th=[ 203], 40.00th=[ 215], 50.00th=[ 226], 60.00th=[ 239],
 | 70.00th=[ 253], 80.00th=[ 271], 90.00th=[ 296], 95.00th=[ 326],
 | 99.00th=[ 625], 99.50th=[ 2735], 99.90th=[ 3171], 99.95th=[ 3440],
 | 99.99th=[ 3540]
 bw ( KiB/s): min= 7, max= 215, per=26.22%, avg=99.10, stdev=33.88, samples=1698
 iops : min= 1, max= 53, avg=24.10, stdev= 8.47, samples=1698
 lat (usec) : 20=0.01%
 lat (msec) : 100=0.08%, 250=67.58%, 500=30.91%, 750=0.48%, 1000=0.04%
 lat (msec) : 2000=0.16%, >=2000=0.76%
 cpu : usr=0.09%, sys=0.90%, ctx=67957, majf=0, minf=20
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=31788,21231,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=116101: Wed Apr 15 03:01:43 2020
 read: IOPS=35, BW=142KiB/s (145kB/s)(125MiB/901171msec)
 slat (usec): min=4, max=2093.4k, avg=18729.42, stdev=40433.47
 clat (msec): min=70, max=3583, avg=255.23, stdev=242.69
 lat (msec): min=85, max=3596, avg=273.96, stdev=250.06
 clat percentiles (msec):
 | 1.00th=[ 130], 5.00th=[ 157], 10.00th=[ 171], 20.00th=[ 190],
 | 30.00th=[ 205], 40.00th=[ 218], 50.00th=[ 230], 60.00th=[ 243],
 | 70.00th=[ 255], 80.00th=[ 271], 90.00th=[ 300], 95.00th=[ 330],
 | 99.00th=[ 642], 99.50th=[ 2802], 99.90th=[ 3239], 99.95th=[ 3473],
 | 99.99th=[ 3540]
 bw ( KiB/s): min= 7, max= 271, per=26.38%, avg=149.04, stdev=40.51, samples=1702
 iops : min= 1, max= 67, avg=36.58, stdev=10.12, samples=1702
 write: IOPS=23, BW=94.3KiB/s (96.6kB/s)(83.0MiB/901171msec); 0 zone resets
 slat (usec): min=4, max=2162.2k, avg=14248.26, stdev=57466.27
 clat (usec): min=11, max=3581.3k, avg=252656.28, stdev=247111.54
 lat (msec): min=82, max=3602, avg=266.91, stdev=257.81
 clat percentiles (msec):
 | 1.00th=[ 127], 5.00th=[ 153], 10.00th=[ 169], 20.00th=[ 188],
 | 30.00th=[ 201], 40.00th=[ 215], 50.00th=[ 226], 60.00th=[ 239],
 | 70.00th=[ 253], 80.00th=[ 271], 90.00th=[ 296], 95.00th=[ 321],
 | 99.00th=[ 701], 99.50th=[ 2836], 99.90th=[ 3205], 99.95th=[ 3339],
 | 99.99th=[ 3540]
 bw ( KiB/s): min= 7, max= 223, per=26.29%, avg=99.37, stdev=34.59, samples=1695
 iops : min= 1, max= 55, avg=24.16, stdev= 8.64, samples=1695
 lat (usec) : 20=0.01%
 lat (msec) : 100=0.11%, 250=67.39%, 500=30.99%, 750=0.56%, 1000=0.05%
 lat (msec) : 2000=0.17%, >=2000=0.73%
 cpu : usr=0.09%, sys=0.89%, ctx=68014, majf=0, minf=20
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=31912,21253,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=116102: Wed Apr 15 03:01:43 2020
 read: IOPS=35, BW=141KiB/s (144kB/s)(124MiB/901172msec)
 slat (usec): min=2, max=2894.9k, avg=19180.06, stdev=54568.73
 clat (usec): min=11, max=3696.7k, avg=255785.90, stdev=247754.36
 lat (msec): min=71, max=3709, avg=274.97, stdev=257.15
 clat percentiles (msec):
 | 1.00th=[ 133], 5.00th=[ 159], 10.00th=[ 174], 20.00th=[ 192],
 | 30.00th=[ 205], 40.00th=[ 218], 50.00th=[ 228], 60.00th=[ 241],
 | 70.00th=[ 255], 80.00th=[ 271], 90.00th=[ 300], 95.00th=[ 330],
 | 99.00th=[ 617], 99.50th=[ 2836], 99.90th=[ 3205], 99.95th=[ 3473],
 | 99.99th=[ 3641]
 bw ( KiB/s): min= 7, max= 279, per=26.23%, avg=148.18, stdev=41.04, samples=1702
 iops : min= 1, max= 69, avg=36.37, stdev=10.25, samples=1702
 write: IOPS=23, BW=95.4KiB/s (97.7kB/s)(83.9MiB/901172msec); 0 zone resets
 slat (usec): min=3, max=2066.5k, avg=13586.55, stdev=41539.32
 clat (msec): min=58, max=3694, avg=251.30, stdev=240.94
 lat (msec): min=63, max=3725, avg=264.88, stdev=248.29
 clat percentiles (msec):
 | 1.00th=[ 129], 5.00th=[ 155], 10.00th=[ 169], 20.00th=[ 188],
 | 30.00th=[ 201], 40.00th=[ 213], 50.00th=[ 226], 60.00th=[ 239],
 | 70.00th=[ 251], 80.00th=[ 268], 90.00th=[ 296], 95.00th=[ 326],
 | 99.00th=[ 667], 99.50th=[ 2769], 99.90th=[ 3205], 99.95th=[ 3440],
 | 99.99th=[ 3641]
 bw ( KiB/s): min= 7, max= 223, per=26.61%, avg=100.59, stdev=34.36, samples=1693
 iops : min= 1, max= 55, avg=24.47, stdev= 8.59, samples=1693
 lat (usec) : 20=0.01%
 lat (msec) : 100=0.09%, 250=67.94%, 500=30.62%, 750=0.40%, 1000=0.05%
 lat (msec) : 2000=0.16%, >=2000=0.73%
 cpu : usr=0.09%, sys=0.89%, ctx=67816, majf=0, minf=19
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=31730,21487,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
 READ: bw=565KiB/s (579kB/s), 141KiB/s-142KiB/s (144kB/s-145kB/s), io=498MiB (522MB), run=900753-901172msec
 WRITE: bw=378KiB/s (387kB/s), 94.3KiB/s-95.4KiB/s (96.5kB/s-97.7kB/s), io=333MiB (349MB), run=900753-901172msec

Disk stats (read/write):
 dm-3: ios=123568/85178, merge=0/0, ticks=1814445/107073064, in_queue=108887509, util=20.35%, aggrios=123568/85178, aggrmerge=0/0, aggrticks=1815844/107585903, aggrin_queue=109401747, aggrutil=20.41%
 dm-0: ios=123568/85178, merge=0/0, ticks=1815844/107585903, in_queue=109401747, util=20.41%, aggrios=123574/84872, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
 md127: ios=123574/84872, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=61787/42433, aggrmerge=0/0, aggrticks=902035/1902456, aggrin_queue=2752314, aggrutil=14.40%
 sdb: ios=61589/42295, merge=0/0, ticks=896372/1921815, in_queue=2766186, util=14.25%
 sda: ios=61985/42572, merge=0/0, ticks=907698/1883097, in_queue=2738442, util=14.40%

machine 4:
NOTE: tests on this machine done under some light io load which cannot be avoided, so they are not very accurate

storage:
SanDisk SDSSDH3 500G (ssd)

storage layout:
ssd: cryptsetup luks with aes-xts-plain64 + lvm

raw linear read io performance with hdparm -Tt (to understand bus limitations):
ssd:

 Timing cached reads: 4814 MB in 2.00 seconds = 2408.90 MB/sec
 Timing buffered disk reads: 1354 MB in 3.00 seconds = 451.18 MB/sec

crypto performance:

[h1]Tests are approximate using memory only (no storage IO).[/h1]

[h1]Algorithm | Key | Encryption | Decryption[/h1]

twofish-xts 512b 125.8 MiB/s 129.4 MiB/s

[h1]Tests are approximate using memory only (no storage IO).[/h1]

[h1]Algorithm | Key | Encryption | Decryption[/h1]

serpent-xts 512b 168.2 MiB/s 171.9 MiB/s

[h1]Tests are approximate using memory only (no storage IO).[/h1]

[h1]Algorithm | Key | Encryption | Decryption[/h1]

aes-xts 512b 738.9 MiB/s 721.8 MiB/s

[h1]Tests are approximate using memory only (no storage IO).[/h1]

[h1]Algorithm | Key | Encryption | Decryption[/h1]

camellia-xts 512b 183.5 MiB/s 184.7 MiB/s

[h1]Tests are approximate using memory only (no storage IO).[/h1]

PBKDF2-ripemd160 70167 iterations per second for 512-bit key

[h1]Tests are approximate using memory only (no storage IO).[/h1]

PBKDF2-sha256 110329 iterations per second for 512-bit key

[h1]Tests are approximate using memory only (no storage IO).[/h1]

PBKDF2-sha512 188321 iterations per second for 512-bit key

[h1]Tests are approximate using memory only (no storage IO).[/h1]

PBKDF2-whirlpool 107789 iterations per second for 512-bit key

xfs filesystem tests:

fio-rand-RW.fio:

ssd array:

file1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.18
Starting 4 processes
file1: Laying out IO file (1 file / 10240MiB)
Jobs: 4 (f=4): [m(4)][100.0%][r=104MiB/s,w=69.6MiB/s][r=26.7k,w=17.8k IOPS][eta 00m:00s]
file1: (groupid=0, jobs=1): err= 0: pid=864500: Wed Apr 15 02:04:11 2020
 read: IOPS=4388, BW=17.1MiB/s (17.0MB/s)(15.1GiB/900001msec)
 slat (usec): min=3, max=2554.5k, avg=119.35, stdev=1869.84
 clat (usec): min=4, max=16401k, avg=2058.41, stdev=31678.60
 lat (usec): min=14, max=16401k, avg=2178.06, stdev=31763.13
 clat percentiles (usec):
 | 1.00th=[ 186], 5.00th=[ 221], 10.00th=[ 249], 20.00th=[ 363],
 | 30.00th=[ 644], 40.00th=[ 988], 50.00th=[ 1352], 60.00th=[ 1713],
 | 70.00th=[ 2114], 80.00th=[ 2671], 90.00th=[ 3621], 95.00th=[ 5080],
 | 99.00th=[ 12649], 99.50th=[ 20579], 99.90th=[ 54264], 99.95th=[ 83362],
 | 99.99th=[154141]
 bw ( KiB/s): min= 7, max=148095, per=26.09%, avg=18286.59, stdev=19853.55, samples=1723
 iops : min= 1, max=37023, avg=4571.29, stdev=4963.39, samples=1723
 write: IOPS=2926, BW=11.4MiB/s (11.0MB/s)(10.0GiB/900001msec); 0 zone resets
 slat (usec): min=4, max=1758.8k, avg=141.70, stdev=1462.20
 clat (usec): min=18, max=16401k, avg=2042.26, stdev=39682.13
 lat (usec): min=49, max=16401k, avg=2184.29, stdev=39771.47
 clat percentiles (usec):
 | 1.00th=[ 186], 5.00th=[ 221], 10.00th=[ 247], 20.00th=[ 351],
 | 30.00th=[ 619], 40.00th=[ 947], 50.00th=[ 1287], 60.00th=[ 1663],
 | 70.00th=[ 2073], 80.00th=[ 2606], 90.00th=[ 3556], 95.00th=[ 4883],
 | 99.00th=[ 12125], 99.50th=[ 20055], 99.90th=[ 52691], 99.95th=[ 80217],
 | 99.99th=[149947]
 bw ( KiB/s): min= 7, max=98794, per=26.11%, avg=12200.11, stdev=13247.13, samples=1722
 iops : min= 1, max=24698, avg=3049.65, stdev=3311.77, samples=1722
 lat (usec) : 10=0.01%, 20=0.01%, 100=0.01%, 250=10.22%, 500=15.13%
 lat (usec) : 750=8.36%, 1000=7.15%
 lat (msec) : 2=26.74%, 4=24.52%, 10=6.51%, 20=0.85%, 50=0.40%
 lat (msec) : 100=0.08%, 250=0.03%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.01%
 cpu : usr=2.96%, sys=19.35%, ctx=4143437, majf=0, minf=24
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=3949966,2633725,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=864501: Wed Apr 15 02:04:11 2020
 read: IOPS=4371, BW=17.1MiB/s (17.9MB/s)(15.0GiB/900001msec)
 slat (usec): min=3, max=447445, avg=117.01, stdev=754.98
 clat (usec): min=78, max=15782k, avg=2093.16, stdev=36392.78
 lat (usec): min=112, max=15782k, avg=2210.48, stdev=36425.14
 clat percentiles (usec):
 | 1.00th=[ 159], 5.00th=[ 212], 10.00th=[ 241], 20.00th=[ 338],
 | 30.00th=[ 619], 40.00th=[ 971], 50.00th=[ 1352], 60.00th=[ 1729],
 | 70.00th=[ 2147], 80.00th=[ 2704], 90.00th=[ 3687], 95.00th=[ 5145],
 | 99.00th=[ 12780], 99.50th=[ 20841], 99.90th=[ 55313], 99.95th=[ 82314],
 | 99.99th=[160433]
 bw ( KiB/s): min= 7, max=187361, per=25.97%, avg=18202.14, stdev=20885.08, samples=1724
 iops : min= 1, max=46840, avg=4550.17, stdev=5221.28, samples=1724
 write: IOPS=2914, BW=11.4MiB/s (11.9MB/s)(10.0GiB/900001msec); 0 zone resets
 slat (usec): min=4, max=2554.6k, avg=146.89, stdev=2574.23
 clat (usec): min=3, max=15791k, avg=2010.83, stdev=29391.42
 lat (usec): min=18, max=15791k, avg=2158.06, stdev=29591.43
 clat percentiles (usec):
 | 1.00th=[ 159], 5.00th=[ 212], 10.00th=[ 241], 20.00th=[ 330],
 | 30.00th=[ 594], 40.00th=[ 930], 50.00th=[ 1303], 60.00th=[ 1680],
 | 70.00th=[ 2089], 80.00th=[ 2638], 90.00th=[ 3589], 95.00th=[ 5014],
 | 99.00th=[ 12256], 99.50th=[ 20055], 99.90th=[ 52691], 99.95th=[ 80217],
 | 99.99th=[154141]
 bw ( KiB/s): min= 7, max=123968, per=25.98%, avg=12137.94, stdev=13935.60, samples=1724
 iops : min= 1, max=30992, avg=3034.11, stdev=3483.90, samples=1724
 lat (usec) : 4=0.01%, 50=0.01%, 100=0.01%, 250=11.59%, 500=14.77%
 lat (usec) : 750=8.02%, 1000=6.88%
 lat (msec) : 2=25.91%, 4=24.69%, 10=6.75%, 20=0.88%, 50=0.41%
 lat (msec) : 100=0.08%, 250=0.03%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.01%
 cpu : usr=2.99%, sys=19.20%, ctx=4148391, majf=0, minf=24
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=3933948,2623346,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=864502: Wed Apr 15 02:04:11 2020
 read: IOPS=4379, BW=17.1MiB/s (17.9MB/s)(15.0GiB/900001msec)
 slat (usec): min=3, max=2206.4k, avg=118.59, stdev=1655.52
 clat (usec): min=5, max=16019k, avg=2076.84, stdev=34200.33
 lat (usec): min=111, max=16019k, avg=2195.74, stdev=34296.39
 clat percentiles (usec):
 | 1.00th=[ 182], 5.00th=[ 219], 10.00th=[ 247], 20.00th=[ 351],
 | 30.00th=[ 635], 40.00th=[ 979], 50.00th=[ 1352], 60.00th=[ 1729],
 | 70.00th=[ 2147], 80.00th=[ 2671], 90.00th=[ 3654], 95.00th=[ 5080],
 | 99.00th=[ 12649], 99.50th=[ 20841], 99.90th=[ 55313], 99.95th=[ 83362],
 | 99.99th=[162530]
 bw ( KiB/s): min= 15, max=172047, per=26.02%, avg=18238.54, stdev=20148.48, samples=1724
 iops : min= 3, max=43011, avg=4559.27, stdev=5037.11, samples=1724
 write: IOPS=2918, BW=11.4MiB/s (11.0MB/s)(10.0GiB/900001msec); 0 zone resets
 slat (usec): min=4, max=2554.6k, avg=143.84, stdev=1820.61
 clat (usec): min=105, max=16010k, avg=2026.25, stdev=34670.41
 lat (usec): min=111, max=16010k, avg=2170.42, stdev=34749.00
 clat percentiles (usec):
 | 1.00th=[ 182], 5.00th=[ 217], 10.00th=[ 245], 20.00th=[ 343],
 | 30.00th=[ 603], 40.00th=[ 938], 50.00th=[ 1303], 60.00th=[ 1663],
 | 70.00th=[ 2089], 80.00th=[ 2606], 90.00th=[ 3556], 95.00th=[ 4948],
 | 99.00th=[ 12256], 99.50th=[ 20055], 99.90th=[ 52691], 99.95th=[ 80217],
 | 99.99th=[160433]
 bw ( KiB/s): min= 7, max=116630, per=26.00%, avg=12147.34, stdev=13451.00, samples=1725
 iops : min= 1, max=29157, avg=3036.47, stdev=3362.75, samples=1725
 lat (usec) : 10=0.01%, 250=10.65%, 500=15.18%, 750=8.20%, 1000=7.01%
 lat (msec) : 2=26.33%, 4=24.64%, 10=6.61%, 20=0.87%, 50=0.40%
 lat (msec) : 100=0.08%, 250=0.03%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.01%
 cpu : usr=2.89%, sys=19.29%, ctx=4154263, majf=0, minf=24
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=3941858,2626995,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=864503: Wed Apr 15 02:04:11 2020
 read: IOPS=4385, BW=17.1MiB/s (17.0MB/s)(15.1GiB/900001msec)
 slat (usec): min=3, max=700760, avg=118.74, stdev=889.52
 clat (usec): min=115, max=16070k, avg=2068.03, stdev=32558.52
 lat (usec): min=119, max=16070k, avg=2187.07, stdev=32600.79
 clat percentiles (usec):
 | 1.00th=[ 186], 5.00th=[ 219], 10.00th=[ 247], 20.00th=[ 359],
 | 30.00th=[ 635], 40.00th=[ 979], 50.00th=[ 1352], 60.00th=[ 1713],
 | 70.00th=[ 2147], 80.00th=[ 2671], 90.00th=[ 3654], 95.00th=[ 5080],
 | 99.00th=[ 12780], 99.50th=[ 20579], 99.90th=[ 54789], 99.95th=[ 81265],
 | 99.99th=[156238]
 bw ( KiB/s): min= 7, max=152247, per=26.07%, avg=18271.75, stdev=20053.83, samples=1723
 iops : min= 1, max=38061, avg=4567.57, stdev=5013.43, samples=1723
 write: IOPS=2920, BW=11.4MiB/s (11.0MB/s)(10.0GiB/900001msec); 0 zone resets
 slat (usec): min=4, max=2554.6k, avg=143.21, stdev=2515.70
 clat (usec): min=3, max=16029k, avg=2034.65, stdev=36777.34
 lat (usec): min=121, max=16029k, avg=2178.19, stdev=36931.23
 clat percentiles (usec):
 | 1.00th=[ 184], 5.00th=[ 219], 10.00th=[ 245], 20.00th=[ 347],
 | 30.00th=[ 611], 40.00th=[ 938], 50.00th=[ 1287], 60.00th=[ 1663],
 | 70.00th=[ 2073], 80.00th=[ 2606], 90.00th=[ 3556], 95.00th=[ 4948],
 | 99.00th=[ 12125], 99.50th=[ 20055], 99.90th=[ 53216], 99.95th=[ 80217],
 | 99.99th=[156238]
 bw ( KiB/s): min= 7, max=101357, per=26.06%, avg=12174.06, stdev=13378.53, samples=1722
 iops : min= 1, max=25339, avg=3043.14, stdev=3344.63, samples=1722
 lat (usec) : 4=0.01%, 250=10.55%, 500=15.05%, 750=8.30%, 1000=7.10%
 lat (msec) : 2=26.45%, 4=24.60%, 10=6.55%, 20=0.88%, 50=0.40%
 lat (msec) : 100=0.08%, 250=0.03%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.01%
 cpu : usr=2.98%, sys=19.22%, ctx=4133616, majf=0, minf=22
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=3946881,2628140,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
 READ: bw=68.5MiB/s (71.8MB/s), 17.1MiB/s-17.1MiB/s (17.9MB/s-17.0MB/s), io=60.2GiB (64.6GB), run=900001-900001msec
 WRITE: bw=45.6MiB/s (47.8MB/s), 11.4MiB/s-11.4MiB/s (11.9MB/s-11.0MB/s), io=40.1GiB (43.1GB), run=900001-900001msec

Disk stats (read/write):
 dm-5: ios=4095694/8382399, merge=0/0, ticks=944686/590705838, in_queue=591655521, util=98.08%, aggrios=4097011/8385705, aggrmerge=0/0, aggrticks=933147/590913671, aggrin_queue=591851814, aggrutil=98.14%
 dm-0: ios=4097011/8385705, merge=0/0, ticks=933147/590913671, in_queue=591851814, util=98.14%, aggrios=4097011/7853564, aggrmerge=0/532163, aggrticks=787654/937480, aggrin_queue=148997, aggrutil=98.17%
 sdc: ios=4097011/7853564, merge=0/532163, ticks=787654/937480, in_queue=148997, util=98.17%

fs_mark on same machines:

btrfs filesystem:

ssd arryy:

[h1]fs_mark -D 1024 -N 1024 -L 100 -n 1024 -p 128 -r 32 -s 4096 -t 64 -d .[/h1]

[h1]Version 3.3, 64 thread(s) starting at Thu Apr 16 23:14:11 2020[/h1]

[h1]Sync method: INBAND FSYNC: fsync() per file in write loop.[/h1]

[h1]Directories: Round Robin between directories across 1024 subdirectories with 1024 files per subdirectory.[/h1]

[h1]File names: 128 bytes long, (96 initial bytes of time stamp with 32 random bytes at end of name)[/h1]

[h1]Files info: size 4096 bytes, written with an IO size of 16384 bytes per write[/h1]

[h1]App overhead is time in microseconds spent in the test not doing file writing related system calls.[/h1]

#
FSUse% Count Size Files/sec App Overhead
 1 65536 4096 10310.7 724416
 1 131072 4096 10780.5 727404
 1 196608 4096 10841.3 727947
 1 262144 4096 9799.0 747258
 1 327680 4096 7313.5 824792
 1 393216 4096 4994.7 902028
 1 458752 4096 4586.5 968847
 1 524288 4096 4771.6 875701
 1 589824 4096 4902.5 964081
 1 655360 4096 4981.5 922469
 1 720896 4096 4953.7 953278
 1 786432 4096 4940.7 970283
 1 851968 4096 4905.0 960853
 1 917504 4096 4866.4 975005
 1 983040 4096 4904.3 961047
 1 1048576 4096 4470.7 971468
 1 1114112 4096 4881.0 928539
 1 1179648 4096 4898.9 985251
 1 1245184 4096 4426.8 998318
 1 1310720 4096 4717.5 1024192
 1 1376256 4096 4795.5 1010882
 1 1441792 4096 4768.4 948396
 1 1507328 4096 4881.9 930210
 1 1572864 4096 4706.8 926804
 1 1638400 4096 4362.5 1014378
 1 1703936 4096 4165.5 1017512
 1 1769472 4096 4454.5 979459
 1 1835008 4096 4414.1 910763
 1 1900544 4096 4584.9 935274
 1 1966080 4096 4245.9 974326
 1 2031616 4096 4192.8 940469
 1 2097152 4096 4390.3 940586
 1 2162688 4096 4618.7 929184
 1 2228224 4096 4399.2 926809
 1 2293760 4096 4120.1 1197906
 1 2359296 4096 4321.8 975645
 1 2424832 4096 4282.7 946189
 1 2490368 4096 4406.6 998897
 1 2555904 4096 4323.5 970225
 1 2621440 4096 4433.8 929924
 1 2686976 4096 3859.8 972714
 1 2752512 4096 3277.5 959557
 1 2818048 4096 3333.6 961802
 1 2883584 4096 3387.0 929832
 1 2949120 4096 3329.5 947360
 1 3014656 4096 3342.3 952607
 1 3080192 4096 3352.9 951114
 1 3145728 4096 3364.2 1045857
 1 3211264 4096 3262.2 993562
 1 3276800 4096 3347.9 984814
 1 3342336 4096 3311.6 1007202
 1 3407872 4096 3302.7 1014171
 1 3473408 4096 3339.7 986689
 1 3538944 4096 3327.9 1179439
 1 3604480 4096 3297.4 1012885
 1 3670016 4096 3315.7 955640
 1 3735552 4096 3273.4 991597
 1 3801088 4096 3271.2 949219
 1 3866624 4096 3187.1 1085862
 1 3932160 4096 3306.2 1034104
 1 3997696 4096 3243.0 1086646
 1 4063232 4096 3118.9 1064811
 1 4128768 4096 3153.9 985299
 1 4194304 4096 3169.8 1043782
 1 4259840 4096 3151.4 1005640
 1 4325376 4096 2933.7 1152361
 1 4390912 4096 3182.4 985131
 1 4456448 4096 3104.6 1024866
 1 4521984 4096 3049.8 975983
 1 4587520 4096 3011.5 1061210
 1 4653056 4096 2894.0 1071062
 1 4718592 4096 3019.0 986460
 1 4784128 4096 3084.8 1041762
 1 4849664 4096 3151.8 1062671
 1 4915200 4096 3152.9 994083
 1 4980736 4096 2962.9 1082241
 1 5046272 4096 3207.6 974949
 1 5111808 4096 2941.4 1251538
 1 5177344 4096 3095.8 1104318
 1 5242880 4096 3130.1 1037710
 1 5308416 4096 3056.7 1010320
 1 5373952 4096 2874.9 1013132
 1 5439488 4096 3155.1 1036078
 1 5505024 4096 3194.9 932772
 1 5570560 4096 3162.1 1013478
 1 5636096 4096 3027.3 1000038
 1 5701632 4096 3064.4 976337
 1 5767168 4096 3114.2 998867
 1 5832704 4096 3038.4 1022219
 1 5898240 4096 3135.7 1063486
 1 5963776 4096 3113.2 974104
 1 6029312 4096 2926.5 1044568
 1 6094848 4096 3022.6 938138
 1 6160384 4096 3080.7 946747
 1 6225920 4096 3035.1 1041103
 1 6291456 4096 2757.0 1070905
 1 6356992 4096 3095.7 1002116
 1 6422528 4096 2960.9 996828
 1 6488064 4096 2834.2 1067899
 1 6553600 4096 3015.3 1025490

xfs filesystem:

ssd array:

[h1]fs_mark -D 1024 -N 1024 -L 100 -n 1024 -p 128 -r 32 -s 4096 -t 64 -d test[/h1]

[h1]Version 3.3, 64 thread(s) starting at Thu Apr 16 23:48:42 2020[/h1]

[h1]Sync method: INBAND FSYNC: fsync() per file in write loop.[/h1]

[h1]Directories: Round Robin between directories across 1024 subdirectories with 1024 files per subdirectory.[/h1]

[h1]File names: 128 bytes long, (96 initial bytes of time stamp with 32 random bytes at end of name)[/h1]

[h1]Files info: size 4096 bytes, written with an IO size of 16384 bytes per write[/h1]

[h1]App overhead is time in microseconds spent in the test not doing file writing related system calls.[/h1]

#
FSUse% Count Size Files/sec App Overhead
 54 65536 4096 7999.0 648553
 54 131072 4096 7017.5 669987
 54 196608 4096 4130.0 800848
 54 262144 4096 3797.9 843377
 54 327680 4096 3129.0 847898
 54 393216 4096 3077.2 873509
 54 458752 4096 2738.3 927161
 54 524288 4096 2994.4 861687
 55 589824 4096 3123.8 822056
 55 655360 4096 3166.8 851053
 55 720896 4096 3080.1 814271
 55 786432 4096 2995.3 874025
 55 851968 4096 3134.7 825648
 55 917504 4096 3108.9 838706
 55 983040 4096 2922.5 882262
 55 1048576 4096 3082.5 844851
 55 1114112 4096 2960.5 939395
 56 1179648 4096 2942.2 873126
 56 1245184 4096 2891.3 932275
 56 1310720 4096 2938.7 868676
 56 1376256 4096 3028.6 823515
 56 1441792 4096 2754.5 865692
 56 1507328 4096 2971.4 846978
 56 1572864 4096 2919.7 862855
 56 1638400 4096 2937.6 884359
 56 1703936 4096 2809.8 920798
 57 1769472 4096 2917.3 851571
 57 1835008 4096 2804.6 887153
 57 1900544 4096 2707.8 889684
 57 1966080 4096 2894.6 910081
 57 2031616 4096 2930.0 973586
 57 2097152 4096 2736.2 873193
 57 2162688 4096 2596.3 1074422
 58 2228224 4096 2586.6 1037621
 58 2293760 4096 2646.0 907799
 58 2359296 4096 2746.2 909818
 58 2424832 4096 2897.5 874643
 58 2490368 4096 2884.1 889218
 58 2555904 4096 2889.7 829827
 58 2621440 4096 2723.1 980979
 59 2686976 4096 2884.7 849886
 59 2752512 4096 2804.5 892372
 59 2818048 4096 2821.7 824137
 59 2883584 4096 2837.9 855938
 59 2949120 4096 2874.3 925673
 59 3014656 4096 2882.0 867375
 59 3080192 4096 2662.0 904706
 59 3145728 4096 2909.3 852527
 59 3211264 4096 2821.0 866250
 60 3276800 4096 2979.5 853112
 60 3342336 4096 2882.4 957123
 59 3407872 4096 2735.5 870543
 60 3473408 4096 2784.8 859071
 60 3538944 4096 2957.5 911676
 60 3604480 4096 2775.3 885856
 60 3670016 4096 2778.3 940815
 60 3735552 4096 2989.9 867906
 60 3801088 4096 2880.0 910774
 61 3866624 4096 2833.5 861072
 61 3932160 4096 2783.7 910543
 61 3997696 4096 2740.0 916634
 61 4063232 4096 2692.5 928860
 61 4128768 4096 2688.5 935526
 61 4194304 4096 2691.0 907043
 61 4259840 4096 2741.5 916414
 61 4325376 4096 2787.0 951843
 61 4390912 4096 2798.5 898106
 62 4456448 4096 2878.6 870864
 62 4521984 4096 2927.3 912141
 62 4587520 4096 2868.6 976210
 62 4653056 4096 2923.0 877059
 62 4718592 4096 2739.4 899465
 62 4784128 4096 2693.2 1005320
 62 4849664 4096 2614.0 965127
 62 4915200 4096 2826.6 895421
 62 4980736 4096 2867.8 865318
 63 5046272 4096 2833.4 896990
 63 5111808 4096 2823.4 896109
 63 5177344 4096 2763.7 880317
 63 5242880 4096 2802.1 865805
 63 5308416 4096 2828.5 874843
 63 5373952 4096 2808.4 855479
 63 5439488 4096 2890.9 871179
 63 5505024 4096 2820.4 958657
 63 5570560 4096 2878.4 842219
 64 5636096 4096 2793.5 832474
 64 5701632 4096 2783.6 872805
 64 5767168 4096 2653.2 874041
 64 5832704 4096 2661.8 905106
 64 5898240 4096 2859.8 864755
 64 5963776 4096 2863.5 879432
 64 6029312 4096 2899.7 862990
 64 6094848 4096 2641.9 929778
 64 6160384 4096 2675.6 851607
 65 6225920 4096 2671.1 935284
 65 6291456 4096 2759.3 865565
 65 6356992 4096 2637.7 904031
 65 6422528 4096 2460.4 941102
 65 6488064 4096 2738.5 895132
 65 6553600 4096 2543.7 969335

ext4 filesystem:

ssd array:

[h1]fs_mark -D 1024 -N 1024 -L 100 -n 1024 -p 128 -r 32 -s 4096 -t 64 -d test[/h1]

[h1]Version 3.3, 64 thread(s) starting at Fri Apr 17 01:09:51 2020[/h1]

[h1]Sync method: INBAND FSYNC: fsync() per file in write loop.[/h1]

[h1]Directories: Round Robin between directories across 1024 subdirectories with 1024 files per subdirectory.[/h1]

[h1]File names: 128 bytes long, (96 initial bytes of time stamp with 32 random bytes at end of name)[/h1]

[h1]Files info: size 4096 bytes, written with an IO size of 16384 bytes per write[/h1]

[h1]App overhead is time in microseconds spent in the test not doing file writing related system calls.[/h1]

#
FSUse% Count Size Files/sec App Overhead
 5 65536 4096 7864.5 552086
 5 131072 4096 6016.0 659293
 5 196608 4096 3132.8 830952
 5 262144 4096 3111.3 868417
 5 327680 4096 3024.1 856514
 5 393216 4096 3054.6 855310
 5 458752 4096 2965.3 855742
 5 524288 4096 2939.1 848230
 5 589824 4096 2954.9 996805
 5 655360 4096 2777.9 902412
 5 720896 4096 2888.6 862273
 5 786432 4096 2896.1 824348
 5 851968 4096 2913.1 845055
 5 917504 4096 2890.0 880318
 5 983040 4096 2894.7 857438
 5 1048576 4096 2801.2 816060
 5 1114112 4096 2911.2 841817
 5 1179648 4096 2713.7 844881
 5 1245184 4096 2906.6 838485
 5 1310720 4096 2680.9 887251
 5 1376256 4096 2699.2 878854
 5 1441792 4096 2853.2 833365
 5 1507328 4096 2607.5 920782
 5 1572864 4096 2696.1 851208
 5 1638400 4096 2855.7 843803
 5 1703936 4096 2481.7 992642
 5 1769472 4096 2775.0 826382
 5 1835008 4096 2882.5 839825
 5 1900544 4096 2781.9 833644
 5 1966080 4096 2847.1 837430
 5 2031616 4096 2425.9 949966
 5 2097152 4096 2765.9 861270
 5 2162688 4096 2645.1 861435
 5 2228224 4096 2643.1 872495
 5 2293760 4096 2696.8 865095
 5 2359296 4096 2788.9 821588
 5 2424832 4096 2701.5 854663
 5 2490368 4096 2720.7 845920
 5 2555904 4096 2715.9 887622
 5 2621440 4096 2677.9 911478
 5 2686976 4096 2769.2 849543
 5 2752512 4096 2724.9 904973
 5 2818048 4096 2750.3 844592
 5 2883584 4096 2754.1 825710
 5 2949120 4096 2773.4 834391
 5 3014656 4096 2494.8 926544
 5 3080192 4096 2646.0 856238
 5 3145728 4096 2618.4 833934
 5 3211264 4096 2773.9 839651
 5 3276800 4096 2589.9 850756
 5 3342336 4096 2657.2 831601
 5 3407872 4096 2370.0 960856
 5 3473408 4096 2706.7 817512
 5 3538944 4096 2797.4 797210
 5 3604480 4096 2529.4 899022
 5 3670016 4096 2596.7 815678
 5 3735552 4096 2711.9 819621
 5 3801088 4096 2631.6 851691
 5 3866624 4096 2751.4 842720
 5 3932160 4096 2565.7 855631
 5 3997696 4096 2778.3 795482
 5 4063232 4096 2723.5 889595
 5 4128768 4096 2658.2 861042
 5 4194304 4096 2699.9 819071
 5 4259840 4096 2644.2 852887
 5 4325376 4096 2754.3 807878
 5 4390912 4096 2692.8 840796
 5 4456448 4096 2819.3 807838
 5 4521984 4096 2690.4 835833
 5 4587520 4096 2778.1 846772
 5 4653056 4096 2629.5 930968
 5 4718592 4096 2708.7 831528
 5 4784128 4096 2558.6 846661
 5 4849664 4096 2663.4 898813
 5 4915200 4096 2591.5 895358
 5 4980736 4096 2765.3 853465
 5 5046272 4096 2539.0 875678
 5 5111808 4096 2720.3 864972
 5 5177344 4096 2757.4 834596
 5 5242880 4096 2640.7 842025
 5 5308416 4096 2686.0 837971
 5 5373952 4096 2642.4 869985
 5 5439488 4096 2744.4 800594
 5 5505024 4096 2736.5 941328
 5 5570560 4096 2649.0 850955
 5 5636096 4096 2729.2 821149
 5 5701632 4096 2648.9 861357
 5 5767168 4096 2548.3 876694
 5 5832704 4096 2525.8 893559
 5 5898240 4096 2703.9 848639
 5 5963776 4096 2553.9 864792
 5 6029312 4096 2548.8 874867
 5 6094848 4096 2592.4 884055
 5 6160384 4096 2548.6 868029
 5 6225920 4096 2667.0 828056
 5 6291456 4096 2559.1 881193
 5 6356992 4096 2661.5 847813
 5 6422528 4096 2501.0 838245
 5 6488064 4096 2674.3 829347
 5 6553600 4096 2566.9 860537