linux block devices io and fs tests

sss

Thu, 18 Mar 2021 13:29:28 +0300

short article with some photos about assembling mechanical keyboards

View article

some time ago i have wrote a bit about keyboards
https://hubzilla.dark-alexandr.net/display/b64.aHR0cHM6Ly9odWJ6aWxsYS5kYXJrLWFsZXhhbmRyLm5ldC9pdGVtL2EwNzFmMDFiLTNmYTEtNGUzMS1hNTRkLTc0NGU4NmZhMjBiOQ

so, i have near perfect #wasd v3 keyboard with cherry mx green switches and nice pleasant pbt keycaps, only thing it lacks  is opensource firmware and ability to program it from pc (it does have integrated method to program layouts and macrosses, but it is limited to keycodes available on keyboard itself, i have talked a bit with their tech support, and most probably they will not implement advanced programming in near future) also, i am using it already for some time, and erase plastic case a bit with my fingers, it started to wearing, you can see glossy shining under space button, it also does not feel nice to put finger where, and as we all know, humans are shitty greedy creatures which always want more ....

this time i decided to make myself present for new year, buy best available keyboard for my needs which currently exists in the world. I am dropped idea of buying shit from large vendors like shittech and company immediately, as they all currently have no quality at all in their so called "products" and funniest thing - more expensive and "top" items have lowest possible quality, so, nah this shit, shittech (лохитеч/лохотех on russian) can burn in hell using their shit as fuel, and i do not want to risk another ~$300-$500 just to learn what things get worsen, so i decided to buy from chinese friends, one more reason to not buy from "big" vendors - they do not have anything what suffice my needs and also does not support opensource software/firmware for sure.

what i want:
1. durable, high weight metal case
2. preferable default layout or as close as possible to it
3. ability to program non-default keycodes into layouts/macrosses
4. preferable opensource firmware (i am targeted at qmk, but at a time of investigating i have near zero knowledge what is qmk really is)

unfortunately very few keyboards in metal case is available, and very few metal cases for keyboards available also, most of  it is ... let's call it special purpose devices, so, no default layout of course (

i am able to find 3 acceptable keyboards.

1. x-bows ranger k6 - ansi  tkl (tkl - ten key less - no numpad).

layout: - default ansi layout without numpad keyboard.

case: - aluminum one piece tube

pcb: - cherry switches hotswap, soldered rgb diodes for each key, Weltrend WT59F164 controller (i did not know this during review, unfortunately)

firmware - proprietary, closed source (currently have no support from open source firmwares, see: https://github.com/qmk/qmk_firmware/issues/11325)

software: - many keyboard features implemented in software (available only for shitos (windows playstation)), software quality is far from usable, it constantly eating a lot of cpu, to recognize keyboard it must be repluged after software gui is launched, e.t.c., e.t.c.

price: ~$160 ( https://aliexpress.ru/item/4000874401110.html?spm=a2g0o.productlist.0.0.75a532ef84yiqD&algo_pvid=6430b693-2dc0-42bb-900c-9f96f9d6fb1b&algo_expid=6430b693-2dc0-42bb-900c-9f96f9d6fb1b-3&btsid=0b8b036d16135466096442838ef4da&ws_ab_test=searchweb0_0,searchweb201602_,searchweb201603_&sku_id=10000010059490671 )

package: keyboard selling with small bag, without switches and keycaps

features: keyboard have 4 programmable layouts, one of which only works with shitos driver, without driver this layout hardcoded to default ansi, 3 additional layouts store persistent state into controller memory, so they available without driver.
doing any programming only possible via software in shitos.

note on rgb diodes for each key: on pcb like this with soldered diodes special switches is normally used, but it is prossible to modify regular cherry switches for this type of pcb

i am used 2mm drill for this

after this simple modification, rgb backlit looks much better and brighter

conclusion: despite what i am very disappointed with software and inability to use opensource firmware, it's overall very good keyboard, i will probably use it on shitos playstation gaming console, maybe in future i will port qmk to it...

2. ymdk melody96 - ansi/iso non-default layout
layout: - in this keyboard designers attempt to place fullsize (including numpad) layout into smaller case ( a bit larger than tkl size case), it's result in slightly non-default layout, but usable

case: - it does support few cases, but since i am interested in durable metal case, my choice is - 3 parts aluminum case with separate backplate, switches plate and upper case part

pcb: - cherry switches hotswap, soldered rgb diodes at pcb bottom (it's usable only in case with half-transparent acrylic backplate, which is not my thing), reset button which switching controller into boot (flashing) mode, atmega32u4 compatible controller

firmware: - qmk as native firmware: https://qmk.fm/

software: - no, not required

price: ~$200 ( https://aliexpress.ru/item/1005001831581291.html?spm=a2g0o.productlist.0.0.6c881464PUSbsR&algo_pvid=f87c96ec-5871-452b-87fa-1799548d4954&algo_expid=f87c96ec-5871-452b-87fa-1799548d4954-6&btsid=0b8b035616135468257053744eb315&ws_ab_test=searchweb0_0,searchweb201602_,searchweb201603_&sku_id=12000017796486980)

package: keyboard selling without switches and keycaps

features: it will take some time to get used to compressed layout, also this make it inconvenient to work on default layout if i need to, so take layout as negative feature.
to be honest i am also little disappointed in case design, fittings look unreliable, i am also used layer of thin sealant to make sure pcb will not fall on backplate when fittings get loose

one more negative thing is what they decided to use same plate for both ansi/and iso layouts which lead to not very tight fitting for few switches near enter key.
i am used silicone grease to make provided stabilizers work better, as without it keys does not work properly if pressed at corners.

it's probably suboptimal grease for this case, but it works fine (at least for now).

keyboard does support most of qmk features available for keyboards and it is a lot, qmk have web frontend for inexperienced users to generate firmware (but of-course this keyboard is not for inexperienced users, so i will not explain it more), for programmers it's very nice, it allow to write software for keyboard in C directly, look at my test code for example:

#include QMK_KEYBOARD_H

#define default_layer 0
#define fn_layer 15

const uint16_t PROGMEM keymaps[][MATRIX_ROWS][MATRIX_COLS] = {
       [default_layer] = LAYOUT(
    KC_ESC,  KC_F1,   KC_F2,   KC_F3,   KC_F4,   KC_F5,   KC_F6,   KC_F7,   KC_F8,   KC_F9,   KC_F10,  KC_F11,  KC_F12,  KC_PSCR, KC_HOME, KC_END,  KC_PGUP, KC_PGDN, KC_DEL,
    KC_GRV,  KC_1,    KC_2,    KC_3,    KC_4,    KC_5,    KC_6,    KC_7,    KC_8,    KC_9,    KC_0,    KC_MINS, KC_EQL,  KC_BSPC, KC_BSPC, KC_NLCK, KC_PSLS, KC_PAST, KC_PMNS,
    KC_TAB,  KC_Q,    KC_W,    KC_E,    KC_R,    KC_T,    KC_Y,    KC_U,    KC_I,    KC_O,    KC_P,    KC_LBRC, KC_RBRC, KC_ENT,          KC_P7,   KC_P8,   KC_P9,   KC_PMNS,
    KC_CAPS, KC_A,    KC_S,    KC_D,    KC_F,    KC_G,    KC_H,    KC_J,    KC_K,    KC_L,    KC_SCLN, KC_QUOT, KC_BSLS,                    KC_P4,   KC_P5,   KC_P6,   KC_PPLS,
    KC_LSFT, KC_NUBS, KC_Z,    KC_X,    KC_C,    KC_V,    KC_B,    KC_N,    KC_M,    KC_COMM, KC_DOT,  KC_SLSH, KC_RSFT,          KC_UP,   KC_P1,   KC_P2,   KC_P3,   KC_PENT,
    KC_LCTL, KC_LGUI, KC_LALT,                            KC_SPC,                             KC_RALT, MO(fn_layer),   KC_LCTL,   KC_LEFT, KC_DOWN, KC_RGHT, KC_P0,   KC_PDOT, KC_PENT),

    [fn_layer] = LAYOUT(
    TO(default_layer), TO(1), TO(2), TO(3), TO(3), TO(4), TO(5), TO(6), TO(7), TO(8), TO(9), TO(10), TO(11), TO(12), TO(13), TO(14), KC_NO, KC_NO, KC_MUTE,
    KC_NO, KC_NO, KC_NO, KC_NO, KC_NO, KC_NO, KC_NO, KC_NO, KC_NO, KC_NO, KC_NO, KC_NO, KC_NO, KC_NO, KC_NO, KC_NO, KC_MPLY, KC_MSTP, KC_VOLD,
    KC_NO, KC_NO, KC_WH_U, KC_NO, KC_NO, KC_NO, KC_NO, KC_NO, KC_NO, KC_NO, KC_NO, KC_NO, KC_NO, KC_NO, KC_NO, KC_MPRV, KC_MNXT, KC_VOLU,
    KC_NO, KC_WH_L, KC_WH_D, KC_WH_R, KC_NO, KC_NO, KC_NO, KC_NO, KC_NO, KC_NO, KC_NO, KC_NO, KC_NO, KC_NO, KC_NO, KC_NO, KC_VOLU,
    KC_NO, KC_NO, KC_NO, KC_CUT, KC_COPY, KC_PASTE, KC_NO, KC_NO, KC_NO, KC_NO, KC_NO, KC_NO, KC_NO, RGB_VAI, RGB_MOD, KC_NO, KC_NO, KC_NO,
    RESET, KC_NO, KC_NO, KC_NO, KC_NO, KC_TRNS, KC_NO, KC_NO, RGB_VAD, RGB_TOG, KC_NO, KC_NO, KC_NO ),
};

bool process_record_user(uint16_t keycode, keyrecord_t *record) {
    switch(keycode)
    {
        default:
            break;
    }
    return true;
}

for more advanced things qmk have api to do most stuff, program macrosses e.t.c., i dropped additional layers in example above, as qmk programming is not main topic of this post, you can learn more on official website https://qmk.fm/

conclusion: most probably i will use melody96 as my main working keyboard, for now i am sometime missing at some keys on this compressed layout, but i hope i will get used to it soon, qmk does allow me to do everything i need from keyboard.

3. id75 (idobo75) - ortholinear 5x15 layout

layout - ortholinear 5x15 layout

case: aluminum two parts with acrylic plate attached to backplate

pcb - cherry switches hotswap, soldered rgb diodes at pcb bottom, reset button which switching controller into boot (flashing) mode, atmega32u4 compatible controller

firmware - qmk as native firmware: https://qmk.fm/

software - no, not required

price: ~$100 ( https://aliexpress.ru/item/4000311498726.html?spm=a2g0o.productlist.0.0.762c2beehcJA5P&algo_pvid=c0f35677-3230-4fba-9aed-948254eccabc&algo_expid=c0f35677-3230-4fba-9aed-948254eccabc-0&btsid=0b8b036316135483975447747ecfd2&ws_ab_test=searchweb0_0,searchweb201602_,searchweb201603_&sku_id=10000001290207378 )

package: keyboard selling without switches and keycaps

feature: i am very unfamiliar with this layout, or more precisely with layout absence, it's just 5x15 key rows, surprisingly i am able to use blind typing on this keyboard immediately, of course with some misses and relatively slow speed, but still, i feared what i will not able to do anything at all....
as for other features, it is also qmk so we have most qmk features for keyboards.
one thing i liked very much about this keyboard - it have tight, very ease to assemble case in which acrylic plate for backpate doing same thing as sealant which i added into melody96 case:

, but additionally it's half transparent, so backside diodes on pcb actually can be seen, this acrylic plate shut tight pcb from the back, also front part of case used as switch fitting plate, have massive stabs as part of it so pcb can be screwed with long reliable screws, so assembled it looks and feel very reliable (which i can't say about melody96)

conclusion: i am not sure if i will use this keyboard as main working board (most probably no), but as it do not take much space on desktop it will be used at least for additional hotkeys, also most probably i will take it with me in travels, and will use it for accessing "terminals" which require physical keyboard connection without disconnecting my main working keyboard

general notes:
i am using cherry mx green switches and their clones from gateron, gateron clones feels a bit softer, but still acceptable and clicky, also looks like gateron switches is durable enough, their price is x5 cheaper than original cherry switches, but for most often used keys like space, fn, delete, ctrl, alt, enter i am using original cherry switches.

it's very easy to broke switch during insertion into hot-plug socket on pcb so you must be extra careful during this step, few times i ended up with screwed legs like this:

, but it can be fixed with something like pliers, of course you must be extra careful during aligning the legs step., fortunately i have not broke completely single switch (yet...).

few words about keycaps, as abs keycaps does not live long enough for me (more info: https://hubzilla.dark-alexandr.net/display/b64.aHR0cHM6Ly9odWJ6aWxsYS5kYXJrLWFsZXhhbmRyLm5ldC9pdGVtL2EwNzFmMDFiLTNmYTEtNGUzMS1hNTRkLTc0NGU4NmZhMjBiOQ ), i prefer pbt or double-shot pbt keycaps, keycaps made whole from pbt plastic are very expensive (price is comparable to keyboard price: $80-$300) i had only two sets of whole pbt keycaps one of which i gifted with keyboard to my friend, second is here:

used for melody96 keyboard, most of my keycaps are double shot pbt keycaps, price for double-shot pbt keycaps are vary from $10 to infinity, most of my sets costs from $10 to $30 - this is lowest price level.

#hardware #kink #keyboard #keyboards #mechanical_keyboards #cherry_mx #programming #c_language

hardware kink keyboard keyboards mechanical_keyboards cherry_mx programming c_language

sss

Mon, 29 Mar 2021 03:30:25 +0300

sss

Fri, 09 Jul 2021 15:15:16 +0300

sss

Tue, 12 Oct 2021 10:18:45 +0300

short description of unix-like os package management systems

sss

Sun, 11 Apr 2021 17:55:13 +0300 last edited: Sun, 12 Sep 2021 00:09:47 +0300

short description of unix-like os package management systems

View article

linux freebsd nixos guix gentoo exherbo archlinux deb rpm

Linux-Windows comparison

sss

Fri, 29 May 2020 08:17:58 +0300 last edited: Fri, 09 Jul 2021 13:57:01 +0300

linux unix windows review comparison humor

“Мне нечего скрывать” и другие ошибочные толкования приватности

sss

Thu, 18 Mar 2021 10:55:24 +0300 last edited: Thu, 18 Mar 2021 11:17:18 +0300

неплохая статья о приватности, и разбор распространенного неверного аргумента “Мне нечего скрывать”

View article

privacy russian приватность

sss

Tue, 06 Apr 2021 09:09:52 +0300

linux block devices io and fs tests

sss

Tue, 12 May 2020 07:14:29 +0300

small linux block devices performance test

#linux #filesystems #xfs #btrfs #ext4 #lvm #raid #mdadm #luks #cryptsetup

linux io test on different storage and diferent filesystems with fio:

NOTES:
tests done on different hw configurations, on devices wrapped by mdadm raid0, cryptsetup luks crypto and lvm, parameters, crypto and raw io performance will be shown for each test.

machine 1:

storage:
2 x Samsung SSD 860 EVO 500GB
3 x WDC WD40PURZ-85TTDY0

storage layout:

ssd array: 2 x ssd mdadm raid 0 + cryptsetup luks with aes-xts-plain64 + lvm
hdd array: 3 x hdd mdadm raid 0 + cryptsetup luks with camellia-xts-plain64 + lvm

raw linear read io performance with hdparm -Tt (to understand bus limitations):

ssd array:

Timing cached reads: 22102 MB in 2.00 seconds = 11064.06 MB/sec
 Timing buffered disk reads: 3108 MB in 3.00 seconds = 1035.40 MB/sec

hdd array:

Timing cached reads: 23258 MB in 2.00 seconds = 11643.60 MB/sec
 Timing buffered disk reads: 1522 MB in 3.00 seconds = 506.96 MB/sec

crypto performance:

[h1]Tests are approximate using memory only (no storage IO).[/h1]

[h1]Algorithm | Key | Encryption | Decryption[/h1]

twofish-xts 512b 353.1 MiB/s 357.0 MiB/s

[h1]Tests are approximate using memory only (no storage IO).[/h1]

[h1]Algorithm | Key | Encryption | Decryption[/h1]

serpent-xts 512b 612.3 MiB/s 609.4 MiB/s

[h1]Tests are approximate using memory only (no storage IO).[/h1]

[h1]Algorithm | Key | Encryption | Decryption[/h1]

aes-xts 512b 2681.4 MiB/s 2723.0 MiB/s

[h1]Tests are approximate using memory only (no storage IO).[/h1]

[h1]Algorithm | Key | Encryption | Decryption[/h1]

camellia-xts 512b 740.9 MiB/s 755.7 MiB/s

[h1]Tests are approximate using memory only (no storage IO).[/h1]

PBKDF2-ripemd160 200722 iterations per second for 512-bit key

[h1]Tests are approximate using memory only (no storage IO).[/h1]

PBKDF2-sha256 586451 iterations per second for 512-bit key

[h1]Tests are approximate using memory only (no storage IO).[/h1]

PBKDF2-sha512 781353 iterations per second for 512-bit key

[h1]Tests are approximate using memory only (no storage IO).[/h1]

PBKDF2-whirlpool 369737 iterations per second for 512-bit key

xfs filesystem tests:

fio-rand-RW.fio:

ssd array:

NOTE: filesystem BLOCK_SIZE=”512”

file1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.18
Starting 4 processes
file1: Laying out IO file (1 file / 10240MiB)
Jobs: 4 (f=4): [m(4)][100.0%][r=125MiB/s,w=83.3MiB/s][r=31.0k,w=21.3k IOPS][eta 00m:00s]
file1: (groupid=0, jobs=1): err= 0: pid=3927: Wed Apr 15 01:25:51 2020
 read: IOPS=14.2k, BW=55.4MiB/s (58.1MB/s)(48.7GiB/900001msec)
 slat (nsec): min=611, max=15747M, avg=30942.93, stdev=10093578.28
 clat (nsec): min=1002, max=22729M, avg=647744.82, stdev=56483253.23
 lat (usec): min=24, max=22729k, avg=678.75, stdev=57420.76
 clat percentiles (usec):
 | 1.00th=[ 53], 5.00th=[ 62], 10.00th=[ 67], 20.00th=[ 73],
 | 30.00th=[ 80], 40.00th=[ 91], 50.00th=[ 198], 60.00th=[ 265],
 | 70.00th=[ 404], 80.00th=[ 611], 90.00th=[ 1057], 95.00th=[ 1647],
 | 99.00th=[ 3097], 99.50th=[ 3687], 99.90th=[ 5145], 99.95th=[ 5866],
 | 99.99th=[10290]
 bw ( KiB/s): min= 8, max=542264, per=36.81%, avg=83508.11, stdev=106473.54, samples=1222
 iops : min= 2, max=135566, avg=20877.03, stdev=26618.40, samples=1222
 write: IOPS=9453, BW=36.9MiB/s (38.7MB/s)(32.5GiB/900001msec); 0 zone resets
 slat (nsec): min=1012, max=17920M, avg=51399.19, stdev=13365178.34
 clat (usec): min=24, max=22729k, avg=615.97, stdev=53329.84
 lat (usec): min=26, max=22729k, avg=667.44, stdev=55364.63
 clat percentiles (usec):
 | 1.00th=[ 52], 5.00th=[ 62], 10.00th=[ 67], 20.00th=[ 73],
 | 30.00th=[ 80], 40.00th=[ 89], 50.00th=[ 186], 60.00th=[ 245],
 | 70.00th=[ 379], 80.00th=[ 578], 90.00th=[ 1012], 95.00th=[ 1582],
 | 99.00th=[ 3032], 99.50th=[ 3621], 99.90th=[ 5014], 99.95th=[ 5735],
 | 99.99th=[ 9372]
 bw ( KiB/s): min= 7, max=364976, per=36.96%, avg=55910.94, stdev=71077.22, samples=1217
 iops : min= 1, max=91244, avg=13977.74, stdev=17769.31, samples=1217
 lat (usec) : 2=0.01%, 50=0.70%, 100=44.30%, 250=14.47%, 500=16.34%
 lat (usec) : 750=8.82%, 1000=4.80%
 lat (msec) : 2=7.22%, 4=3.01%, 10=0.32%, 20=0.01%, 50=0.01%
 lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.01%
 cpu : usr=2.22%, sys=17.10%, ctx=7748150, majf=0, minf=23
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=12759994,8508306,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=3928: Wed Apr 15 01:25:51 2020
 read: IOPS=14.2k, BW=55.4MiB/s (58.1MB/s)(48.7GiB/900001msec)
 slat (nsec): min=632, max=14055M, avg=28707.68, stdev=8367525.63
 clat (nsec): min=1262, max=22585M, avg=641342.80, stdev=54006164.38
 lat (usec): min=25, max=22585k, avg=670.12, stdev=54889.64
 clat percentiles (usec):
 | 1.00th=[ 52], 5.00th=[ 62], 10.00th=[ 67], 20.00th=[ 74],
 | 30.00th=[ 81], 40.00th=[ 91], 50.00th=[ 198], 60.00th=[ 269],
 | 70.00th=[ 404], 80.00th=[ 611], 90.00th=[ 1057], 95.00th=[ 1647],
 | 99.00th=[ 3097], 99.50th=[ 3687], 99.90th=[ 5145], 99.95th=[ 5866],
 | 99.99th=[10028]
 bw ( KiB/s): min= 8, max=534096, per=36.83%, avg=83552.06, stdev=106239.96, samples=1221
 iops : min= 2, max=133524, avg=20888.01, stdev=26559.98, samples=1221
 write: IOPS=9459, BW=36.0MiB/s (38.7MB/s)(32.5GiB/900001msec); 0 zone resets
 slat (nsec): min=1002, max=17920M, avg=54725.10, stdev=15045267.29
 clat (usec): min=24, max=22585k, avg=625.45, stdev=55917.05
 lat (usec): min=26, max=22585k, avg=680.25, stdev=58145.95
 clat percentiles (usec):
 | 1.00th=[ 51], 5.00th=[ 62], 10.00th=[ 67], 20.00th=[ 73],
 | 30.00th=[ 80], 40.00th=[ 89], 50.00th=[ 186], 60.00th=[ 245],
 | 70.00th=[ 379], 80.00th=[ 578], 90.00th=[ 1012], 95.00th=[ 1582],
 | 99.00th=[ 3032], 99.50th=[ 3589], 99.90th=[ 5014], 99.95th=[ 5735],
 | 99.99th=[ 9765]
 bw ( KiB/s): min= 7, max=359128, per=37.01%, avg=55994.86, stdev=71016.78, samples=1216
 iops : min= 1, max=89782, avg=13998.72, stdev=17754.19, samples=1216
 lat (usec) : 2=0.01%, 50=0.85%, 100=44.08%, 250=14.51%, 500=16.41%
 lat (usec) : 750=8.79%, 1000=4.79%
 lat (msec) : 2=7.23%, 4=3.02%, 10=0.32%, 20=0.01%, 50=0.01%
 lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.01%
 cpu : usr=2.24%, sys=17.08%, ctx=7664942, majf=0, minf=24
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=12756139,8513933,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=3929: Wed Apr 15 01:25:51 2020
 read: IOPS=14.2k, BW=55.4MiB/s (58.1MB/s)(48.7GiB/900001msec)
 slat (nsec): min=641, max=17920M, avg=30274.08, stdev=10110966.15
 clat (usec): min=22, max=22670k, avg=649.54, stdev=55193.99
 lat (usec): min=23, max=22670k, avg=679.88, stdev=56204.10
 clat percentiles (usec):
 | 1.00th=[ 51], 5.00th=[ 62], 10.00th=[ 67], 20.00th=[ 74],
 | 30.00th=[ 81], 40.00th=[ 91], 50.00th=[ 198], 60.00th=[ 265],
 | 70.00th=[ 404], 80.00th=[ 611], 90.00th=[ 1057], 95.00th=[ 1647],
 | 99.00th=[ 3097], 99.50th=[ 3687], 99.90th=[ 5145], 99.95th=[ 5932],
 | 99.99th=[10159]
 bw ( KiB/s): min= 8, max=537344, per=36.82%, avg=83527.72, stdev=106442.07, samples=1222
 iops : min= 2, max=134336, avg=20881.93, stdev=26610.52, samples=1222
 write: IOPS=9457, BW=36.9MiB/s (38.7MB/s)(32.5GiB/900001msec); 0 zone resets
 slat (nsec): min=991, max=15747M, avg=52380.86, stdev=13341407.56
 clat (nsec): min=1192, max=22670M, avg=612775.08, stdev=55147764.50
 lat (usec): min=23, max=22670k, avg=665.23, stdev=57123.82
 clat percentiles (usec):
 | 1.00th=[ 49], 5.00th=[ 61], 10.00th=[ 67], 20.00th=[ 73],
 | 30.00th=[ 80], 40.00th=[ 90], 50.00th=[ 186], 60.00th=[ 245],
 | 70.00th=[ 375], 80.00th=[ 578], 90.00th=[ 1012], 95.00th=[ 1582],
 | 99.00th=[ 3032], 99.50th=[ 3589], 99.90th=[ 5014], 99.95th=[ 5800],
 | 99.99th=[ 9503]
 bw ( KiB/s): min= 8, max=358760, per=37.03%, avg=56023.15, stdev=71031.30, samples=1215
 iops : min= 2, max=89690, avg=14005.79, stdev=17757.82, samples=1215
 lat (usec) : 2=0.01%, 50=1.01%, 100=43.83%, 250=14.60%, 500=16.42%
 lat (usec) : 750=8.80%, 1000=4.79%
 lat (msec) : 2=7.21%, 4=3.02%, 10=0.32%, 20=0.01%, 50=0.01%
 lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.01%
 cpu : usr=2.25%, sys=17.08%, ctx=7573034, majf=0, minf=23
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=12763389,8511685,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=3930: Wed Apr 15 01:25:51 2020
 read: IOPS=14.2k, BW=55.4MiB/s (58.1MB/s)(48.7GiB/900001msec)
 slat (nsec): min=602, max=15251M, avg=26931.70, stdev=8058751.14
 clat (usec): min=24, max=22701k, avg=647.36, stdev=56304.54
 lat (usec): min=25, max=22701k, avg=674.36, stdev=56896.75
 clat percentiles (usec):
 | 1.00th=[ 54], 5.00th=[ 62], 10.00th=[ 67], 20.00th=[ 73],
 | 30.00th=[ 80], 40.00th=[ 91], 50.00th=[ 198], 60.00th=[ 269],
 | 70.00th=[ 404], 80.00th=[ 611], 90.00th=[ 1057], 95.00th=[ 1647],
 | 99.00th=[ 3097], 99.50th=[ 3654], 99.90th=[ 5080], 99.95th=[ 5866],
 | 99.99th=[10290]
 bw ( KiB/s): min= 8, max=538712, per=36.89%, avg=83694.31, stdev=106111.60, samples=1220
 iops : min= 2, max=134678, avg=20923.57, stdev=26527.89, samples=1220
 write: IOPS=9449, BW=36.9MiB/s (38.7MB/s)(32.4GiB/900001msec); 0 zone resets
 slat (nsec): min=982, max=17920M, avg=57355.87, stdev=15301847.63
 clat (nsec): min=1393, max=22701M, avg=616232.01, stdev=53793093.23
 lat (usec): min=26, max=22701k, avg=673.66, stdev=56413.37
 clat percentiles (usec):
 | 1.00th=[ 53], 5.00th=[ 62], 10.00th=[ 67], 20.00th=[ 73],
 | 30.00th=[ 80], 40.00th=[ 89], 50.00th=[ 188], 60.00th=[ 245],
 | 70.00th=[ 375], 80.00th=[ 578], 90.00th=[ 1004], 95.00th=[ 1582],
 | 99.00th=[ 2999], 99.50th=[ 3589], 99.90th=[ 5014], 99.95th=[ 5735],
 | 99.99th=[10028]
 bw ( KiB/s): min= 8, max=358920, per=36.94%, avg=55889.27, stdev=70681.64, samples=1217
 iops : min= 2, max=89730, avg=13972.32, stdev=17670.41, samples=1217
 lat (usec) : 2=0.01%, 50=0.60%, 100=44.09%, 250=14.70%, 500=16.51%
 lat (usec) : 750=8.80%, 1000=4.77%
 lat (msec) : 2=7.21%, 4=3.00%, 10=0.31%, 20=0.01%, 50=0.01%
 lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.01%
 cpu : usr=2.22%, sys=17.14%, ctx=7555022, majf=0, minf=23
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=12767350,8504713,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
 READ: bw=222MiB/s (232MB/s), 55.4MiB/s-55.4MiB/s (58.1MB/s-58.1MB/s), io=195GiB (209GB), run=900001-900001msec
 WRITE: bw=148MiB/s (155MB/s), 36.9MiB/s-36.0MiB/s (38.7MB/s-38.7MB/s), io=130GiB (139GB), run=900001-900001msec

Disk stats (read/write):
 dm-3: ios=4275603/17609796, merge=0/0, ticks=1415389/1013248346, in_queue=1014664460, util=98.65%, aggrios=4277523/17609907, aggrmerge=0/0, aggrticks=1411929/1013681205, aggrin_queue=1015093858, aggrutil=98.65%
 dm-0: ios=4277523/17609907, merge=0/0, ticks=1411929/1013681205, in_queue=1015093858, util=98.65%, aggrios=4278851/17673756, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
 md128: ios=4278851/17673756, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=2139263/6387625, aggrmerge=160/2449587, aggrticks=294633/673499, aggrin_queue=32241, aggrutil=97.61%
 sdd: ios=2139623/6390350, merge=75/2451332, ticks=299397/894150, in_queue=43498, util=91.78%
 sda: ios=2138904/6384901, merge=245/2447843, ticks=289869/452848, in_queue=20984, util=97.61%

hdd array:

file1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.18
Starting 4 processes
file1: Laying out IO file (1 file / 10240MiB)
Jobs: 4 (f=4): [m(4)][100.0%][r=534KiB/s,w=403KiB/s][r=133,w=100 IOPS][eta 00m:00s]
file1: (groupid=0, jobs=1): err= 0: pid=6427: Wed Apr 15 01:45:56 2020
 read: IOPS=40, BW=163KiB/s (167kB/s)(143MiB/900027msec)
 slat (nsec): min=1212, max=878192k, avg=16018904.58, stdev=21699428.23
 clat (msec): min=32, max=2822, avg=221.78, stdev=153.22
 lat (msec): min=40, max=2862, avg=237.80, stdev=161.48
 clat percentiles (msec):
 | 1.00th=[ 89], 5.00th=[ 113], 10.00th=[ 126], 20.00th=[ 142],
 | 30.00th=[ 155], 40.00th=[ 165], 50.00th=[ 178], 60.00th=[ 192],
 | 70.00th=[ 211], 80.00th=[ 257], 90.00th=[ 384], 95.00th=[ 481],
 | 99.00th=[ 860], 99.50th=[ 1083], 99.90th=[ 1687], 99.95th=[ 2005],
 | 99.99th=[ 2366]
 bw ( KiB/s): min= 8, max= 328, per=25.35%, avg=163.77, stdev=73.33, samples=1789
 iops : min= 2, max= 82, avg=40.92, stdev=18.35, samples=1789
 write: IOPS=27, BW=109KiB/s (111kB/s)(95.4MiB/900027msec); 0 zone resets
 slat (nsec): min=1673, max=797254k, avg=12810828.53, stdev=20531401.74
 clat (nsec): min=1874, max=2770.1M, avg=219834449.76, stdev=158340065.25
 lat (msec): min=34, max=2870, avg=232.65, stdev=165.75
 clat percentiles (msec):
 | 1.00th=[ 85], 5.00th=[ 108], 10.00th=[ 123], 20.00th=[ 138],
 | 30.00th=[ 150], 40.00th=[ 163], 50.00th=[ 174], 60.00th=[ 188],
 | 70.00th=[ 209], 80.00th=[ 257], 90.00th=[ 380], 95.00th=[ 477],
 | 99.00th=[ 902], 99.50th=[ 1167], 99.90th=[ 1670], 99.95th=[ 2039],
 | 99.99th=[ 2500]
 bw ( KiB/s): min= 7, max= 288, per=25.33%, avg=109.69, stdev=53.25, samples=1780
 iops : min= 1, max= 72, avg=27.40, stdev=13.32, samples=1780
 lat (usec) : 2=0.01%
 lat (msec) : 50=0.03%, 100=2.56%, 250=76.40%, 500=16.74%, 750=2.84%
 lat (msec) : 1000=0.74%, 2000=0.64%, >=2000=0.05%
 cpu : usr=0.02%, sys=0.70%, ctx=77997, majf=0, minf=19
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=36642,24424,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=6428: Wed Apr 15 01:45:56 2020
 read: IOPS=39, BW=159KiB/s (163kB/s)(140MiB/900027msec)
 slat (nsec): min=1242, max=753470k, avg=16238498.10, stdev=22017584.22
 clat (msec): min=50, max=2980, avg=226.92, stdev=161.24
 lat (msec): min=58, max=3103, avg=243.15, stdev=169.55
 clat percentiles (msec):
 | 1.00th=[ 93], 5.00th=[ 114], 10.00th=[ 128], 20.00th=[ 144],
 | 30.00th=[ 157], 40.00th=[ 167], 50.00th=[ 180], 60.00th=[ 194],
 | 70.00th=[ 215], 80.00th=[ 264], 90.00th=[ 388], 95.00th=[ 493],
 | 99.00th=[ 936], 99.50th=[ 1167], 99.90th=[ 1720], 99.95th=[ 2056],
 | 99.99th=[ 2802]
 bw ( KiB/s): min= 7, max= 336, per=24.79%, avg=160.17, stdev=72.25, samples=1790
 iops : min= 1, max= 84, avg=40.02, stdev=18.07, samples=1790
 write: IOPS=26, BW=106KiB/s (109kB/s)(93.6MiB/900027msec); 0 zone resets
 slat (nsec): min=1603, max=576628k, avg=13256187.30, stdev=20718121.87
 clat (nsec): min=1543, max=3015.3M, avg=223824877.06, stdev=163028368.52
 lat (msec): min=34, max=3088, avg=237.08, stdev=170.93
 clat percentiles (msec):
 | 1.00th=[ 89], 5.00th=[ 112], 10.00th=[ 125], 20.00th=[ 140],
 | 30.00th=[ 153], 40.00th=[ 165], 50.00th=[ 178], 60.00th=[ 190],
 | 70.00th=[ 211], 80.00th=[ 262], 90.00th=[ 384], 95.00th=[ 485],
 | 99.00th=[ 944], 99.50th=[ 1200], 99.90th=[ 1770], 99.95th=[ 2265],
 | 99.99th=[ 2668]
 bw ( KiB/s): min= 7, max= 264, per=24.85%, avg=107.59, stdev=53.57, samples=1780
 iops : min= 1, max= 66, avg=26.87, stdev=13.40, samples=1780
 lat (usec) : 2=0.01%
 lat (msec) : 50=0.01%, 100=2.08%, 250=76.35%, 500=16.82%, 750=3.20%
 lat (msec) : 1000=0.66%, 2000=0.83%, >=2000=0.06%
 cpu : usr=0.02%, sys=0.60%, ctx=77652, majf=0, minf=19
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=35859,23955,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=6429: Wed Apr 15 01:45:56 2020
 read: IOPS=40, BW=163KiB/s (167kB/s)(143MiB/900027msec)
 slat (nsec): min=1443, max=766424k, avg=15892940.95, stdev=21352833.36
 clat (nsec): min=1884, max=2955.3M, avg=221958326.00, stdev=162678407.80
 lat (msec): min=44, max=3029, avg=237.85, stdev=171.07
 clat percentiles (msec):
 | 1.00th=[ 91], 5.00th=[ 113], 10.00th=[ 125], 20.00th=[ 140],
 | 30.00th=[ 153], 40.00th=[ 165], 50.00th=[ 176], 60.00th=[ 190],
 | 70.00th=[ 209], 80.00th=[ 255], 90.00th=[ 380], 95.00th=[ 485],
 | 99.00th=[ 911], 99.50th=[ 1167], 99.90th=[ 1854], 99.95th=[ 2232],
 | 99.99th=[ 2534]
 bw ( KiB/s): min= 8, max= 384, per=25.40%, avg=164.08, stdev=74.91, samples=1789
 iops : min= 2, max= 96, avg=40.99, stdev=18.74, samples=1789
 write: IOPS=27, BW=109KiB/s (112kB/s)(95.9MiB/900027msec); 0 zone resets
 slat (nsec): min=1583, max=589310k, avg=12886125.28, stdev=20885914.54
 clat (msec): min=31, max=2931, avg=217.75, stdev=155.35
 lat (msec): min=41, max=3035, avg=230.64, stdev=162.83
 clat percentiles (msec):
 | 1.00th=[ 87], 5.00th=[ 109], 10.00th=[ 122], 20.00th=[ 138],
 | 30.00th=[ 148], 40.00th=[ 161], 50.00th=[ 174], 60.00th=[ 186],
 | 70.00th=[ 207], 80.00th=[ 255], 90.00th=[ 380], 95.00th=[ 481],
 | 99.00th=[ 869], 99.50th=[ 1099], 99.90th=[ 1636], 99.95th=[ 2005],
 | 99.99th=[ 2903]
 bw ( KiB/s): min= 7, max= 304, per=25.57%, avg=110.72, stdev=53.36, samples=1773
 iops : min= 1, max= 76, avg=27.65, stdev=13.35, samples=1773
 lat (usec) : 2=0.01%
 lat (msec) : 50=0.02%, 100=2.46%, 250=77.00%, 500=16.15%, 750=2.86%
 lat (msec) : 1000=0.75%, 2000=0.68%, >=2000=0.07%
 cpu : usr=0.03%, sys=0.59%, ctx=77951, majf=0, minf=18
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=36713,24552,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=6430: Wed Apr 15 01:45:56 2020
 read: IOPS=40, BW=162KiB/s (165kB/s)(142MiB/900027msec)
 slat (nsec): min=1322, max=824107k, avg=15921847.02, stdev=21301902.36
 clat (usec): min=2, max=2898.1k, avg=223385.40, stdev=158901.08
 lat (msec): min=27, max=3079, avg=239.31, stdev=167.29
 clat percentiles (msec):
 | 1.00th=[ 92], 5.00th=[ 113], 10.00th=[ 126], 20.00th=[ 142],
 | 30.00th=[ 155], 40.00th=[ 165], 50.00th=[ 178], 60.00th=[ 192],
 | 70.00th=[ 211], 80.00th=[ 259], 90.00th=[ 388], 95.00th=[ 485],
 | 99.00th=[ 894], 99.50th=[ 1133], 99.90th=[ 1770], 99.95th=[ 2072],
 | 99.99th=[ 2836]
 bw ( KiB/s): min= 7, max= 344, per=25.16%, avg=162.53, stdev=73.59, samples=1788
 iops : min= 1, max= 86, avg=40.61, stdev=18.41, samples=1788
 write: IOPS=27, BW=109KiB/s (112kB/s)(95.8MiB/900027msec); 0 zone resets
 slat (nsec): min=1583, max=656166k, avg=13099102.70, stdev=21441180.89
 clat (msec): min=29, max=2542, avg=219.31, stdev=158.69
 lat (msec): min=32, max=2556, avg=232.41, stdev=165.99
 clat percentiles (msec):
 | 1.00th=[ 88], 5.00th=[ 110], 10.00th=[ 122], 20.00th=[ 138],
 | 30.00th=[ 150], 40.00th=[ 161], 50.00th=[ 174], 60.00th=[ 188],
 | 70.00th=[ 207], 80.00th=[ 255], 90.00th=[ 384], 95.00th=[ 481],
 | 99.00th=[ 919], 99.50th=[ 1133], 99.90th=[ 1720], 99.95th=[ 2039],
 | 99.99th=[ 2433]
 bw ( KiB/s): min= 7, max= 304, per=25.52%, avg=110.50, stdev=54.44, samples=1774
 iops : min= 1, max= 76, avg=27.60, stdev=13.62, samples=1774
 lat (usec) : 4=0.01%
 lat (msec) : 50=0.03%, 100=2.30%, 250=76.78%, 500=16.50%, 750=2.89%
 lat (msec) : 1000=0.74%, 2000=0.69%, >=2000=0.07%
 cpu : usr=0.02%, sys=0.63%, ctx=77963, majf=0, minf=18
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=36345,24519,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
 READ: bw=647KiB/s (662kB/s), 159KiB/s-163KiB/s (163kB/s-167kB/s), io=569MiB (596MB), run=900027-900027msec
 WRITE: bw=433KiB/s (443kB/s), 106KiB/s-109KiB/s (109kB/s-112kB/s), io=381MiB (399MB), run=900027-900027msec

Disk stats (read/write):
 dm-2: ios=147608/115656, merge=0/0, ticks=1852342/56226343, in_queue=58078685, util=21.65%, aggrios=147616/115661, aggrmerge=0/0, aggrticks=1852266/56226297, aggrin_queue=58078563, aggrutil=21.66%
 dm-1: ios=147616/115661, merge=0/0, ticks=1852266/56226297, in_queue=58078563, util=21.66%, aggrios=147617/115710, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
 md127: ios=147617/115710, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=49206/38650, aggrmerge=0/17, aggrticks=615920/1354928, aggrin_queue=1927653, aggrutil=12.05%
 sdb: ios=49620/38693, merge=0/23, ticks=675653/1576403, in_queue=2208415, util=12.05%
 sde: ios=49050/38640, merge=0/17, ticks=590031/1274911, in_queue=1821844, util=12.01%
 sdc: ios=48949/38617, merge=0/12, ticks=582078/1213470, in_queue=1752702, util=11.99%

ssd-test.fio:

ssd array:

NOTE: filesystem BLOCK_SIZE=”512”

seq-read: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
rand-read: (g=1): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
seq-write: (g=2): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
rand-write: (g=3): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
fio-3.18
Starting 4 processes
seq-read: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [_(3),w(1)][97.1%][w=144MiB/s][w=36.0k IOPS][eta 00m:07s] 
seq-read: (groupid=0, jobs=1): err= 0: pid=493789: Wed Apr 15 09:36:28 2020
 read: IOPS=47.3k, BW=185MiB/s (194MB/s)(10.0GiB/55432msec)
 slat (nsec): min=1734, max=7910.4k, avg=6939.26, stdev=6107.82
 clat (usec): min=18, max=18890, avg=77.00, stdev=58.03
 lat (usec): min=39, max=18894, avg=84.06, stdev=57.46
 clat percentiles (usec):
 | 1.00th=[ 46], 5.00th=[ 55], 10.00th=[ 57], 20.00th=[ 60],
 | 30.00th=[ 61], 40.00th=[ 63], 50.00th=[ 73], 60.00th=[ 87],
 | 70.00th=[ 91], 80.00th=[ 93], 90.00th=[ 95], 95.00th=[ 96],
 | 99.00th=[ 176], 99.50th=[ 212], 99.90th=[ 330], 99.95th=[ 379],
 | 99.99th=[ 2245]
 bw ( KiB/s): min=165624, max=198600, per=100.00%, avg=189438.94, stdev=5253.06, samples=110
 iops : min=41406, max=49650, avg=47359.73, stdev=1313.24, samples=110
 lat (usec) : 20=0.01%, 50=2.23%, 100=95.33%, 250=2.25%, 500=0.16%
 lat (usec) : 750=0.01%, 1000=0.01%
 lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
 cpu : usr=5.21%, sys=33.67%, ctx=1870277, majf=0, minf=12
 IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=2621440,0,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=4
rand-read: (groupid=1, jobs=1): err= 0: pid=493941: Wed Apr 15 09:36:28 2020
 read: IOPS=21.4k, BW=83.7MiB/s (87.8MB/s)(5021MiB/60001msec)
 slat (nsec): min=1993, max=698901, avg=5083.29, stdev=2238.84
 clat (usec): min=10, max=5835, avg=180.83, stdev=107.98
 lat (usec): min=64, max=5841, avg=186.03, stdev=107.39
 clat percentiles (usec):
 | 1.00th=[ 85], 5.00th=[ 87], 10.00th=[ 89], 20.00th=[ 92],
 | 30.00th=[ 96], 40.00th=[ 113], 50.00th=[ 133], 60.00th=[ 178],
 | 70.00th=[ 249], 80.00th=[ 265], 90.00th=[ 343], 95.00th=[ 363],
 | 99.00th=[ 457], 99.50th=[ 519], 99.90th=[ 676], 99.95th=[ 881],
 | 99.99th=[ 2343]
 bw ( KiB/s): min=75832, max=90616, per=100.00%, avg=85828.91, stdev=2325.89, samples=119
 iops : min=18958, max=22654, avg=21457.24, stdev=581.46, samples=119
 lat (usec) : 20=0.01%, 100=35.33%, 250=34.90%, 500=29.15%, 750=0.54%
 lat (usec) : 1000=0.03%
 lat (msec) : 2=0.03%, 4=0.01%, 10=0.01%
 cpu : usr=2.97%, sys=13.06%, ctx=1000451, majf=0, minf=11
 IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=1285489,0,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=4
seq-write: (groupid=2, jobs=1): err= 0: pid=494174: Wed Apr 15 09:36:28 2020
 write: IOPS=41.5k, BW=162MiB/s (170MB/s)(9724MiB/60001msec); 0 zone resets
 slat (nsec): min=1302, max=505648, avg=3797.37, stdev=1015.23
 clat (usec): min=9, max=85537, avg=91.98, stdev=119.30
 lat (usec): min=42, max=85543, avg=95.89, stdev=119.29
 clat percentiles (usec):
 | 1.00th=[ 58], 5.00th=[ 72], 10.00th=[ 76], 20.00th=[ 78],
 | 30.00th=[ 79], 40.00th=[ 80], 50.00th=[ 87], 60.00th=[ 96],
 | 70.00th=[ 100], 80.00th=[ 104], 90.00th=[ 108], 95.00th=[ 111],
 | 99.00th=[ 204], 99.50th=[ 239], 99.90th=[ 379], 99.95th=[ 553],
 | 99.99th=[ 1237]
 bw ( KiB/s): min=139312, max=174680, per=100.00%, avg=166184.06, stdev=5209.64, samples=119
 iops : min=34828, max=43670, avg=41546.03, stdev=1302.41, samples=119
 lat (usec) : 10=0.01%, 20=0.01%, 50=0.23%, 100=69.39%, 250=29.98%
 lat (usec) : 500=0.34%, 750=0.03%, 1000=0.01%
 lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
 lat (msec) : 100=0.01%
 cpu : usr=5.76%, sys=19.09%, ctx=1840937, majf=0, minf=13
 IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=0,2489393,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=4
rand-write: (groupid=3, jobs=1): err= 0: pid=494628: Wed Apr 15 09:36:28 2020
 write: IOPS=39.1k, BW=153MiB/s (160MB/s)(9155MiB/60001msec); 0 zone resets
 slat (nsec): min=1373, max=1069.0k, avg=3878.94, stdev=1213.13
 clat (usec): min=10, max=647688, avg=97.77, stdev=994.90
 lat (usec): min=41, max=647745, avg=101.76, stdev=994.94
 clat percentiles (usec):
 | 1.00th=[ 45], 5.00th=[ 47], 10.00th=[ 49], 20.00th=[ 52],
 | 30.00th=[ 56], 40.00th=[ 60], 50.00th=[ 70], 60.00th=[ 86],
 | 70.00th=[ 115], 80.00th=[ 129], 90.00th=[ 161], 95.00th=[ 180],
 | 99.00th=[ 281], 99.50th=[ 363], 99.90th=[ 1860], 99.95th=[ 2900],
 | 99.99th=[ 7308]
 bw ( KiB/s): min=58584, max=185840, per=100.00%, avg=157849.32, stdev=21750.44, samples=118
 iops : min=14646, max=46460, avg=39462.38, stdev=5437.63, samples=118
 lat (usec) : 20=0.01%, 50=15.62%, 100=49.24%, 250=33.66%, 500=1.20%
 lat (usec) : 750=0.09%, 1000=0.04%
 lat (msec) : 2=0.06%, 4=0.06%, 10=0.03%, 20=0.01%, 50=0.01%
 lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
 cpu : usr=6.20%, sys=18.91%, ctx=2049357, majf=0, minf=13
 IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=0,2343796,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
 READ: bw=185MiB/s (194MB/s), 185MiB/s-185MiB/s (194MB/s-194MB/s), io=10.0GiB (10.7GB), run=55432-55432msec

Run status group 1 (all jobs):
 READ: bw=83.7MiB/s (87.8MB/s), 83.7MiB/s-83.7MiB/s (87.8MB/s-87.8MB/s), io=5021MiB (5265MB), run=60001-60001msec

Run status group 2 (all jobs):
 WRITE: bw=162MiB/s (170MB/s), 162MiB/s-162MiB/s (170MB/s-170MB/s), io=9724MiB (10.2GB), run=60001-60001msec

Run status group 3 (all jobs):
 WRITE: bw=153MiB/s (160MB/s), 153MiB/s-153MiB/s (160MB/s-160MB/s), io=9155MiB (9600MB), run=60001-60001msec

Disk stats (read/write):
 dm-3: ios=3920958/4851512, merge=0/0, ticks=445073/1856767, in_queue=2302040, util=99.14%, aggrios=3920967/4854312, aggrmerge=0/0, aggrticks=442211/1947582, aggrin_queue=2389993, aggrutil=99.23%
 dm-0: ios=3920967/4854312, merge=0/0, ticks=442211/1947582, in_queue=2389993, util=99.23%, aggrios=3923477/4857404, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
 md128: ios=3923477/4857404, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=1638145/2116515, aggrmerge=323580/311709, aggrticks=181144/169117, aggrin_queue=11661, aggrutil=87.44%
 sdd: ios=1962398/2416430, merge=341/10761, ticks=141224/141199, in_queue=13519, util=80.41%
 sda: ios=1313892/1816601, merge=646819/612657, ticks=221064/197036, in_queue=9804, util=87.44%

fio-seq-RW.fio:

ssd array:

NOTE: filesystem BLOCK_SIZE=”512”

file1: (g=0): rw=rw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=16
...
fio-3.18
Starting 4 processes
file1: Laying out IO file (1 file / 10240MiB)
Jobs: 4 (f=4): [M(4)][100.0%][eta 00m:00s] 
file1: (groupid=0, jobs=1): err= 0: pid=42891: Thu Apr 16 22:22:20 2020
 read: IOPS=1814, BW=454MiB/s (476MB/s)(400GiB/902747msec)
 slat (usec): min=13, max=2717.6k, avg=184.83, stdev=5610.28
 clat (usec): min=10, max=9105.8k, avg=5027.03, stdev=103760.79
 lat (usec): min=501, max=9106.0k, avg=5212.05, stdev=103913.63
 clat percentiles (usec):
 | 1.00th=[ 1106], 5.00th=[ 1418], 10.00th=[ 1614],
 | 20.00th=[ 1876], 30.00th=[ 2114], 40.00th=[ 2343],
 | 50.00th=[ 2573], 60.00th=[ 2900], 70.00th=[ 3359],
 | 80.00th=[ 4113], 90.00th=[ 5473], 95.00th=[ 6915],
 | 99.00th=[ 11207], 99.50th=[ 14222], 99.90th=[ 25297],
 | 99.95th=[ 40109], 99.99th=[5737808]
 bw ( KiB/s): min= 1024, max=1362944, per=36.93%, avg=686251.06, stdev=248132.23, samples=1222
 iops : min= 4, max= 5324, avg=2680.66, stdev=969.26, samples=1222
 write: IOPS=1209, BW=302MiB/s (317MB/s)(267GiB/902747msec); 0 zone resets
 slat (usec): min=26, max=3839.1k, avg=279.70, stdev=3882.85
 clat (usec): min=481, max=9105.6k, avg=4859.48, stdev=100268.29
 lat (usec): min=540, max=9105.6k, avg=5139.40, stdev=100344.72
 clat percentiles (usec):
 | 1.00th=[ 1090], 5.00th=[ 1401], 10.00th=[ 1598],
 | 20.00th=[ 1844], 30.00th=[ 2073], 40.00th=[ 2278],
 | 50.00th=[ 2540], 60.00th=[ 2835], 70.00th=[ 3261],
 | 80.00th=[ 3949], 90.00th=[ 5342], 95.00th=[ 6718],
 | 99.00th=[ 10945], 99.50th=[ 13829], 99.90th=[ 24511],
 | 99.95th=[ 34341], 99.99th=[5268046]
 bw ( KiB/s): min= 505, max=936960, per=36.89%, avg=456976.76, stdev=166758.28, samples=1223
 iops : min= 1, max= 3660, avg=1785.06, stdev=651.40, samples=1223
 lat (usec) : 20=0.01%, 500=0.01%, 750=0.13%, 1000=0.38%
 lat (msec) : 2=25.57%, 4=53.44%, 10=19.08%, 20=1.21%, 50=0.14%
 lat (msec) : 100=0.01%, 250=0.01%, 750=0.01%, 1000=0.01%, 2000=0.01%
 lat (msec) : >=2000=0.03%
 cpu : usr=1.10%, sys=51.43%, ctx=14324594, majf=0, minf=21
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=1638401,1091922,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=42892: Thu Apr 16 22:22:20 2020
 read: IOPS=1814, BW=454MiB/s (476MB/s)(400GiB/902746msec)
 slat (usec): min=11, max=3839.3k, avg=184.51, stdev=6310.31
 clat (usec): min=8, max=8994.8k, avg=5096.20, stdev=105688.68
 lat (usec): min=570, max=8994.9k, avg=5280.91, stdev=105880.31
 clat percentiles (usec):
 | 1.00th=[ 1123], 5.00th=[ 1434], 10.00th=[ 1614],
 | 20.00th=[ 1876], 30.00th=[ 2114], 40.00th=[ 2311],
 | 50.00th=[ 2573], 60.00th=[ 2900], 70.00th=[ 3326],
 | 80.00th=[ 4080], 90.00th=[ 5473], 95.00th=[ 6849],
 | 99.00th=[ 11207], 99.50th=[ 14222], 99.90th=[ 25297],
 | 99.95th=[ 40109], 99.99th=[5536482]
 bw ( KiB/s): min= 512, max=1246720, per=37.08%, avg=689074.87, stdev=243035.95, samples=1217
 iops : min= 2, max= 4870, avg=2691.69, stdev=949.36, samples=1217
 write: IOPS=1209, BW=302MiB/s (317MB/s)(267GiB/902746msec); 0 zone resets
 slat (usec): min=24, max=1319.9k, avg=278.72, stdev=1498.99
 clat (usec): min=536, max=8994.7k, avg=4753.99, stdev=98651.40
 lat (usec): min=599, max=8994.8k, avg=5032.93, stdev=98661.83
 clat percentiles (usec):
 | 1.00th=[ 1123], 5.00th=[ 1418], 10.00th=[ 1598],
 | 20.00th=[ 1860], 30.00th=[ 2073], 40.00th=[ 2278],
 | 50.00th=[ 2540], 60.00th=[ 2835], 70.00th=[ 3228],
 | 80.00th=[ 3916], 90.00th=[ 5276], 95.00th=[ 6718],
 | 99.00th=[ 10945], 99.50th=[ 13829], 99.90th=[ 24511],
 | 99.95th=[ 33162], 99.99th=[5469373]
 bw ( KiB/s): min= 505, max=813056, per=37.09%, avg=459402.89, stdev=162443.85, samples=1217
 iops : min= 1, max= 3176, avg=1794.53, stdev=634.55, samples=1217
 lat (usec) : 10=0.01%, 750=0.04%, 1000=0.39%
 lat (msec) : 2=25.53%, 4=53.81%, 10=18.85%, 20=1.17%, 50=0.16%
 lat (msec) : 100=0.01%, 250=0.01%, 750=0.01%, 1000=0.01%, 2000=0.01%
 lat (msec) : >=2000=0.03%
 cpu : usr=1.09%, sys=51.99%, ctx=14033638, majf=0, minf=21
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=1638401,1092319,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=42893: Thu Apr 16 22:22:20 2020
 read: IOPS=1814, BW=454MiB/s (476MB/s)(400GiB/902745msec)
 slat (usec): min=13, max=2572.8k, avg=178.75, stdev=4798.26
 clat (usec): min=9, max=8953.0k, avg=4957.37, stdev=100759.70
 lat (usec): min=424, max=8953.0k, avg=5136.31, stdev=100874.82
 clat percentiles (usec):
 | 1.00th=[ 1106], 5.00th=[ 1418], 10.00th=[ 1614],
 | 20.00th=[ 1876], 30.00th=[ 2089], 40.00th=[ 2311],
 | 50.00th=[ 2573], 60.00th=[ 2900], 70.00th=[ 3359],
 | 80.00th=[ 4113], 90.00th=[ 5473], 95.00th=[ 6849],
 | 99.00th=[ 11207], 99.50th=[ 14222], 99.90th=[ 25560],
 | 99.95th=[ 38536], 99.99th=[5402264]
 bw ( KiB/s): min= 512, max=1317376, per=36.90%, avg=685699.53, stdev=248396.06, samples=1223
 iops : min= 2, max= 5146, avg=2678.50, stdev=970.29, samples=1223
 write: IOPS=1211, BW=303MiB/s (317MB/s)(267GiB/902745msec); 0 zone resets
 slat (usec): min=22, max=3839.1k, avg=287.84, stdev=5264.77
 clat (usec): min=415, max=8952.5k, avg=4957.54, stdev=103987.50
 lat (usec): min=483, max=8952.6k, avg=5245.60, stdev=104124.31
 clat percentiles (usec):
 | 1.00th=[ 1090], 5.00th=[ 1401], 10.00th=[ 1598],
 | 20.00th=[ 1844], 30.00th=[ 2073], 40.00th=[ 2278],
 | 50.00th=[ 2540], 60.00th=[ 2835], 70.00th=[ 3261],
 | 80.00th=[ 3949], 90.00th=[ 5342], 95.00th=[ 6718],
 | 99.00th=[ 10814], 99.50th=[ 13698], 99.90th=[ 25035],
 | 99.95th=[ 36963], 99.99th=[5469373]
 bw ( KiB/s): min= 512, max=848896, per=36.97%, avg=457954.36, stdev=164997.43, samples=1222
 iops : min= 2, max= 3316, avg=1788.87, stdev=644.52, samples=1222
 lat (usec) : 10=0.01%, 500=0.01%, 750=0.22%, 1000=0.39%
 lat (msec) : 2=25.59%, 4=53.40%, 10=19.04%, 20=1.15%, 50=0.16%
 lat (msec) : 100=0.01%, 250=0.01%, 750=0.01%, 1000=0.01%, 2000=0.01%
 lat (msec) : >=2000=0.03%
 cpu : usr=1.09%, sys=51.41%, ctx=14245479, majf=0, minf=21
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=1638401,1093352,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=42894: Thu Apr 16 22:22:20 2020
 read: IOPS=1814, BW=454MiB/s (476MB/s)(400GiB/902746msec)
 slat (usec): min=12, max=3839.2k, avg=203.77, stdev=6385.25
 clat (usec): min=9, max=9027.8k, avg=5012.82, stdev=103448.02
 lat (usec): min=501, max=9027.0k, avg=5216.78, stdev=103648.60
 clat percentiles (usec):
 | 1.00th=[ 996], 5.00th=[ 1418], 10.00th=[ 1614],
 | 20.00th=[ 1893], 30.00th=[ 2114], 40.00th=[ 2376],
 | 50.00th=[ 2638], 60.00th=[ 2933], 70.00th=[ 3392],
 | 80.00th=[ 4113], 90.00th=[ 5473], 95.00th=[ 6849],
 | 99.00th=[ 11207], 99.50th=[ 14222], 99.90th=[ 25560],
 | 99.95th=[ 39060], 99.99th=[5402264]
 bw ( KiB/s): min= 512, max=1742848, per=36.90%, avg=685701.93, stdev=248867.68, samples=1223
 iops : min= 2, max= 6808, avg=2678.51, stdev=972.14, samples=1223
 write: IOPS=1208, BW=302MiB/s (317MB/s)(266GiB/902746msec); 0 zone resets
 slat (usec): min=20, max=953698, avg=253.24, stdev=1060.72
 clat (usec): min=462, max=9027.6k, avg=4886.78, stdev=100816.16
 lat (usec): min=506, max=9027.9k, avg=5140.24, stdev=100821.59
 clat percentiles (usec):
 | 1.00th=[ 988], 5.00th=[ 1385], 10.00th=[ 1598],
 | 20.00th=[ 1860], 30.00th=[ 2089], 40.00th=[ 2311],
 | 50.00th=[ 2573], 60.00th=[ 2868], 70.00th=[ 3294],
 | 80.00th=[ 3949], 90.00th=[ 5276], 95.00th=[ 6652],
 | 99.00th=[ 10814], 99.50th=[ 13829], 99.90th=[ 24511],
 | 99.95th=[ 36963], 99.99th=[5268046]
 bw ( KiB/s): min= 505, max=1180160, per=36.85%, avg=456424.24, stdev=166213.07, samples=1223
 iops : min= 1, max= 4610, avg=1782.90, stdev=649.27, samples=1223
 lat (usec) : 10=0.01%, 500=0.01%, 750=0.59%, 1000=0.45%
 lat (msec) : 2=23.97%, 4=54.39%, 10=19.25%, 20=1.17%, 50=0.14%
 lat (msec) : 100=0.01%, 250=0.01%, 750=0.01%, 1000=0.01%, 2000=0.01%
 lat (msec) : >=2000=0.03%
 cpu : usr=1.12%, sys=52.21%, ctx=14216879, majf=0, minf=23
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=1638401,1090585,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
 READ: bw=1815MiB/s (1903MB/s), 454MiB/s-454MiB/s (476MB/s-476MB/s), io=1600GiB (1718GB), run=902745-902747msec
 WRITE: bw=1210MiB/s (1268MB/s), 302MiB/s-303MiB/s (317MB/s-317MB/s), io=1066GiB (1145GB), run=902745-902747msec

Disk stats (read/write):
 dm-3: ios=3862919/26552569, merge=0/0, ticks=12686654/1044094996, in_queue=1056784029, util=76.85%, aggrios=3862919/26620232, aggrmerge=0/0, aggrticks=12688443/1281930380, aggrin_queue=1294621201, aggrutil=79.59%
 dm-0: ios=3862919/26620232, merge=0/0, ticks=12688443/1281930380, in_queue=1294621201, util=79.59%, aggrios=4339895/26927913, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
 md128: ios=4339895/26927913, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=2048871/356161, aggrmerge=101818/13108098, aggrticks=6068873/720584, aggrin_queue=5628966, aggrutil=70.02%
 sdd: ios=2105652/369456, merge=57410/13091934, ticks=3621523/872452, in_queue=3320778, util=67.30%
 sda: ios=1992090/342866, merge=146227/13124263, ticks=8516224/568716, in_queue=7937154, util=70.02%

btrfs filesystem tests:

fio-rand-RW.fio:

ssd array:

file1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.18
Starting 4 processes
file1: Laying out IO file (1 file / 10240MiB)
Jobs: 4 (f=4): [m(4)][100.0%][r=174MiB/s,w=117MiB/s][r=44.5k,w=29.9k IOPS][eta 00m:00s] 
file1: (groupid=0, jobs=1): err= 0: pid=488554: Wed Apr 15 09:15:29 2020
 read: IOPS=6112, BW=23.9MiB/s (25.0MB/s)(20.0GiB/900103msec)
 slat (nsec): min=571, max=1947.5M, avg=67475.51, stdev=2205656.85
 clat (usec): min=22, max=3821.9k, avg=1499.71, stdev=30697.76
 lat (usec): min=23, max=3821.9k, avg=1567.27, stdev=30783.88
 clat percentiles (usec):
 | 1.00th=[ 46], 5.00th=[ 61], 10.00th=[ 75],
 | 20.00th=[ 99], 30.00th=[ 127], 40.00th=[ 172],
 | 50.00th=[ 245], 60.00th=[ 396], 70.00th=[ 627],
 | 80.00th=[ 963], 90.00th=[ 1516], 95.00th=[ 2278],
 | 99.00th=[ 5473], 99.50th=[ 7308], 99.90th=[ 37487],
 | 99.95th=[ 968885], 99.99th=[1367344]
 bw ( KiB/s): min= 16, max=151848, per=35.00%, avg=34250.51, stdev=31158.93, samples=1285
 iops : min= 4, max=37962, avg=8562.63, stdev=7789.73, samples=1285
 write: IOPS=4075, BW=15.9MiB/s (16.7MB/s)(13.0GiB/900103msec); 0 zone resets
 slat (usec): min=2, max=1557.9k, avg=140.13, stdev=11835.59
 clat (nsec): min=1853, max=3821.8M, avg=1432078.82, stdev=29974401.79
 lat (usec): min=26, max=3821.8k, avg=1572.28, stdev=32228.56
 clat percentiles (usec):
 | 1.00th=[ 46], 5.00th=[ 61], 10.00th=[ 75],
 | 20.00th=[ 100], 30.00th=[ 128], 40.00th=[ 172],
 | 50.00th=[ 245], 60.00th=[ 396], 70.00th=[ 627],
 | 80.00th=[ 963], 90.00th=[ 1516], 95.00th=[ 2278],
 | 99.00th=[ 5473], 99.50th=[ 7242], 99.90th=[ 29754],
 | 99.95th=[ 918553], 99.99th=[1350566]
 bw ( KiB/s): min= 24, max=101080, per=35.05%, avg=22854.64, stdev=20765.75, samples=1284
 iops : min= 6, max=25270, avg=5713.66, stdev=5191.44, samples=1284
 lat (usec) : 2=0.01%, 50=1.89%, 100=18.49%, 250=30.01%, 500=14.57%
 lat (usec) : 750=9.24%, 1000=6.71%
 lat (msec) : 2=12.98%, 4=4.01%, 10=1.84%, 20=0.13%, 50=0.03%
 lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.02%
 lat (msec) : 2000=0.05%, >=2000=0.01%
 cpu : usr=0.95%, sys=7.46%, ctx=1312493, majf=0, minf=22
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=5501661,3668295,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=488555: Wed Apr 15 09:15:29 2020
 read: IOPS=6112, BW=23.9MiB/s (25.0MB/s)(20.0GiB/900104msec)
 slat (nsec): min=561, max=1465.5M, avg=67391.92, stdev=2092061.09
 clat (usec): min=21, max=3823.5k, avg=1510.35, stdev=31053.86
 lat (usec): min=22, max=3823.5k, avg=1577.82, stdev=31126.44
 clat percentiles (usec):
 | 1.00th=[ 46], 5.00th=[ 61], 10.00th=[ 74],
 | 20.00th=[ 98], 30.00th=[ 126], 40.00th=[ 169],
 | 50.00th=[ 245], 60.00th=[ 396], 70.00th=[ 627],
 | 80.00th=[ 963], 90.00th=[ 1516], 95.00th=[ 2278],
 | 99.00th=[ 5538], 99.50th=[ 7373], 99.90th=[ 36439],
 | 99.95th=[ 977273], 99.99th=[1367344]
 bw ( KiB/s): min= 16, max=148960, per=34.95%, avg=34200.94, stdev=31141.73, samples=1287
 iops : min= 4, max=37240, avg=8550.23, stdev=7785.43, samples=1287
 write: IOPS=4076, BW=15.9MiB/s (16.7MB/s)(13.0GiB/900104msec); 0 zone resets
 slat (usec): min=2, max=1947.7k, avg=140.52, stdev=11867.77
 clat (usec): min=4, max=3823.5k, avg=1415.34, stdev=29048.97
 lat (usec): min=27, max=3823.5k, avg=1555.92, stdev=31388.25
 clat percentiles (usec):
 | 1.00th=[ 45], 5.00th=[ 61], 10.00th=[ 75],
 | 20.00th=[ 99], 30.00th=[ 128], 40.00th=[ 172],
 | 50.00th=[ 247], 60.00th=[ 396], 70.00th=[ 627],
 | 80.00th=[ 963], 90.00th=[ 1516], 95.00th=[ 2311],
 | 99.00th=[ 5538], 99.50th=[ 7373], 99.90th=[ 29230],
 | 99.95th=[ 910164], 99.99th=[1350566]
 bw ( KiB/s): min= 8, max=100416, per=35.00%, avg=22827.97, stdev=20768.03, samples=1286
 iops : min= 2, max=25104, avg=5706.99, stdev=5192.01, samples=1286
 lat (usec) : 10=0.01%, 50=1.97%, 100=18.55%, 250=29.86%, 500=14.57%
 lat (usec) : 750=9.25%, 1000=6.73%
 lat (msec) : 2=12.94%, 4=4.03%, 10=1.84%, 20=0.14%, 50=0.03%
 lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.02%
 lat (msec) : 2000=0.05%, >=2000=0.01%
 cpu : usr=0.97%, sys=7.30%, ctx=1311783, majf=0, minf=23
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=5502258,3669714,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=488556: Wed Apr 15 09:15:29 2020
 read: IOPS=6116, BW=23.9MiB/s (25.1MB/s)(21.0GiB/900102msec)
 slat (nsec): min=541, max=1557.7M, avg=68480.10, stdev=2406561.16
 clat (usec): min=21, max=3848.9k, avg=1499.25, stdev=30707.96
 lat (usec): min=22, max=3848.9k, avg=1567.81, stdev=30804.46
 clat percentiles (usec):
 | 1.00th=[ 45], 5.00th=[ 61], 10.00th=[ 74],
 | 20.00th=[ 98], 30.00th=[ 126], 40.00th=[ 169],
 | 50.00th=[ 245], 60.00th=[ 396], 70.00th=[ 627],
 | 80.00th=[ 963], 90.00th=[ 1516], 95.00th=[ 2278],
 | 99.00th=[ 5538], 99.50th=[ 7373], 99.90th=[ 38011],
 | 99.95th=[ 968885], 99.99th=[1367344]
 bw ( KiB/s): min= 32, max=153216, per=34.98%, avg=34222.23, stdev=31198.49, samples=1287
 iops : min= 8, max=38304, avg=8555.56, stdev=7799.62, samples=1287
 write: IOPS=4075, BW=15.9MiB/s (16.7MB/s)(13.0GiB/900102msec); 0 zone resets
 slat (usec): min=2, max=1947.5k, avg=138.80, stdev=11780.17
 clat (usec): min=8, max=3848.7k, avg=1431.08, stdev=29641.13
 lat (usec): min=26, max=3848.8k, avg=1569.95, stdev=31905.49
 clat percentiles (usec):
 | 1.00th=[ 45], 5.00th=[ 61], 10.00th=[ 75],
 | 20.00th=[ 99], 30.00th=[ 127], 40.00th=[ 172],
 | 50.00th=[ 245], 60.00th=[ 396], 70.00th=[ 627],
 | 80.00th=[ 963], 90.00th=[ 1516], 95.00th=[ 2278],
 | 99.00th=[ 5538], 99.50th=[ 7308], 99.90th=[ 28705],
 | 99.95th=[ 926942], 99.99th=[1350566]
 bw ( KiB/s): min= 32, max=101216, per=34.96%, avg=22801.94, stdev=20801.58, samples=1287
 iops : min= 8, max=25304, avg=5700.48, stdev=5200.39, samples=1287
 lat (usec) : 10=0.01%, 50=2.01%, 100=18.63%, 250=29.77%, 500=14.52%
 lat (usec) : 750=9.25%, 1000=6.72%
 lat (msec) : 2=13.00%, 4=3.98%, 10=1.85%, 20=0.14%, 50=0.03%
 lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.02%
 lat (msec) : 2000=0.05%, >=2000=0.01%
 cpu : usr=0.95%, sys=7.35%, ctx=1308964, majf=0, minf=23
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=5505680,3668390,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=488557: Wed Apr 15 09:15:29 2020
 read: IOPS=6120, BW=23.9MiB/s (25.1MB/s)(21.0GiB/900102msec)
 slat (nsec): min=531, max=1520.5M, avg=68615.13, stdev=2477276.87
 clat (usec): min=22, max=3761.5k, avg=1504.46, stdev=30983.41
 lat (usec): min=23, max=3761.5k, avg=1573.15, stdev=31084.50
 clat percentiles (usec):
 | 1.00th=[ 45], 5.00th=[ 61], 10.00th=[ 74],
 | 20.00th=[ 98], 30.00th=[ 126], 40.00th=[ 169],
 | 50.00th=[ 245], 60.00th=[ 396], 70.00th=[ 627],
 | 80.00th=[ 963], 90.00th=[ 1516], 95.00th=[ 2278],
 | 99.00th=[ 5538], 99.50th=[ 7373], 99.90th=[ 37487],
 | 99.95th=[ 968885], 99.99th=[1367344]
 bw ( KiB/s): min= 40, max=152888, per=35.08%, avg=34321.05, stdev=31303.41, samples=1284
 iops : min= 10, max=38222, avg=8580.26, stdev=7825.85, samples=1284
 write: IOPS=4075, BW=15.9MiB/s (16.7MB/s)(13.0GiB/900102msec); 0 zone resets
 slat (nsec): min=1953, max=1947.7M, avg=137847.28, stdev=11731710.62
 clat (usec): min=3, max=3761.5k, avg=1421.93, stdev=29600.36
 lat (usec): min=26, max=3761.5k, avg=1559.84, stdev=31849.23
 clat percentiles (usec):
 | 1.00th=[ 45], 5.00th=[ 61], 10.00th=[ 74],
 | 20.00th=[ 99], 30.00th=[ 127], 40.00th=[ 172],
 | 50.00th=[ 245], 60.00th=[ 396], 70.00th=[ 627],
 | 80.00th=[ 963], 90.00th=[ 1516], 95.00th=[ 2278],
 | 99.00th=[ 5473], 99.50th=[ 7308], 99.90th=[ 28181],
 | 99.95th=[ 910164], 99.99th=[1350566]
 bw ( KiB/s): min= 16, max=100128, per=35.05%, avg=22856.83, stdev=20864.27, samples=1284
 iops : min= 4, max=25032, avg=5714.21, stdev=5216.07, samples=1284
 lat (usec) : 4=0.01%, 50=2.04%, 100=18.75%, 250=29.65%, 500=14.54%
 lat (usec) : 750=9.23%, 1000=6.73%
 lat (msec) : 2=12.96%, 4=4.00%, 10=1.85%, 20=0.13%, 50=0.03%
 lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.02%
 lat (msec) : 2000=0.05%, >=2000=0.01%
 cpu : usr=0.94%, sys=7.63%, ctx=1325019, majf=0, minf=22
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=5508710,3668641,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
 READ: bw=95.6MiB/s (100MB/s), 23.9MiB/s-23.9MiB/s (25.0MB/s-25.1MB/s), io=83.0GiB (90.2GB), run=900102-900104msec
 WRITE: bw=63.7MiB/s (66.8MB/s), 15.9MiB/s-15.9MiB/s (16.7MB/s-16.7MB/s), io=55.0GiB (60.1GB), run=900102-900104msec

NOTE: deleting 10gb test file tooke like one minute or more on btrfs.

ssd-test.fio:

ssd array:

seq-read: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
rand-read: (g=1): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
seq-write: (g=2): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
rand-write: (g=3): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
fio-3.18
Starting 4 processes
seq-read: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [_(3),w(1)][95.6%][w=145MiB/s][w=37.2k IOPS][eta 00m:11s] 
seq-read: (groupid=0, jobs=1): err= 0: pid=491399: Wed Apr 15 09:23:03 2020
 read: IOPS=47.3k, BW=185MiB/s (194MB/s)(10.0GiB/55385msec)
 slat (usec): min=2, max=340, avg=10.66, stdev= 4.89
 clat (usec): min=21, max=15591, avg=73.22, stdev=41.49
 lat (usec): min=43, max=15630, avg=84.02, stdev=39.98
 clat percentiles (usec):
 | 1.00th=[ 44], 5.00th=[ 49], 10.00th=[ 52], 20.00th=[ 54],
 | 30.00th=[ 55], 40.00th=[ 59], 50.00th=[ 74], 60.00th=[ 84],
 | 70.00th=[ 86], 80.00th=[ 89], 90.00th=[ 93], 95.00th=[ 96],
 | 99.00th=[ 178], 99.50th=[ 212], 99.90th=[ 326], 99.95th=[ 375],
 | 99.99th=[ 1795]
 bw ( KiB/s): min=171528, max=198360, per=100.00%, avg=189631.37, stdev=5075.54, samples=110
 iops : min=42882, max=49590, avg=47407.82, stdev=1268.90, samples=110
 lat (usec) : 50=6.37%, 100=90.79%, 250=2.65%, 500=0.16%, 750=0.01%
 lat (usec) : 1000=0.01%
 lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
 cpu : usr=5.30%, sys=49.00%, ctx=971079, majf=0, minf=15
 IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=2621440,0,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=4
rand-read: (groupid=1, jobs=1): err= 0: pid=491498: Wed Apr 15 09:23:03 2020
 read: IOPS=21.3k, BW=83.1MiB/s (87.2MB/s)(4989MiB/60001msec)
 slat (usec): min=3, max=767, avg= 8.89, stdev= 2.77
 clat (usec): min=14, max=30037, avg=178.14, stdev=121.64
 lat (usec): min=72, max=30045, avg=187.17, stdev=121.07
 clat percentiles (usec):
 | 1.00th=[ 88], 5.00th=[ 91], 10.00th=[ 93], 20.00th=[ 96],
 | 30.00th=[ 103], 40.00th=[ 120], 50.00th=[ 137], 60.00th=[ 172],
 | 70.00th=[ 231], 80.00th=[ 258], 90.00th=[ 326], 95.00th=[ 351],
 | 99.00th=[ 445], 99.50th=[ 498], 99.90th=[ 660], 99.95th=[ 840],
 | 99.99th=[ 2442]
 bw ( KiB/s): min=76015, max=88776, per=100.00%, avg=85235.61, stdev=2436.96, samples=119
 iops : min=19003, max=22194, avg=21308.89, stdev=609.28, samples=119
 lat (usec) : 20=0.01%, 50=0.01%, 100=27.27%, 250=47.88%, 500=24.35%
 lat (usec) : 750=0.43%, 1000=0.03%
 lat (msec) : 2=0.02%, 4=0.02%, 10=0.01%, 20=0.01%, 50=0.01%
 cpu : usr=3.30%, sys=20.80%, ctx=1025418, majf=0, minf=14
 IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=1277191,0,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=4
seq-write: (groupid=2, jobs=1): err= 0: pid=491822: Wed Apr 15 09:23:03 2020
 write: IOPS=39.7k, BW=155MiB/s (162MB/s)(9295MiB/60001msec); 0 zone resets
 slat (usec): min=4, max=296, avg=11.64, stdev= 2.44
 clat (usec): min=27, max=102728, avg=88.55, stdev=222.10
 lat (usec): min=48, max=102736, avg=100.32, stdev=222.12
 clat percentiles (usec):
 | 1.00th=[ 57], 5.00th=[ 64], 10.00th=[ 67], 20.00th=[ 70],
 | 30.00th=[ 71], 40.00th=[ 73], 50.00th=[ 84], 60.00th=[ 96],
 | 70.00th=[ 99], 80.00th=[ 102], 90.00th=[ 105], 95.00th=[ 110],
 | 99.00th=[ 204], 99.50th=[ 239], 99.90th=[ 379], 99.95th=[ 586],
 | 99.99th=[ 2900]
 bw ( KiB/s): min=76064, max=170440, per=100.00%, avg=158811.59, stdev=9809.61, samples=119
 iops : min=19016, max=42610, avg=39702.91, stdev=2452.40, samples=119
 lat (usec) : 50=0.23%, 100=74.24%, 250=25.14%, 500=0.33%, 750=0.02%
 lat (usec) : 1000=0.01%
 lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
 lat (msec) : 100=0.01%, 250=0.01%
 cpu : usr=6.06%, sys=48.71%, ctx=1685703, majf=0, minf=14
 IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=0,2379610,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=4
rand-write: (groupid=3, jobs=1): err= 0: pid=492071: Wed Apr 15 09:23:03 2020
 write: IOPS=36.8k, BW=144MiB/s (151MB/s)(8626MiB/60002msec); 0 zone resets
 slat (usec): min=5, max=160439, avg=14.43, stdev=232.09
 clat (usec): min=26, max=160506, avg=93.44, stdev=486.35
 lat (usec): min=46, max=179366, avg=108.00, stdev=541.71
 clat percentiles (usec):
 | 1.00th=[ 46], 5.00th=[ 50], 10.00th=[ 54], 20.00th=[ 61],
 | 30.00th=[ 67], 40.00th=[ 70], 50.00th=[ 74], 60.00th=[ 94],
 | 70.00th=[ 98], 80.00th=[ 103], 90.00th=[ 125], 95.00th=[ 163],
 | 99.00th=[ 249], 99.50th=[ 302], 99.90th=[ 1045], 99.95th=[ 2024],
 | 99.99th=[12387]
 bw ( KiB/s): min=34888, max=163664, per=100.00%, avg=147452.25, stdev=19099.00, samples=119
 iops : min= 8722, max=40916, avg=36863.08, stdev=4774.77, samples=119
 lat (usec) : 50=5.95%, 100=68.34%, 250=24.72%, 500=0.84%, 750=0.04%
 lat (usec) : 1000=0.02%
 lat (msec) : 2=0.05%, 4=0.03%, 10=0.01%, 20=0.01%, 50=0.01%
 lat (msec) : 100=0.01%, 250=0.01%
 cpu : usr=6.21%, sys=51.58%, ctx=1384080, majf=0, minf=12
 IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=0,2208289,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
 READ: bw=185MiB/s (194MB/s), 185MiB/s-185MiB/s (194MB/s-194MB/s), io=10.0GiB (10.7GB), run=55385-55385msec

Run status group 1 (all jobs):
 READ: bw=83.1MiB/s (87.2MB/s), 83.1MiB/s-83.1MiB/s (87.2MB/s-87.2MB/s), io=4989MiB (5231MB), run=60001-60001msec

Run status group 2 (all jobs):
 WRITE: bw=155MiB/s (162MB/s), 155MiB/s-155MiB/s (162MB/s-162MB/s), io=9295MiB (9747MB), run=60001-60001msec

Run status group 3 (all jobs):
 WRITE: bw=144MiB/s (151MB/s), 144MiB/s-144MiB/s (151MB/s-151MB/s), io=8626MiB (9045MB), run=60002-60002msec

fio-seq-RW.fio:

ssd array:

file1: (g=0): rw=rw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=16
...
fio-3.18
Starting 4 processes
file1: Laying out IO file (1 file / 10240MiB)
Jobs: 3 (f=3): [M(3),_(1)][100.0%][r=2813KiB/s,w=1278KiB/s][r=10,w=4 IOPS][eta 00m:00s] 
file1: (groupid=0, jobs=1): err= 0: pid=46321: Thu Apr 16 23:02:16 2020
 read: IOPS=1908, BW=477MiB/s (500MB/s)(420GiB/901390msec)
 slat (usec): min=10, max=3292.9k, avg=204.17, stdev=6557.26
 clat (usec): min=390, max=9498.3k, avg=4851.96, stdev=113854.72
 lat (usec): min=418, max=9498.4k, avg=5056.25, stdev=114423.64
 clat percentiles (usec):
 | 1.00th=[ 685], 5.00th=[ 857], 10.00th=[ 955],
 | 20.00th=[ 1074], 30.00th=[ 1188], 40.00th=[ 1287],
 | 50.00th=[ 1418], 60.00th=[ 1631], 70.00th=[ 2180],
 | 80.00th=[ 3032], 90.00th=[ 4752], 95.00th=[ 7832],
 | 99.00th=[ 17171], 99.50th=[ 20055], 99.90th=[ 59507],
 | 99.95th=[ 918553], 99.99th=[5737808]
 bw ( KiB/s): min= 512, max=2663424, per=41.53%, avg=811549.54, stdev=498872.78, samples=1085
 iops : min= 2, max=10404, avg=3170.11, stdev=1948.72, samples=1085
 write: IOPS=1272, BW=318MiB/s (333MB/s)(280GiB/901390msec); 0 zone resets
 slat (usec): min=17, max=7010.7k, avg=246.03, stdev=18576.42
 clat (usec): min=7, max=9498.2k, avg=4513.35, stdev=108187.37
 lat (usec): min=433, max=9498.3k, avg=4759.52, stdev=109821.97
 clat percentiles (usec):
 | 1.00th=[ 668], 5.00th=[ 848], 10.00th=[ 938],
 | 20.00th=[ 1074], 30.00th=[ 1172], 40.00th=[ 1270],
 | 50.00th=[ 1385], 60.00th=[ 1565], 70.00th=[ 1991],
 | 80.00th=[ 2835], 90.00th=[ 4490], 95.00th=[ 7504],
 | 99.00th=[ 16909], 99.50th=[ 19792], 99.90th=[ 50594],
 | 99.95th=[ 350225], 99.99th=[5603591]
 bw ( KiB/s): min= 512, max=1752064, per=41.84%, avg=544907.86, stdev=330632.85, samples=1077
 iops : min= 2, max= 6844, avg=2128.55, stdev=1291.53, samples=1077
 lat (usec) : 10=0.01%, 500=0.02%, 750=2.13%, 1000=11.57%
 lat (msec) : 2=54.87%, 4=18.91%, 10=9.22%, 20=2.79%, 50=0.38%
 lat (msec) : 100=0.03%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.04%
 cpu : usr=0.89%, sys=52.00%, ctx=11350080, majf=0, minf=19
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=1720322,1146597,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=46322: Thu Apr 16 23:02:16 2020
 read: IOPS=1908, BW=477MiB/s (500MB/s)(420GiB/901390msec)
 slat (usec): min=10, max=3293.5k, avg=200.37, stdev=6405.68
 clat (usec): min=323, max=9485.2k, avg=4939.08, stdev=116035.25
 lat (usec): min=344, max=9485.2k, avg=5139.56, stdev=116579.45
 clat percentiles (usec):
 | 1.00th=[ 652], 5.00th=[ 865], 10.00th=[ 963],
 | 20.00th=[ 1090], 30.00th=[ 1188], 40.00th=[ 1303],
 | 50.00th=[ 1434], 60.00th=[ 1631], 70.00th=[ 2147],
 | 80.00th=[ 2999], 90.00th=[ 4621], 95.00th=[ 7635],
 | 99.00th=[ 17171], 99.50th=[ 20055], 99.90th=[ 58983],
 | 99.95th=[1216349], 99.99th=[5737808]
 bw ( KiB/s): min= 512, max=2759680, per=42.15%, avg=823612.52, stdev=494622.50, samples=1069
 iops : min= 2, max=10780, avg=3217.23, stdev=1932.12, samples=1069
 write: IOPS=1272, BW=318MiB/s (334MB/s)(280GiB/901390msec); 0 zone resets
 slat (usec): min=17, max=7010.9k, avg=246.65, stdev=18480.59
 clat (usec): min=8, max=9484.8k, avg=4382.28, stdev=105404.90
 lat (usec): min=356, max=9484.9k, avg=4629.06, stdev=107053.86
 clat percentiles (usec):
 | 1.00th=[ 644], 5.00th=[ 848], 10.00th=[ 955],
 | 20.00th=[ 1074], 30.00th=[ 1172], 40.00th=[ 1270],
 | 50.00th=[ 1401], 60.00th=[ 1565], 70.00th=[ 1958],
 | 80.00th=[ 2802], 90.00th=[ 4424], 95.00th=[ 7439],
 | 99.00th=[ 16909], 99.50th=[ 19530], 99.90th=[ 45876],
 | 99.95th=[ 299893], 99.99th=[5670700]
 bw ( KiB/s): min= 512, max=1901056, per=42.32%, avg=551189.49, stdev=329238.16, samples=1065
 iops : min= 2, max= 7426, avg=2153.08, stdev=1286.09, samples=1065
 lat (usec) : 10=0.01%, 500=0.12%, 750=2.12%, 1000=10.72%
 lat (msec) : 2=56.17%, 4=18.62%, 10=9.04%, 20=2.73%, 50=0.38%
 lat (msec) : 100=0.03%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.04%
 cpu : usr=0.87%, sys=52.48%, ctx=11428364, majf=0, minf=21
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=1720153,1146882,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=46323: Thu Apr 16 23:02:16 2020
 read: IOPS=1907, BW=477MiB/s (500MB/s)(420GiB/901390msec)
 slat (usec): min=10, max=3293.0k, avg=199.17, stdev=5943.95
 clat (usec): min=426, max=9500.2k, avg=4796.43, stdev=112188.75
 lat (usec): min=453, max=9500.2k, avg=4995.71, stdev=112701.60
 clat percentiles (usec):
 | 1.00th=[ 685], 5.00th=[ 865], 10.00th=[ 955],
 | 20.00th=[ 1090], 30.00th=[ 1188], 40.00th=[ 1287],
 | 50.00th=[ 1418], 60.00th=[ 1631], 70.00th=[ 2147],
 | 80.00th=[ 3032], 90.00th=[ 4686], 95.00th=[ 7767],
 | 99.00th=[ 16909], 99.50th=[ 19792], 99.90th=[ 56361],
 | 99.95th=[ 759170], 99.99th=[5737808]
 bw ( KiB/s): min= 512, max=2656256, per=41.86%, avg=817886.86, stdev=490288.18, samples=1076
 iops : min= 2, max=10376, avg=3194.87, stdev=1915.19, samples=1076
 write: IOPS=1272, BW=318MiB/s (334MB/s)(280GiB/901390msec); 0 zone resets
 slat (usec): min=19, max=7010.9k, avg=252.68, stdev=18856.09
 clat (usec): min=8, max=9500.1k, avg=4599.49, stdev=112152.04
 lat (usec): min=466, max=9500.2k, avg=4852.30, stdev=113800.28
 clat percentiles (usec):
 | 1.00th=[ 676], 5.00th=[ 848], 10.00th=[ 947],
 | 20.00th=[ 1074], 30.00th=[ 1172], 40.00th=[ 1270],
 | 50.00th=[ 1385], 60.00th=[ 1565], 70.00th=[ 1975],
 | 80.00th=[ 2835], 90.00th=[ 4490], 95.00th=[ 7439],
 | 99.00th=[ 16909], 99.50th=[ 19792], 99.90th=[ 50594],
 | 99.95th=[ 434111], 99.99th=[5737808]
 bw ( KiB/s): min= 512, max=1813504, per=42.28%, avg=550665.49, stdev=324374.68, samples=1066
 iops : min= 2, max= 7084, avg=2151.03, stdev=1267.09, samples=1066
 lat (usec) : 10=0.01%, 500=0.01%, 750=1.98%, 1000=11.29%
 lat (msec) : 2=55.58%, 4=18.75%, 10=9.13%, 20=2.79%, 50=0.37%
 lat (msec) : 100=0.03%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.04%
 cpu : usr=0.87%, sys=52.21%, ctx=11309634, majf=0, minf=22
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=1719377,1146884,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=46324: Thu Apr 16 23:02:16 2020
 read: IOPS=1910, BW=478MiB/s (501MB/s)(420GiB/900276msec)
 slat (usec): min=10, max=3292.0k, avg=208.68, stdev=7003.33
 clat (usec): min=7, max=9491.9k, avg=4823.59, stdev=112823.41
 lat (usec): min=456, max=9491.0k, avg=5032.38, stdev=113451.79
 clat percentiles (usec):
 | 1.00th=[ 668], 5.00th=[ 857], 10.00th=[ 955],
 | 20.00th=[ 1090], 30.00th=[ 1188], 40.00th=[ 1303],
 | 50.00th=[ 1434], 60.00th=[ 1680], 70.00th=[ 2245],
 | 80.00th=[ 3064], 90.00th=[ 4752], 95.00th=[ 7832],
 | 99.00th=[ 17171], 99.50th=[ 20055], 99.90th=[ 58983],
 | 99.95th=[ 692061], 99.99th=[5872026]
 bw ( KiB/s): min= 512, max=2155008, per=41.53%, avg=811541.40, stdev=486199.95, samples=1085
 iops : min= 2, max= 8418, avg=3170.08, stdev=1899.22, samples=1085
 write: IOPS=1272, BW=318MiB/s (333MB/s)(280GiB/900276msec); 0 zone resets
 slat (usec): min=19, max=7010.8k, avg=246.79, stdev=18222.38
 clat (usec): min=406, max=9491.6k, avg=4546.45, stdev=108194.79
 lat (usec): min=462, max=9491.7k, avg=4793.38, stdev=109812.04
 clat percentiles (usec):
 | 1.00th=[ 660], 5.00th=[ 848], 10.00th=[ 947],
 | 20.00th=[ 1074], 30.00th=[ 1172], 40.00th=[ 1287],
 | 50.00th=[ 1401], 60.00th=[ 1598], 70.00th=[ 2057],
 | 80.00th=[ 2900], 90.00th=[ 4555], 95.00th=[ 7504],
 | 99.00th=[ 16909], 99.50th=[ 19792], 99.90th=[ 52691],
 | 99.95th=[ 358613], 99.99th=[5737808]
 bw ( KiB/s): min= 512, max=1421824, per=41.56%, avg=541277.40, stdev=323462.89, samples=1083
 iops : min= 2, max= 5554, avg=2114.36, stdev=1263.52, samples=1083
 lat (usec) : 10=0.01%, 500=0.02%, 750=2.22%, 1000=10.98%
 lat (msec) : 2=54.39%, 4=19.69%, 10=9.43%, 20=2.78%, 50=0.38%
 lat (msec) : 100=0.03%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.04%
 cpu : usr=0.87%, sys=51.56%, ctx=10983262, majf=0, minf=22
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=1720324,1145310,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
 READ: bw=1908MiB/s (2001MB/s), 477MiB/s-478MiB/s (500MB/s-501MB/s), io=1680GiB (1804GB), run=900276-901390msec
 WRITE: bw=1272MiB/s (1334MB/s), 318MiB/s-318MiB/s (333MB/s-334MB/s), io=1120GiB (1202GB), run=900276-901390msec

ext4 filesystem tests:

fio-rand-RW.fio:

ssd array:

file1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.18
Starting 4 processes
file1: Laying out IO file (1 file / 10240MiB)
Jobs: 4 (f=4): [m(4)][100.0%][eta 00m:00s] 
file1: (groupid=0, jobs=1): err= 0: pid=4187: Thu Apr 16 20:47:28 2020
 read: IOPS=26.8k, BW=105MiB/s (110MB/s)(92.2GiB/902250msec)
 slat (nsec): min=521, max=22937M, avg=24337.34, stdev=12851101.99
 clat (usec): min=7, max=25704k, avg=336.13, stdev=58337.67
 lat (usec): min=22, max=25704k, avg=360.52, stdev=59985.21
 clat percentiles (usec):
 | 1.00th=[ 26], 5.00th=[ 29], 10.00th=[ 31], 20.00th=[ 35],
 | 30.00th=[ 38], 40.00th=[ 42], 50.00th=[ 46], 60.00th=[ 52],
 | 70.00th=[ 72], 80.00th=[ 178], 90.00th=[ 351], 95.00th=[ 570],
 | 99.00th=[ 1385], 99.50th=[ 1860], 99.90th=[ 2868], 99.95th=[ 3359],
 | 99.99th=[ 5735]
 bw ( KiB/s): min= 8, max=850920, per=54.82%, avg=234830.66, stdev=237231.04, samples=823
 iops : min= 2, max=212730, avg=58707.67, stdev=59307.77, samples=823
 write: IOPS=17.8k, BW=69.7MiB/s (73.1MB/s)(61.4GiB/902250msec); 0 zone resets
 slat (nsec): min=751, max=19321M, avg=12303.51, stdev=10578386.16
 clat (usec): min=20, max=25704k, avg=336.73, stdev=60134.65
 lat (usec): min=22, max=25704k, avg=349.08, stdev=61123.80
 clat percentiles (usec):
 | 1.00th=[ 26], 5.00th=[ 29], 10.00th=[ 31], 20.00th=[ 35],
 | 30.00th=[ 39], 40.00th=[ 42], 50.00th=[ 46], 60.00th=[ 52],
 | 70.00th=[ 72], 80.00th=[ 178], 90.00th=[ 351], 95.00th=[ 570],
 | 99.00th=[ 1385], 99.50th=[ 1860], 99.90th=[ 2868], 99.95th=[ 3392],
 | 99.99th=[ 5800]
 bw ( KiB/s): min= 8, max=564768, per=54.54%, avg=155777.15, stdev=158137.96, samples=827
 iops : min= 2, max=141192, avg=38944.29, stdev=39534.49, samples=827
 lat (usec) : 10=0.01%, 50=57.99%, 100=12.64%, 250=14.02%, 500=9.16%
 lat (usec) : 750=2.97%, 1000=1.30%
 lat (msec) : 2=1.50%, 4=0.37%, 10=0.03%, 20=0.01%, 50=0.01%
 lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.01%
 cpu : usr=3.06%, sys=18.16%, ctx=6486230, majf=0, minf=24
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=24158675,16103789,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=4188: Thu Apr 16 20:47:28 2020
 read: IOPS=26.8k, BW=105MiB/s (110MB/s)(92.1GiB/902250msec)
 slat (nsec): min=511, max=19324M, avg=24262.33, stdev=11823631.98
 clat (usec): min=7, max=25701k, avg=331.05, stdev=56646.11
 lat (usec): min=22, max=25701k, avg=355.36, stdev=58246.64
 clat percentiles (usec):
 | 1.00th=[ 26], 5.00th=[ 29], 10.00th=[ 31], 20.00th=[ 35],
 | 30.00th=[ 38], 40.00th=[ 42], 50.00th=[ 46], 60.00th=[ 52],
 | 70.00th=[ 72], 80.00th=[ 178], 90.00th=[ 355], 95.00th=[ 570],
 | 99.00th=[ 1385], 99.50th=[ 1860], 99.90th=[ 2835], 99.95th=[ 3326],
 | 99.99th=[ 5735]
 bw ( KiB/s): min= 8, max=845840, per=54.28%, avg=232522.77, stdev=236563.19, samples=831
 iops : min= 2, max=211460, avg=58130.68, stdev=59140.78, samples=831
 write: IOPS=17.9k, BW=69.7MiB/s (73.1MB/s)(61.5GiB/902250msec); 0 zone resets
 slat (nsec): min=751, max=18028M, avg=12467.36, stdev=10905218.24
 clat (usec): min=20, max=25701k, avg=344.32, stdev=59589.00
 lat (usec): min=23, max=25701k, avg=356.83, stdev=60581.24
 clat percentiles (usec):
 | 1.00th=[ 26], 5.00th=[ 29], 10.00th=[ 31], 20.00th=[ 35],
 | 30.00th=[ 39], 40.00th=[ 42], 50.00th=[ 46], 60.00th=[ 52],
 | 70.00th=[ 72], 80.00th=[ 178], 90.00th=[ 351], 95.00th=[ 570],
 | 99.00th=[ 1385], 99.50th=[ 1860], 99.90th=[ 2868], 99.95th=[ 3359],
 | 99.99th=[ 5735]
 bw ( KiB/s): min= 8, max=564408, per=54.43%, avg=155455.95, stdev=157786.99, samples=829
 iops : min= 2, max=141104, avg=38863.98, stdev=39446.74, samples=829
 lat (usec) : 10=0.01%, 50=57.83%, 100=12.83%, 250=13.96%, 500=9.19%
 lat (usec) : 750=2.98%, 1000=1.29%
 lat (msec) : 2=1.50%, 4=0.37%, 10=0.03%, 20=0.01%, 50=0.01%
 lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.01%
 cpu : usr=3.09%, sys=18.11%, ctx=6347526, majf=0, minf=25
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=24153782,16109439,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=4189: Thu Apr 16 20:47:28 2020
 read: IOPS=26.8k, BW=105MiB/s (110MB/s)(92.1GiB/902250msec)
 slat (nsec): min=511, max=19321M, avg=22579.21, stdev=10573514.31
 clat (usec): min=7, max=25718k, avg=333.62, stdev=57753.62
 lat (usec): min=22, max=25718k, avg=356.25, stdev=59144.99
 clat percentiles (usec):
 | 1.00th=[ 26], 5.00th=[ 29], 10.00th=[ 31], 20.00th=[ 35],
 | 30.00th=[ 38], 40.00th=[ 42], 50.00th=[ 46], 60.00th=[ 51],
 | 70.00th=[ 71], 80.00th=[ 178], 90.00th=[ 351], 95.00th=[ 578],
 | 99.00th=[ 1385], 99.50th=[ 1860], 99.90th=[ 2868], 99.95th=[ 3392],
 | 99.99th=[ 5866]
 bw ( KiB/s): min= 8, max=850224, per=54.47%, avg=233345.08, stdev=237852.27, samples=828
 iops : min= 2, max=212556, avg=58336.27, stdev=59463.07, samples=828
 write: IOPS=17.9k, BW=69.7MiB/s (73.1MB/s)(61.4GiB/902250msec); 0 zone resets
 slat (nsec): min=742, max=18031M, avg=14999.88, stdev=12288734.93
 clat (usec): min=21, max=25718k, avg=340.53, stdev=58896.28
 lat (usec): min=22, max=25718k, avg=355.58, stdev=60247.68
 clat percentiles (usec):
 | 1.00th=[ 26], 5.00th=[ 29], 10.00th=[ 31], 20.00th=[ 35],
 | 30.00th=[ 39], 40.00th=[ 42], 50.00th=[ 46], 60.00th=[ 52],
 | 70.00th=[ 71], 80.00th=[ 178], 90.00th=[ 351], 95.00th=[ 570],
 | 99.00th=[ 1385], 99.50th=[ 1860], 99.90th=[ 2868], 99.95th=[ 3359],
 | 99.99th=[ 5800]
 bw ( KiB/s): min= 8, max=569056, per=54.36%, avg=155259.24, stdev=158673.45, samples=830
 iops : min= 2, max=142264, avg=38814.81, stdev=39668.36, samples=830
 lat (usec) : 10=0.01%, 50=58.15%, 100=12.55%, 250=13.97%, 500=9.15%
 lat (usec) : 750=2.97%, 1000=1.30%
 lat (msec) : 2=1.50%, 4=0.38%, 10=0.03%, 20=0.01%, 50=0.01%
 lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.01%
 cpu : usr=3.08%, sys=18.10%, ctx=6328985, majf=0, minf=24
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=24151688,16108475,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=4190: Thu Apr 16 20:47:28 2020
 read: IOPS=26.8k, BW=105MiB/s (110MB/s)(92.2GiB/902250msec)
 slat (nsec): min=511, max=20650M, avg=23949.60, stdev=11590584.38
 clat (usec): min=8, max=25693k, avg=338.63, stdev=58590.52
 lat (usec): min=21, max=25693k, avg=362.64, stdev=60059.25
 clat percentiles (usec):
 | 1.00th=[ 26], 5.00th=[ 29], 10.00th=[ 31], 20.00th=[ 35],
 | 30.00th=[ 38], 40.00th=[ 42], 50.00th=[ 46], 60.00th=[ 51],
 | 70.00th=[ 71], 80.00th=[ 178], 90.00th=[ 351], 95.00th=[ 570],
 | 99.00th=[ 1385], 99.50th=[ 1860], 99.90th=[ 2868], 99.95th=[ 3359],
 | 99.99th=[ 5735]
 bw ( KiB/s): min= 8, max=855224, per=54.49%, avg=233415.42, stdev=237744.80, samples=828
 iops : min= 2, max=213806, avg=58353.86, stdev=59436.20, samples=828
 write: IOPS=17.8k, BW=69.7MiB/s (73.1MB/s)(61.4GiB/902250msec); 0 zone resets
 slat (nsec): min=741, max=19324M, avg=12901.57, stdev=11251956.87
 clat (usec): min=20, max=25692k, avg=332.98, stdev=58218.59
 lat (usec): min=22, max=25692k, avg=345.93, stdev=59474.78
 clat percentiles (usec):
 | 1.00th=[ 26], 5.00th=[ 29], 10.00th=[ 31], 20.00th=[ 35],
 | 30.00th=[ 38], 40.00th=[ 42], 50.00th=[ 46], 60.00th=[ 52],
 | 70.00th=[ 72], 80.00th=[ 178], 90.00th=[ 351], 95.00th=[ 570],
 | 99.00th=[ 1385], 99.50th=[ 1860], 99.90th=[ 2868], 99.95th=[ 3392],
 | 99.99th=[ 5866]
 bw ( KiB/s): min= 8, max=572496, per=54.67%, avg=156138.08, stdev=158425.86, samples=825
 iops : min= 2, max=143124, avg=39034.52, stdev=39606.47, samples=825
 lat (usec) : 10=0.01%, 50=58.27%, 100=12.37%, 250=14.02%, 500=9.17%
 lat (usec) : 750=2.98%, 1000=1.29%
 lat (msec) : 2=1.50%, 4=0.38%, 10=0.03%, 20=0.01%, 50=0.01%
 lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.01%
 cpu : usr=2.99%, sys=18.21%, ctx=6419172, majf=0, minf=23
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=24158978,16102061,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
 READ: bw=418MiB/s (439MB/s), 105MiB/s-105MiB/s (110MB/s-110MB/s), io=369GiB (396GB), run=902250-902250msec
 WRITE: bw=279MiB/s (292MB/s), 69.7MiB/s-69.7MiB/s (73.1MB/s-73.1MB/s), io=246GiB (264GB), run=902250-902250msec

Disk stats (read/write):
 dm-4: ios=5993954/14497630, merge=0/0, ticks=1634288/1037657607, in_queue=1039291895, util=97.78%, aggrios=5998052/14574477, aggrmerge=0/0, aggrticks=2123171/1462952472, aggrin_queue=1465076704, aggrutil=97.99%
 dm-0: ios=5998052/14574477, merge=0/0, ticks=2123171/1462952472, in_queue=1465076704, util=97.99%, aggrios=5998766/14562384, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
 md128: ios=5998766/14562384, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=2999228/5929551, aggrmerge=147/1352125, aggrticks=524887/1056422, aggrin_queue=47791, aggrutil=97.33%
 sdd: ios=2997639/5937272, merge=110/1346600, ticks=387298/1487985, in_queue=87156, util=93.40%
 sda: ios=3000817/5921831, merge=185/1357651, ticks=662477/624859, in_queue=8426, util=97.33%

ssd-test.fio:

ssd array:

seq-read: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
rand-read: (g=1): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
seq-write: (g=2): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
rand-write: (g=3): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
fio-3.18
Starting 4 processes
seq-read: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [_(3),w(1)][99.2%][w=169MiB/s][w=43.3k IOPS][eta 00m:02s] 
seq-read: (groupid=0, jobs=1): err= 0: pid=25992: Thu Apr 16 20:59:39 2020
 read: IOPS=47.4k, BW=185MiB/s (194MB/s)(10.0GiB/55281msec)
 slat (nsec): min=1683, max=301896, avg=6826.06, stdev=3563.61
 clat (usec): min=16, max=7686, avg=76.89, stdev=41.30
 lat (usec): min=39, max=7693, avg=83.83, stdev=40.16
 clat percentiles (usec):
 | 1.00th=[ 46], 5.00th=[ 55], 10.00th=[ 57], 20.00th=[ 59],
 | 30.00th=[ 61], 40.00th=[ 63], 50.00th=[ 75], 60.00th=[ 87],
 | 70.00th=[ 90], 80.00th=[ 92], 90.00th=[ 95], 95.00th=[ 97],
 | 99.00th=[ 182], 99.50th=[ 206], 99.90th=[ 289], 99.95th=[ 355],
 | 99.99th=[ 1582]
 bw ( KiB/s): min=162408, max=199288, per=100.00%, avg=189975.23, stdev=5129.09, samples=110
 iops : min=40602, max=49822, avg=47493.80, stdev=1282.28, samples=110
 lat (usec) : 20=0.01%, 50=2.25%, 100=94.94%, 250=2.66%, 500=0.13%
 lat (usec) : 750=0.01%, 1000=0.01%
 lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%
 cpu : usr=5.25%, sys=33.18%, ctx=1879641, majf=0, minf=12
 IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=2621440,0,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=4
rand-read: (groupid=1, jobs=1): err= 0: pid=26072: Thu Apr 16 20:59:39 2020
 read: IOPS=21.1k, BW=82.6MiB/s (86.6MB/s)(4953MiB/60001msec)
 slat (usec): min=2, max=339, avg= 5.07, stdev= 1.96
 clat (usec): min=23, max=28131, avg=183.42, stdev=131.33
 lat (usec): min=64, max=28139, avg=188.62, stdev=130.84
 clat percentiles (usec):
 | 1.00th=[ 85], 5.00th=[ 88], 10.00th=[ 90], 20.00th=[ 93],
 | 30.00th=[ 96], 40.00th=[ 112], 50.00th=[ 135], 60.00th=[ 180],
 | 70.00th=[ 253], 80.00th=[ 269], 90.00th=[ 347], 95.00th=[ 367],
 | 99.00th=[ 461], 99.50th=[ 519], 99.90th=[ 660], 99.95th=[ 865],
 | 99.99th=[ 2474]
 bw ( KiB/s): min=74800, max=89048, per=100.00%, avg=84649.53, stdev=2408.84, samples=119
 iops : min=18700, max=22262, avg=21162.38, stdev=602.20, samples=119
 lat (usec) : 50=0.01%, 100=35.20%, 250=33.60%, 500=30.60%, 750=0.53%
 lat (usec) : 1000=0.02%
 lat (msec) : 2=0.02%, 4=0.02%, 10=0.01%, 20=0.01%, 50=0.01%
 cpu : usr=3.11%, sys=12.85%, ctx=978765, majf=0, minf=12
 IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=1268052,0,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=4
seq-write: (groupid=2, jobs=1): err= 0: pid=26145: Thu Apr 16 20:59:39 2020
 write: IOPS=42.8k, BW=167MiB/s (175MB/s)(9.79GiB/60003msec); 0 zone resets
 slat (nsec): min=1543, max=10207k, avg=4043.19, stdev=6419.44
 clat (usec): min=35, max=82562, avg=88.87, stdev=139.06
 lat (usec): min=44, max=82565, avg=93.03, stdev=139.21
 clat percentiles (usec):
 | 1.00th=[ 55], 5.00th=[ 74], 10.00th=[ 76], 20.00th=[ 78],
 | 30.00th=[ 79], 40.00th=[ 80], 50.00th=[ 84], 60.00th=[ 95],
 | 70.00th=[ 96], 80.00th=[ 97], 90.00th=[ 100], 95.00th=[ 102],
 | 99.00th=[ 178], 99.50th=[ 212], 99.90th=[ 302], 99.95th=[ 359],
 | 99.99th=[ 832]
 bw ( KiB/s): min=145760, max=176824, per=100.00%, avg=171344.39, stdev=4452.16, samples=119
 iops : min=36440, max=44206, avg=42836.09, stdev=1113.04, samples=119
 lat (usec) : 50=0.31%, 100=91.40%, 250=8.09%, 500=0.17%, 750=0.01%
 lat (usec) : 1000=0.01%
 lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
 lat (msec) : 100=0.01%
 cpu : usr=5.71%, sys=20.67%, ctx=1905968, majf=0, minf=14
 IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=0,2566351,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=4
rand-write: (groupid=3, jobs=1): err= 0: pid=26269: Thu Apr 16 20:59:39 2020
 write: IOPS=42.1k, BW=165MiB/s (173MB/s)(9871MiB/60001msec); 0 zone resets
 slat (nsec): min=1653, max=154630, avg=4258.08, stdev=690.88
 clat (usec): min=14, max=69914, avg=89.99, stdev=138.70
 lat (usec): min=39, max=69919, avg=94.37, stdev=138.71
 clat percentiles (usec):
 | 1.00th=[ 44], 5.00th=[ 46], 10.00th=[ 47], 20.00th=[ 50],
 | 30.00th=[ 53], 40.00th=[ 58], 50.00th=[ 67], 60.00th=[ 85],
 | 70.00th=[ 115], 80.00th=[ 127], 90.00th=[ 161], 95.00th=[ 174],
 | 99.00th=[ 260], 99.50th=[ 302], 99.90th=[ 420], 99.95th=[ 515],
 | 99.99th=[ 2311]
 bw ( KiB/s): min=136816, max=189304, per=100.00%, avg=168739.86, stdev=11097.33, samples=119
 iops : min=34204, max=47326, avg=42184.97, stdev=2774.35, samples=119
 lat (usec) : 20=0.01%, 50=22.63%, 100=42.21%, 250=33.94%, 500=1.16%
 lat (usec) : 750=0.03%, 1000=0.01%
 lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
 lat (msec) : 100=0.01%
 cpu : usr=6.15%, sys=22.35%, ctx=2177658, majf=0, minf=12
 IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=0,2526915,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
 READ: bw=185MiB/s (194MB/s), 185MiB/s-185MiB/s (194MB/s-194MB/s), io=10.0GiB (10.7GB), run=55281-55281msec

Run status group 1 (all jobs):
 READ: bw=82.6MiB/s (86.6MB/s), 82.6MiB/s-82.6MiB/s (86.6MB/s-86.6MB/s), io=4953MiB (5194MB), run=60001-60001msec

Run status group 2 (all jobs):
 WRITE: bw=167MiB/s (175MB/s), 167MiB/s-167MiB/s (175MB/s-175MB/s), io=9.79GiB (10.5GB), run=60003-60003msec

Run status group 3 (all jobs):
 WRITE: bw=165MiB/s (173MB/s), 165MiB/s-165MiB/s (173MB/s-173MB/s), io=9871MiB (10.3GB), run=60001-60001msec

Disk stats (read/write):
 dm-4: ios=3889492/5087822, merge=0/0, ticks=435735/446565, in_queue=882300, util=99.40%, aggrios=3890468/5113230, aggrmerge=0/0, aggrticks=433045/1974911, aggrin_queue=2408127, aggrutil=99.55%
 dm-0: ios=3890468/5113230, merge=0/0, ticks=433045/1974911, in_queue=2408127, util=99.55%, aggrios=3890470/5113458, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
 md128: ios=3890470/5113458, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=1621878/2239322, aggrmerge=323357/317244, aggrticks=175205/174238, aggrin_queue=1788, aggrutil=87.64%
 sdd: ios=1945635/2553523, merge=123/2765, ticks=134406/145088, in_queue=3176, util=81.04%
 sda: ios=1298121/1925121, merge=646591/631723, ticks=216004/203388, in_queue=400, util=87.64%

fio-seq-RW.fio:

ssd array:

file1: (g=0): rw=rw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=16
...
fio-3.18
Starting 4 processes
file1: Laying out IO file (1 file / 10240MiB)
Jobs: 4 (f=4): [M(4)][100.0%][r=4876MiB/s,w=3196MiB/s][r=19.5k,w=12.8k IOPS][eta 00m:00s]
file1: (groupid=0, jobs=1): err= 0: pid=31629: Thu Apr 16 22:02:18 2020
 read: IOPS=1755, BW=439MiB/s (460MB/s)(386GiB/900001msec)
 slat (usec): min=10, max=6001.5k, avg=243.42, stdev=13604.69
 clat (usec): min=239, max=10819k, avg=5185.92, stdev=109885.26
 lat (usec): min=281, max=10819k, avg=5429.46, stdev=111139.70
 clat percentiles (usec):
 | 1.00th=[ 889], 5.00th=[ 1221], 10.00th=[ 1418],
 | 20.00th=[ 1680], 30.00th=[ 1893], 40.00th=[ 2114],
 | 50.00th=[ 2376], 60.00th=[ 2737], 70.00th=[ 3261],
 | 80.00th=[ 4015], 90.00th=[ 5276], 95.00th=[ 6783],
 | 99.00th=[ 10945], 99.50th=[ 13304], 99.90th=[ 36963],
 | 99.95th=[ 792724], 99.99th=[7281312]
 bw ( KiB/s): min= 512, max=1683968, per=38.57%, avg=693499.08, stdev=320473.55, samples=1165
 iops : min= 2, max= 6578, avg=2708.97, stdev=1251.85, samples=1165
 write: IOPS=1170, BW=293MiB/s (307MB/s)(257GiB/900001msec); 0 zone resets
 slat (usec): min=24, max=4547.7k, avg=272.29, stdev=10969.43
 clat (nsec): min=1943, max=10818M, avg=5038702.95, stdev=110500052.27
 lat (usec): min=238, max=10818k, avg=5311.12, stdev=111106.80
 clat percentiles (usec):
 | 1.00th=[ 865], 5.00th=[ 1188], 10.00th=[ 1385],
 | 20.00th=[ 1631], 30.00th=[ 1844], 40.00th=[ 2040],
 | 50.00th=[ 2278], 60.00th=[ 2540], 70.00th=[ 2999],
 | 80.00th=[ 3720], 90.00th=[ 4948], 95.00th=[ 6521],
 | 99.00th=[ 10683], 99.50th=[ 13042], 99.90th=[ 36963],
 | 99.95th=[ 650118], 99.99th=[7281312]
 bw ( KiB/s): min= 512, max=1103872, per=38.91%, avg=466225.56, stdev=210492.39, samples=1155
 iops : min= 2, max= 4312, avg=1821.18, stdev=822.24, samples=1155
 lat (usec) : 2=0.01%, 250=0.01%, 500=0.01%, 750=0.36%, 1000=1.58%
 lat (msec) : 2=34.04%, 4=45.13%, 10=17.43%, 20=1.27%, 50=0.10%
 lat (msec) : 100=0.02%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.04%
 cpu : usr=0.82%, sys=56.44%, ctx=7701664, majf=0, minf=21
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=1580114,1053170,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=31630: Thu Apr 16 22:02:18 2020
 read: IOPS=1755, BW=439MiB/s (460MB/s)(386GiB/900001msec)
 slat (usec): min=11, max=6001.3k, avg=236.29, stdev=12943.27
 clat (usec): min=418, max=9439.3k, avg=5315.54, stdev=111617.09
 lat (usec): min=435, max=9439.3k, avg=5551.96, stdev=112738.52
 clat percentiles (usec):
 | 1.00th=[ 906], 5.00th=[ 1237], 10.00th=[ 1434],
 | 20.00th=[ 1713], 30.00th=[ 1926], 40.00th=[ 2147],
 | 50.00th=[ 2409], 60.00th=[ 2737], 70.00th=[ 3228],
 | 80.00th=[ 3982], 90.00th=[ 5211], 95.00th=[ 6783],
 | 99.00th=[ 10945], 99.50th=[ 13173], 99.90th=[ 41157],
 | 99.95th=[1132463], 99.99th=[7214203]
 bw ( KiB/s): min= 512, max=1685504, per=38.81%, avg=697806.97, stdev=310838.43, samples=1158
 iops : min= 2, max= 6584, avg=2725.80, stdev=1214.22, samples=1158
 write: IOPS=1170, BW=293MiB/s (307MB/s)(257GiB/900001msec); 0 zone resets
 slat (usec): min=27, max=4547.6k, avg=283.11, stdev=11936.31
 clat (nsec): min=1353, max=9438.9M, avg=4841527.39, stdev=107041842.36
 lat (usec): min=278, max=9438.0k, avg=5124.77, stdev=107745.97
 clat percentiles (usec):
 | 1.00th=[ 889], 5.00th=[ 1221], 10.00th=[ 1401],
 | 20.00th=[ 1663], 30.00th=[ 1860], 40.00th=[ 2057],
 | 50.00th=[ 2278], 60.00th=[ 2573], 70.00th=[ 2999],
 | 80.00th=[ 3687], 90.00th=[ 4883], 95.00th=[ 6456],
 | 99.00th=[ 10552], 99.50th=[ 12780], 99.90th=[ 34341],
 | 99.95th=[ 170918], 99.99th=[7214203]
 bw ( KiB/s): min= 512, max=1175552, per=38.88%, avg=465971.38, stdev=207316.31, samples=1156
 iops : min= 2, max= 4592, avg=1820.19, stdev=809.83, samples=1156
 lat (usec) : 2=0.01%, 500=0.01%, 750=0.31%, 1000=1.47%
 lat (msec) : 2=33.17%, 4=46.48%, 10=17.13%, 20=1.25%, 50=0.11%
 lat (msec) : 100=0.02%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.04%
 cpu : usr=0.81%, sys=57.17%, ctx=7700112, majf=0, minf=24
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=1580380,1053470,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=31631: Thu Apr 16 22:02:18 2020
 read: IOPS=1755, BW=439MiB/s (460MB/s)(386GiB/900001msec)
 slat (usec): min=10, max=6001.6k, avg=231.26, stdev=12228.15
 clat (nsec): min=1082, max=9452.4M, avg=5205366.88, stdev=109985047.72
 lat (usec): min=36, max=9452.4k, avg=5436.74, stdev=111086.16
 clat percentiles (usec):
 | 1.00th=[ 914], 5.00th=[ 1237], 10.00th=[ 1434],
 | 20.00th=[ 1696], 30.00th=[ 1909], 40.00th=[ 2147],
 | 50.00th=[ 2376], 60.00th=[ 2737], 70.00th=[ 3261],
 | 80.00th=[ 4015], 90.00th=[ 5276], 95.00th=[ 6783],
 | 99.00th=[ 10945], 99.50th=[ 13042], 99.90th=[ 37487],
 | 99.95th=[ 884999], 99.99th=[7281312]
 bw ( KiB/s): min= 512, max=1386496, per=38.87%, avg=698891.65, stdev=310544.20, samples=1156
 iops : min= 2, max= 5416, avg=2730.03, stdev=1213.07, samples=1156
 write: IOPS=1171, BW=293MiB/s (307MB/s)(257GiB/900001msec); 0 zone resets
 slat (usec): min=29, max=4547.4k, avg=290.22, stdev=12847.83
 clat (usec): min=303, max=9451.9k, avg=5004.77, stdev=108932.38
 lat (usec): min=534, max=9452.1k, avg=5295.11, stdev=109724.17
 clat percentiles (usec):
 | 1.00th=[ 889], 5.00th=[ 1205], 10.00th=[ 1385],
 | 20.00th=[ 1647], 30.00th=[ 1844], 40.00th=[ 2057],
 | 50.00th=[ 2278], 60.00th=[ 2540], 70.00th=[ 2999],
 | 80.00th=[ 3687], 90.00th=[ 4948], 95.00th=[ 6456],
 | 99.00th=[ 10683], 99.50th=[ 12780], 99.90th=[ 36439],
 | 99.95th=[ 734004], 99.99th=[7147095]
 bw ( KiB/s): min= 512, max=912384, per=38.91%, avg=466288.45, stdev=207371.91, samples=1156
 iops : min= 2, max= 3564, avg=1821.43, stdev=810.05, samples=1156
 lat (usec) : 2=0.01%, 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%
 lat (usec) : 750=0.33%, 1000=1.42%
 lat (msec) : 2=33.79%, 4=45.70%, 10=17.30%, 20=1.27%, 50=0.10%
 lat (msec) : 100=0.02%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.04%
 cpu : usr=0.80%, sys=56.71%, ctx=7578493, majf=0, minf=24
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=1580092,1054191,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=31632: Thu Apr 16 22:02:18 2020
 read: IOPS=1755, BW=439MiB/s (460MB/s)(386GiB/900001msec)
 slat (usec): min=10, max=7776.5k, avg=238.59, stdev=12921.86
 clat (nsec): min=1242, max=9439.3M, avg=5241348.12, stdev=109996869.99
 lat (usec): min=74, max=9439.4k, avg=5480.06, stdev=111197.22
 clat percentiles (usec):
 | 1.00th=[ 898], 5.00th=[ 1237], 10.00th=[ 1434],
 | 20.00th=[ 1713], 30.00th=[ 1926], 40.00th=[ 2147],
 | 50.00th=[ 2409], 60.00th=[ 2769], 70.00th=[ 3326],
 | 80.00th=[ 4047], 90.00th=[ 5276], 95.00th=[ 6849],
 | 99.00th=[ 10945], 99.50th=[ 13173], 99.90th=[ 33162],
 | 99.95th=[ 884999], 99.99th=[7281312]
 bw ( KiB/s): min= 512, max=1449984, per=38.38%, avg=689968.99, stdev=307834.81, samples=1171
 iops : min= 2, max= 5664, avg=2695.18, stdev=1202.49, samples=1171
 write: IOPS=1169, BW=292MiB/s (306MB/s)(257GiB/900001msec); 0 zone resets
 slat (usec): min=29, max=4367.0k, avg=285.30, stdev=11879.82
 clat (usec): min=75, max=9415.3k, avg=4960.03, stdev=107558.96
 lat (usec): min=313, max=9415.4k, avg=5245.46, stdev=108213.90
 clat percentiles (usec):
 | 1.00th=[ 881], 5.00th=[ 1205], 10.00th=[ 1401],
 | 20.00th=[ 1663], 30.00th=[ 1876], 40.00th=[ 2073],
 | 50.00th=[ 2311], 60.00th=[ 2606], 70.00th=[ 3064],
 | 80.00th=[ 3752], 90.00th=[ 5014], 95.00th=[ 6587],
 | 99.00th=[ 10683], 99.50th=[ 12911], 99.90th=[ 32900],
 | 99.95th=[ 450888], 99.99th=[7147095]
 bw ( KiB/s): min= 512, max=935424, per=38.67%, avg=463388.66, stdev=201307.47, samples=1161
 iops : min= 2, max= 3654, avg=1810.10, stdev=786.36, samples=1161
 lat (usec) : 2=0.01%, 100=0.01%, 500=0.01%, 750=0.38%, 1000=1.43%
 lat (msec) : 2=32.60%, 4=46.20%, 10=17.92%, 20=1.29%, 50=0.10%
 lat (msec) : 100=0.02%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.04%
 cpu : usr=0.80%, sys=56.16%, ctx=7520294, majf=0, minf=25
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=1580181,1052150,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
 READ: bw=1756MiB/s (1841MB/s), 439MiB/s-439MiB/s (460MB/s-460MB/s), io=1543GiB (1657GB), run=900001-900001msec
 WRITE: bw=1170MiB/s (1227MB/s), 292MiB/s-293MiB/s (306MB/s-307MB/s), io=1029GiB (1104GB), run=900001-900001msec

Disk stats (read/write):
 dm-4: ios=3563816/5044708, merge=0/0, ticks=13307671/18446744071674149170, in_queue=18446744071687456841, util=70.15%, aggrios=3565147/5185081, aggrmerge=0/0, aggrticks=13426178/18446744071890657751, aggrin_queue=18446744071904087582, aggrutil=73.91%
 dm-0: ios=3565147/5185081, merge=0/0, ticks=13426178/18446744071890657751, in_queue=18446744071904087582, util=73.91%, aggrios=4097296/5789839, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
 md128: ios=4097296/5789839, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=1864869/331459, aggrmerge=135781/2563833, aggrticks=4842566/833110, aggrin_queue=4628710, aggrutil=65.40%
 sdd: ios=1933063/340559, merge=81504/2554693, ticks=2903166/1168077, in_queue=3010995, util=64.10%
 sda: ios=1796675/322360, merge=190058/2572974, ticks=6781967/498143, in_queue=6246426, util=65.40%

machine 2:

storage: WDC WDS500G2B0A-00SM50 (ssd)

storage layout:
cryptsetup luks with camellia-xts-plain64 + lvm

raw linear read io performance with hdparm -Tt (to understand bus limitations):

/dev/sda:

Timing cached reads: 5738 MB in 2.00 seconds = 2870.00 MB/sec
 Timing buffered disk reads: 226 MB in 3.02 seconds = 74.74 MB/sec

for some reason very low speed here, probably because of external sata3 card
02:00.0 SATA controller: ASMedia Technology Inc. Device 0622 (rev 01)
smartctl show 6gb link speed here:
SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
anyway, let’s continue…

crypto performance:

[h1]Tests are approximate using memory only (no storage IO).[/h1]

[h1]Algorithm | Key | Encryption | Decryption[/h1]

twofish-xts 512b 181.5 MiB/s 186.3 MiB/s

[h1]Tests are approximate using memory only (no storage IO).[/h1]

[h1]Algorithm | Key | Encryption | Decryption[/h1]

serpent-xts 512b 170.6 MiB/s 179.2 MiB/s

[h1]Tests are approximate using memory only (no storage IO).[/h1]

[h1]Algorithm | Key | Encryption | Decryption[/h1]

aes-xts 512b 122.8 MiB/s 125.9 MiB/s

[h1]Tests are approximate using memory only (no storage IO).[/h1]

[h1]Algorithm | Key | Encryption | Decryption[/h1]

camellia-xts 512b 167.3 MiB/s 173.4 MiB/s

[h1]Tests are approximate using memory only (no storage IO).[/h1]

PBKDF2-ripemd160 129007 iterations per second for 512-bit key

[h1]Tests are approximate using memory only (no storage IO).[/h1]

PBKDF2-sha256 220659 iterations per second for 512-bit key

[h1]Tests are approximate using memory only (no storage IO).[/h1]

PBKDF2-sha512 332670 iterations per second for 512-bit key

[h1]Tests are approximate using memory only (no storage IO).[/h1]

PBKDF2-whirlpool 199197 iterations per second for 512-bit key

obvously camellia is not the best choice here for crypto, probably i should switch to twofish or serpent in future

xfs filesystem tests:

fio-rand-RW.fio:

file1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.18
Starting 4 processes
file1: Laying out IO file (1 file / 10240MiB)
Jobs: 4 (f=4): [m(4)][100.0%][r=32.4MiB/s,w=22.3MiB/s][r=8306,w=5708 IOPS][eta 00m:00s]
file1: (groupid=0, jobs=1): err= 0: pid=621474: Wed Apr 15 02:06:48 2020
 read: IOPS=1935, BW=7741KiB/s (7926kB/s)(6803MiB/900001msec)
 slat (nsec): min=1891, max=5893.4M, avg=300566.15, stdev=10037311.50
 clat (usec): min=66, max=6080.1k, avg=4678.04, stdev=40385.49
 lat (usec): min=69, max=6080.1k, avg=4979.19, stdev=42253.34
 clat percentiles (usec):
 | 1.00th=[ 685], 5.00th=[ 1352], 10.00th=[ 1778],
 | 20.00th=[ 2278], 30.00th=[ 2638], 40.00th=[ 2999],
 | 50.00th=[ 3392], 60.00th=[ 3851], 70.00th=[ 4359],
 | 80.00th=[ 5145], 90.00th=[ 6521], 95.00th=[ 8455],
 | 99.00th=[ 22676], 99.50th=[ 30016], 99.90th=[ 49021],
 | 99.95th=[ 64226], 99.99th=[2734687]
 bw ( KiB/s): min= 7, max=15936, per=27.09%, avg=8355.01, stdev=3468.68, samples=1666
 iops : min= 1, max= 3984, avg=2088.62, stdev=867.19, samples=1666
 write: IOPS=1289, BW=5159KiB/s (5283kB/s)(4534MiB/900001msec); 0 zone resets
 slat (usec): min=2, max=3014.6k, avg=312.79, stdev=6120.86
 clat (usec): min=8, max=6053.9k, avg=4614.70, stdev=41890.22
 lat (usec): min=70, max=6080.6k, avg=4928.04, stdev=42745.15
 clat percentiles (usec):
 | 1.00th=[ 660], 5.00th=[ 1303], 10.00th=[ 1713],
 | 20.00th=[ 2212], 30.00th=[ 2573], 40.00th=[ 2966],
 | 50.00th=[ 3326], 60.00th=[ 3785], 70.00th=[ 4293],
 | 80.00th=[ 5080], 90.00th=[ 6390], 95.00th=[ 8356],
 | 99.00th=[ 21627], 99.50th=[ 29230], 99.90th=[ 47973],
 | 99.95th=[ 63701], 99.99th=[2902459]
 bw ( KiB/s): min= 7, max=10672, per=27.23%, avg=5598.61, stdev=2288.35, samples=1657
 iops : min= 1, max= 2668, avg=1399.51, stdev=572.10, samples=1657
 lat (usec) : 10=0.01%, 100=0.02%, 250=0.06%, 500=0.38%, 750=0.85%
 lat (usec) : 1000=1.21%
 lat (msec) : 2=12.05%, 4=49.09%, 10=32.74%, 20=2.38%, 50=1.13%
 lat (msec) : 100=0.06%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.01%
 cpu : usr=1.59%, sys=8.53%, ctx=2707503, majf=2, minf=20
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=1741637,1160773,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=621475: Wed Apr 15 02:06:48 2020
 read: IOPS=1922, BW=7690KiB/s (7874kB/s)(6759MiB/900001msec)
 slat (nsec): min=1916, max=5612.2M, avg=294667.69, stdev=8843280.94
 clat (usec): min=5, max=6000.1k, avg=4705.97, stdev=40069.66
 lat (usec): min=69, max=6000.1k, avg=5001.22, stdev=41468.73
 clat percentiles (usec):
 | 1.00th=[ 693], 5.00th=[ 1385], 10.00th=[ 1795],
 | 20.00th=[ 2278], 30.00th=[ 2671], 40.00th=[ 3032],
 | 50.00th=[ 3425], 60.00th=[ 3851], 70.00th=[ 4424],
 | 80.00th=[ 5145], 90.00th=[ 6587], 95.00th=[ 8586],
 | 99.00th=[ 22676], 99.50th=[ 30016], 99.90th=[ 49021],
 | 99.95th=[ 65274], 99.99th=[2835350]
 bw ( KiB/s): min= 7, max=14352, per=26.88%, avg=8290.15, stdev=3431.62, samples=1668
 iops : min= 1, max= 3588, avg=2072.39, stdev=857.92, samples=1668
 write: IOPS=1281, BW=5127KiB/s (5250kB/s)(4506MiB/900001msec); 0 zone resets
 slat (usec): min=2, max=4509.8k, avg=327.40, stdev=8306.99
 clat (usec): min=66, max=6000.1k, avg=4647.82, stdev=42006.68
 lat (usec): min=69, max=6000.1k, avg=4975.78, stdev=43569.84
 clat percentiles (usec):
 | 1.00th=[ 668], 5.00th=[ 1336], 10.00th=[ 1729],
 | 20.00th=[ 2212], 30.00th=[ 2606], 40.00th=[ 2966],
 | 50.00th=[ 3359], 60.00th=[ 3818], 70.00th=[ 4359],
 | 80.00th=[ 5145], 90.00th=[ 6521], 95.00th=[ 8586],
 | 99.00th=[ 21890], 99.50th=[ 29230], 99.90th=[ 47973],
 | 99.95th=[ 63177], 99.99th=[2701132]
 bw ( KiB/s): min= 7, max= 9776, per=27.05%, avg=5560.95, stdev=2246.97, samples=1658
 iops : min= 1, max= 2444, avg=1390.10, stdev=561.75, samples=1658
 lat (usec) : 10=0.01%, 100=0.02%, 250=0.07%, 500=0.36%, 750=0.80%
 lat (usec) : 1000=1.17%
 lat (msec) : 2=11.68%, 4=49.07%, 10=33.14%, 20=2.45%, 50=1.14%
 lat (msec) : 100=0.06%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.01%
 cpu : usr=1.59%, sys=8.40%, ctx=2711432, majf=1, minf=23
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=1730177,1153643,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=621476: Wed Apr 15 02:06:48 2020
 read: IOPS=1925, BW=7701KiB/s (7886kB/s)(6769MiB/900001msec)
 slat (nsec): min=1835, max=3014.6M, avg=293882.01, stdev=7975456.23
 clat (usec): min=16, max=6081.5k, avg=4701.02, stdev=40545.51
 lat (usec): min=65, max=6081.5k, avg=4995.47, stdev=41788.33
 clat percentiles (usec):
 | 1.00th=[ 676], 5.00th=[ 1385], 10.00th=[ 1795],
 | 20.00th=[ 2278], 30.00th=[ 2671], 40.00th=[ 3032],
 | 50.00th=[ 3425], 60.00th=[ 3851], 70.00th=[ 4424],
 | 80.00th=[ 5145], 90.00th=[ 6521], 95.00th=[ 8586],
 | 99.00th=[ 22676], 99.50th=[ 30278], 99.90th=[ 50070],
 | 99.95th=[ 68682], 99.99th=[2868904]
 bw ( KiB/s): min= 7, max=14626, per=26.97%, avg=8317.50, stdev=3430.16, samples=1665
 iops : min= 1, max= 3656, avg=2079.23, stdev=857.57, samples=1665
 write: IOPS=1284, BW=5137KiB/s (5260kB/s)(4515MiB/900001msec); 0 zone resets
 slat (usec): min=2, max=5770.0k, avg=327.47, stdev=9706.72
 clat (usec): min=62, max=6081.3k, avg=4635.87, stdev=41925.59
 lat (usec): min=66, max=6081.6k, avg=4963.87, stdev=43686.92
 clat percentiles (usec):
 | 1.00th=[ 644], 5.00th=[ 1319], 10.00th=[ 1729],
 | 20.00th=[ 2212], 30.00th=[ 2606], 40.00th=[ 2966],
 | 50.00th=[ 3359], 60.00th=[ 3785], 70.00th=[ 4359],
 | 80.00th=[ 5080], 90.00th=[ 6456], 95.00th=[ 8455],
 | 99.00th=[ 21890], 99.50th=[ 29230], 99.90th=[ 47449],
 | 99.95th=[ 63177], 99.99th=[2835350]
 bw ( KiB/s): min= 8, max= 9720, per=27.12%, avg=5574.94, stdev=2265.45, samples=1657
 iops : min= 2, max= 2430, avg=1393.59, stdev=566.38, samples=1657
 lat (usec) : 20=0.01%, 100=0.02%, 250=0.07%, 500=0.38%, 750=0.86%
 lat (usec) : 1000=1.17%
 lat (msec) : 2=11.63%, 4=49.28%, 10=32.89%, 20=2.47%, 50=1.14%
 lat (msec) : 100=0.06%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.01%
 cpu : usr=1.60%, sys=8.41%, ctx=2726842, majf=0, minf=24
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=1732741,1155858,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=621477: Wed Apr 15 02:06:48 2020
 read: IOPS=1928, BW=7714KiB/s (7900kB/s)(6780MiB/900001msec)
 slat (nsec): min=1882, max=4509.8M, avg=295620.68, stdev=8875316.44
 clat (usec): min=2, max=6079.9k, avg=4726.13, stdev=43107.65
 lat (usec): min=65, max=6079.9k, avg=5022.33, stdev=44581.37
 clat percentiles (usec):
 | 1.00th=[ 693], 5.00th=[ 1385], 10.00th=[ 1795],
 | 20.00th=[ 2278], 30.00th=[ 2671], 40.00th=[ 3032],
 | 50.00th=[ 3425], 60.00th=[ 3851], 70.00th=[ 4424],
 | 80.00th=[ 5145], 90.00th=[ 6521], 95.00th=[ 8586],
 | 99.00th=[ 22414], 99.50th=[ 29754], 99.90th=[ 47973],
 | 99.95th=[ 64226], 99.99th=[2969568]
 bw ( KiB/s): min= 7, max=14872, per=27.04%, avg=8341.89, stdev=3408.53, samples=1663
 iops : min= 1, max= 3718, avg=2085.33, stdev=852.15, samples=1663
 write: IOPS=1283, BW=5136KiB/s (5259kB/s)(4514MiB/900001msec); 0 zone resets
 slat (usec): min=2, max=5611.0k, avg=324.11, stdev=8194.12
 clat (usec): min=62, max=6063.8k, avg=4587.10, stdev=38392.47
 lat (usec): min=66, max=6080.2k, avg=4911.76, stdev=39836.92
 clat percentiles (usec):
 | 1.00th=[ 668], 5.00th=[ 1336], 10.00th=[ 1729],
 | 20.00th=[ 2212], 30.00th=[ 2606], 40.00th=[ 2966],
 | 50.00th=[ 3359], 60.00th=[ 3785], 70.00th=[ 4359],
 | 80.00th=[ 5080], 90.00th=[ 6456], 95.00th=[ 8455],
 | 99.00th=[ 21890], 99.50th=[ 29230], 99.90th=[ 46400],
 | 99.95th=[ 61080], 99.99th=[1870660]
 bw ( KiB/s): min= 7, max= 9768, per=27.08%, avg=5566.76, stdev=2257.52, samples=1659
 iops : min= 1, max= 2442, avg=1391.55, stdev=564.40, samples=1659
 lat (usec) : 4=0.01%, 100=0.02%, 250=0.06%, 500=0.36%, 750=0.82%
 lat (usec) : 1000=1.18%
 lat (msec) : 2=11.73%, 4=49.18%, 10=32.94%, 20=2.47%, 50=1.15%
 lat (msec) : 100=0.05%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.01%
 cpu : usr=1.53%, sys=8.46%, ctx=2720607, majf=0, minf=24
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=1735751,1155553,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
 READ: bw=30.1MiB/s (31.6MB/s), 7690KiB/s-7741KiB/s (7874kB/s-7926kB/s), io=26.5GiB (28.4GB), run=900001-900001msec
 WRITE: bw=20.1MiB/s (21.1MB/s), 5127KiB/s-5159KiB/s (5250kB/s-5283kB/s), io=17.6GiB (18.9GB), run=900001-900001msec

Disk stats (read/write):
 dm-1: ios=3685533/4166016, merge=0/0, ticks=1286849/119447523, in_queue=120737260, util=99.40%, aggrios=3685677/4166016, aggrmerge=0/0, aggrticks=1277080/119441923, aggrin_queue=120721887, aggrutil=99.40%
 dm-0: ios=3685677/4166016, merge=0/0, ticks=1277080/119441923, in_queue=120721887, util=99.40%, aggrios=3685677/4160531, aggrmerge=0/5487, aggrticks=797504/1490474, aggrin_queue=73134, aggrutil=99.38%
 sda: ios=3685677/4160531, merge=0/5487, ticks=797504/1490474, in_queue=73134, util=99.38%

machine 3:

storage:

WDC WDS500G2B0A-00SM50 (ssd)
2 x WDC WD10EZRX-00L4HB0

storage layout:

ssd: cryptsetup luks with aes-xts-plain64 + lvm
hdd array: 2x hdd mdadm raid0 + cryptsetup luks with camellia-xts-plain64 + lvm

raw linear read io performance with hdparm -Tt (to understand bus limitations):

ssd:

Timing cached reads: 8664 MB in 2.00 seconds = 4335.58 MB/sec
 Timing buffered disk reads: 1248 MB in 3.00 seconds = 415.96 MB/sec

hdd array:

Timing cached reads: 8692 MB in 2.00 seconds = 4349.16 MB/sec
 Timing buffered disk reads: 892 MB in 3.00 seconds = 297.28 MB/sec

crypto performance:

[h1]Tests are approximate using memory only (no storage IO).[/h1]

[h1]Algorithm | Key | Encryption | Decryption[/h1]

twofish-xts 512b 219.6 MiB/s 225.9 MiB/s

[h1]Tests are approximate using memory only (no storage IO).[/h1]

[h1]Algorithm | Key | Encryption | Decryption[/h1]

serpent-xts 512b 226.4 MiB/s 233.5 MiB/s

[h1]Tests are approximate using memory only (no storage IO).[/h1]

[h1]Algorithm | Key | Encryption | Decryption[/h1]

aes-xts 512b 1198.2 MiB/s 1102.4 MiB/s

[h1]Tests are approximate using memory only (no storage IO).[/h1]

[h1]Algorithm | Key | Encryption | Decryption[/h1]

camellia-xts 512b 259.6 MiB/s 267.7 MiB/s

[h1]Tests are approximate using memory only (no storage IO).[/h1]

PBKDF2-ripemd160 168473 iterations per second for 512-bit key

[h1]Tests are approximate using memory only (no storage IO).[/h1]

PBKDF2-sha256 230355 iterations per second for 512-bit key

[h1]Tests are approximate using memory only (no storage IO).[/h1]

PBKDF2-sha512 392431 iterations per second for 512-bit key

[h1]Tests are approximate using memory only (no storage IO).[/h1]

PBKDF2-whirlpool 215934 iterations per second for 512-bit key

xfs filesystem tests:

fio-rand-RW.fio:

ssd:

file1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.18
Starting 4 processes
file1: Laying out IO file (1 file / 10240MiB)
Jobs: 4 (f=4): [m(4)][100.0%][eta 00m:00s] 
file1: (groupid=0, jobs=1): err= 0: pid=115788: Wed Apr 15 02:42:20 2020
 read: IOPS=4398, BW=17.2MiB/s (18.0MB/s)(15.2GiB/904453msec)
 slat (nsec): min=1713, max=19778M, avg=106136.70, stdev=11322677.73
 clat (usec): min=52, max=27685k, avg=2068.36, stdev=71398.30
 lat (usec): min=54, max=27685k, avg=2174.71, stdev=72597.41
 clat percentiles (usec):
 | 1.00th=[ 83], 5.00th=[ 100], 10.00th=[ 110], 20.00th=[ 165],
 | 30.00th=[ 469], 40.00th=[ 775], 50.00th=[ 1090], 60.00th=[ 1467],
 | 70.00th=[ 1909], 80.00th=[ 2474], 90.00th=[ 3392], 95.00th=[ 4359],
 | 99.00th=[ 7570], 99.50th=[ 10683], 99.90th=[ 29230], 99.95th=[ 41157],
 | 99.99th=[187696]
 bw ( KiB/s): min= 8, max=340064, per=31.70%, avg=22272.22, stdev=33531.22, samples=1428
 iops : min= 2, max=85016, avg=5567.98, stdev=8382.81, samples=1428
 write: IOPS=2932, BW=11.5MiB/s (12.0MB/s)(10.1GiB/904453msec); 0 zone resets
 slat (usec): min=2, max=12792k, avg=169.85, stdev=19676.64
 clat (usec): min=4, max=27685k, avg=2015.24, stdev=70796.95
 lat (usec): min=55, max=27685k, avg=2185.32, stdev=74041.06
 clat percentiles (usec):
 | 1.00th=[ 82], 5.00th=[ 99], 10.00th=[ 110], 20.00th=[ 159],
 | 30.00th=[ 449], 40.00th=[ 734], 50.00th=[ 1045], 60.00th=[ 1401],
 | 70.00th=[ 1844], 80.00th=[ 2409], 90.00th=[ 3326], 95.00th=[ 4293],
 | 99.00th=[ 7439], 99.50th=[ 10290], 99.90th=[ 27657], 99.95th=[ 39584],
 | 99.99th=[160433]
 bw ( KiB/s): min= 8, max=228088, per=31.78%, avg=14883.21, stdev=22374.69, samples=1425
 iops : min= 2, max=57022, avg=3720.72, stdev=5593.66, samples=1425
 lat (usec) : 10=0.01%, 100=5.29%, 250=18.63%, 500=7.73%, 750=8.09%
 lat (usec) : 1000=7.88%
 lat (msec) : 2=24.60%, 4=21.55%, 10=5.69%, 20=0.34%, 50=0.17%
 lat (msec) : 100=0.02%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.01%
 cpu : usr=2.18%, sys=13.19%, ctx=3611418, majf=0, minf=23
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=3977929,2652645,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=115789: Wed Apr 15 02:42:20 2020
 read: IOPS=4387, BW=17.1MiB/s (17.0MB/s)(15.1GiB/904453msec)
 slat (nsec): min=1492, max=9319.9M, avg=114899.83, stdev=10620345.06
 clat (usec): min=9, max=27835k, avg=2066.00, stdev=71693.91
 lat (usec): min=56, max=27835k, avg=2181.12, stdev=72886.66
 clat percentiles (usec):
 | 1.00th=[ 83], 5.00th=[ 100], 10.00th=[ 110], 20.00th=[ 163],
 | 30.00th=[ 465], 40.00th=[ 766], 50.00th=[ 1090], 60.00th=[ 1467],
 | 70.00th=[ 1926], 80.00th=[ 2507], 90.00th=[ 3425], 95.00th=[ 4359],
 | 99.00th=[ 7570], 99.50th=[ 10683], 99.90th=[ 29230], 99.95th=[ 40109],
 | 99.99th=[147850]
 bw ( KiB/s): min= 8, max=347336, per=31.64%, avg=22231.52, stdev=33886.02, samples=1427
 iops : min= 2, max=86834, avg=5557.80, stdev=8471.50, samples=1427
 write: IOPS=2925, BW=11.4MiB/s (11.0MB/s)(10.1GiB/904453msec); 0 zone resets
 slat (usec): min=2, max=19745k, avg=157.48, stdev=20211.37
 clat (usec): min=54, max=27835k, avg=2031.82, stdev=73292.55
 lat (usec): min=57, max=27835k, avg=2189.53, stdev=76492.99
 clat percentiles (usec):
 | 1.00th=[ 83], 5.00th=[ 99], 10.00th=[ 109], 20.00th=[ 159],
 | 30.00th=[ 441], 40.00th=[ 734], 50.00th=[ 1045], 60.00th=[ 1418],
 | 70.00th=[ 1860], 80.00th=[ 2442], 90.00th=[ 3359], 95.00th=[ 4293],
 | 99.00th=[ 7439], 99.50th=[ 10290], 99.90th=[ 28181], 99.95th=[ 39060],
 | 99.99th=[137364]
 bw ( KiB/s): min= 8, max=231312, per=31.85%, avg=14918.22, stdev=22628.46, samples=1418
 iops : min= 2, max=57828, avg=3729.47, stdev=5657.11, samples=1418
 lat (usec) : 10=0.01%, 100=5.33%, 250=18.74%, 500=7.82%, 750=8.01%
 lat (usec) : 1000=7.79%
 lat (msec) : 2=24.35%, 4=21.61%, 10=5.80%, 20=0.35%, 50=0.17%
 lat (msec) : 100=0.02%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.01%
 cpu : usr=2.16%, sys=13.21%, ctx=3592523, majf=0, minf=23
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=3967874,2645804,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=115790: Wed Apr 15 02:42:20 2020
 read: IOPS=4393, BW=17.2MiB/s (17.0MB/s)(15.2GiB/904453msec)
 slat (nsec): min=1681, max=12792M, avg=116265.18, stdev=13332344.36
 clat (usec): min=53, max=28423k, avg=2089.26, stdev=73339.30
 lat (usec): min=56, max=28423k, avg=2205.74, stdev=74861.83
 clat percentiles (usec):
 | 1.00th=[ 87], 5.00th=[ 100], 10.00th=[ 110], 20.00th=[ 172],
 | 30.00th=[ 469], 40.00th=[ 775], 50.00th=[ 1090], 60.00th=[ 1467],
 | 70.00th=[ 1909], 80.00th=[ 2474], 90.00th=[ 3392], 95.00th=[ 4359],
 | 99.00th=[ 7570], 99.50th=[ 10552], 99.90th=[ 28967], 99.95th=[ 39584],
 | 99.99th=[198181]
 bw ( KiB/s): min= 8, max=342864, per=31.75%, avg=22311.96, stdev=33541.39, samples=1424
 iops : min= 2, max=85716, avg=5577.91, stdev=8385.34, samples=1424
 write: IOPS=2928, BW=11.4MiB/s (11.0MB/s)(10.1GiB/904453msec); 0 zone resets
 slat (usec): min=2, max=19745k, avg=154.84, stdev=17365.04
 clat (usec): min=10, max=28423k, avg=1990.12, stdev=69995.95
 lat (usec): min=56, max=28423k, avg=2145.20, stdev=72725.24
 clat percentiles (usec):
 | 1.00th=[ 86], 5.00th=[ 100], 10.00th=[ 110], 20.00th=[ 163],
 | 30.00th=[ 445], 40.00th=[ 734], 50.00th=[ 1045], 60.00th=[ 1401],
 | 70.00th=[ 1844], 80.00th=[ 2409], 90.00th=[ 3326], 95.00th=[ 4293],
 | 99.00th=[ 7439], 99.50th=[ 10290], 99.90th=[ 28181], 99.95th=[ 39060],
 | 99.99th=[141558]
 bw ( KiB/s): min= 8, max=229368, per=31.84%, avg=14913.15, stdev=22414.86, samples=1420
 iops : min= 2, max=57342, avg=3728.20, stdev=5603.70, samples=1420
 lat (usec) : 20=0.01%, 100=5.01%, 250=18.72%, 500=7.96%, 750=8.05%
 lat (usec) : 1000=7.88%
 lat (msec) : 2=24.63%, 4=21.49%, 10=5.71%, 20=0.35%, 50=0.17%
 lat (msec) : 100=0.02%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.01%
 cpu : usr=2.15%, sys=13.25%, ctx=3649760, majf=0, minf=23
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=3973815,2648617,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=115791: Wed Apr 15 02:42:20 2020
 read: IOPS=4387, BW=17.1MiB/s (17.0MB/s)(15.1GiB/904454msec)
 slat (nsec): min=1743, max=19778M, avg=121148.63, stdev=16032951.75
 clat (usec): min=53, max=27687k, avg=2051.97, stdev=66316.94
 lat (usec): min=56, max=27687k, avg=2173.34, stdev=68621.44
 clat percentiles (usec):
 | 1.00th=[ 80], 5.00th=[ 99], 10.00th=[ 109], 20.00th=[ 165],
 | 30.00th=[ 469], 40.00th=[ 775], 50.00th=[ 1106], 60.00th=[ 1483],
 | 70.00th=[ 1926], 80.00th=[ 2507], 90.00th=[ 3425], 95.00th=[ 4359],
 | 99.00th=[ 7570], 99.50th=[ 10683], 99.90th=[ 28443], 99.95th=[ 39584],
 | 99.99th=[198181]
 bw ( KiB/s): min= 8, max=349384, per=31.64%, avg=22234.08, stdev=33880.89, samples=1427
 iops : min= 2, max=87346, avg=5558.44, stdev=8470.21, samples=1427
 write: IOPS=2921, BW=11.4MiB/s (11.0MB/s)(10.1GiB/904454msec); 0 zone resets
 slat (usec): min=2, max=12792k, avg=148.36, stdev=13818.41
 clat (usec): min=10, max=27686k, avg=2055.01, stdev=77333.17
 lat (usec): min=56, max=27686k, avg=2203.60, stdev=78954.75
 clat percentiles (usec):
 | 1.00th=[ 79], 5.00th=[ 99], 10.00th=[ 109], 20.00th=[ 159],
 | 30.00th=[ 449], 40.00th=[ 742], 50.00th=[ 1057], 60.00th=[ 1418],
 | 70.00th=[ 1860], 80.00th=[ 2442], 90.00th=[ 3359], 95.00th=[ 4293],
 | 99.00th=[ 7439], 99.50th=[ 10290], 99.90th=[ 27919], 99.95th=[ 39060],
 | 99.99th=[198181]
 bw ( KiB/s): min= 8, max=232600, per=31.68%, avg=14837.29, stdev=22596.83, samples=1424
 iops : min= 2, max=58150, avg=3709.24, stdev=5649.20, samples=1424
 lat (usec) : 20=0.01%, 100=5.68%, 250=18.35%, 500=7.66%, 750=7.96%
 lat (usec) : 1000=7.82%
 lat (msec) : 2=24.51%, 4=21.71%, 10=5.78%, 20=0.35%, 50=0.16%
 lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.01%
 cpu : usr=2.18%, sys=13.17%, ctx=3600493, majf=0, minf=23
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=3968340,2642604,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
 READ: bw=68.6MiB/s (71.0MB/s), 17.1MiB/s-17.2MiB/s (17.0MB/s-18.0MB/s), io=60.6GiB (65.1GB), run=904453-904454msec
 WRITE: bw=45.7MiB/s (47.0MB/s), 11.4MiB/s-11.5MiB/s (11.0MB/s-12.0MB/s), io=40.4GiB (43.4GB), run=904453-904454msec

Disk stats (read/write):
 dm-2: ios=3315692/8009253, merge=0/0, ticks=1040390/18446744071668371254, in_queue=18446744071669412014, util=98.70%, aggrios=3315692/8009253, aggrmerge=0/0, aggrticks=1036954/18446744071689086446, aggrin_queue=18446744071690123769, aggrutil=98.70%
 dm-1: ios=3315692/8009253, merge=0/0, ticks=1036954/18446744071689086446, in_queue=18446744071690123769, util=98.70%, aggrios=3315692/7323541, aggrmerge=0/447753, aggrticks=464244/1102433, aggrin_queue=46733, aggrutil=98.65%
 sdc: ios=3315692/7323541, merge=0/447753, ticks=464244/1102433, in_queue=46733, util=98.65%

hdd array:

file1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.18
Starting 4 processes
file1: Laying out IO file (1 file / 10240MiB)
Jobs: 3 (f=3): [m(1),_(1),m(2)][50.1%][r=47KiB/s,w=15KiB/s][r=11,w=3 IOPS][eta 14m:59s]
file1: (groupid=0, jobs=1): err= 0: pid=116099: Wed Apr 15 03:01:43 2020
 read: IOPS=35, BW=142KiB/s (145kB/s)(125MiB/901171msec)
 slat (usec): min=4, max=2393.1k, avg=19205.65, stdev=52749.56
 clat (msec): min=68, max=3719, avg=255.72, stdev=245.40
 lat (msec): min=78, max=3735, avg=274.93, stdev=255.14
 clat percentiles (msec):
 | 1.00th=[ 131], 5.00th=[ 159], 10.00th=[ 174], 20.00th=[ 192],
 | 30.00th=[ 205], 40.00th=[ 218], 50.00th=[ 230], 60.00th=[ 243],
 | 70.00th=[ 255], 80.00th=[ 271], 90.00th=[ 300], 95.00th=[ 326],
 | 99.00th=[ 651], 99.50th=[ 2836], 99.90th=[ 3205], 99.95th=[ 3473],
 | 99.99th=[ 3708]
 bw ( KiB/s): min= 7, max= 247, per=26.52%, avg=149.84, stdev=39.31, samples=1694
 iops : min= 1, max= 61, avg=36.78, stdev= 9.82, samples=1694
 write: IOPS=23, BW=94.3KiB/s (96.5kB/s)(82.0MiB/901171msec); 0 zone resets
 slat (usec): min=3, max=2581.7k, avg=13522.18, stdev=44734.86
 clat (usec): min=11, max=3695.6k, avg=251779.96, stdev=245770.99
 lat (msec): min=71, max=3708, avg=265.30, stdev=252.69
 clat percentiles (msec):
 | 1.00th=[ 128], 5.00th=[ 155], 10.00th=[ 169], 20.00th=[ 188],
 | 30.00th=[ 201], 40.00th=[ 213], 50.00th=[ 226], 60.00th=[ 239],
 | 70.00th=[ 251], 80.00th=[ 268], 90.00th=[ 296], 95.00th=[ 326],
 | 99.00th=[ 651], 99.50th=[ 2836], 99.90th=[ 3239], 99.95th=[ 3540],
 | 99.99th=[ 3641]
 bw ( KiB/s): min= 7, max= 215, per=26.32%, avg=99.49, stdev=33.86, samples=1692
 iops : min= 1, max= 53, avg=24.20, stdev= 8.47, samples=1692
 lat (usec) : 20=0.01%
 lat (msec) : 100=0.12%, 250=67.49%, 500=30.99%, 750=0.45%, 1000=0.03%
 lat (msec) : 2000=0.18%, >=2000=0.72%
 cpu : usr=0.09%, sys=0.87%, ctx=68032, majf=0, minf=18
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=31933,21241,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=116100: Wed Apr 15 03:01:43 2020
 read: IOPS=35, BW=141KiB/s (145kB/s)(124MiB/900753msec)
 slat (usec): min=3, max=2189.7k, avg=19250.42, stdev=50318.66
 clat (usec): min=12, max=3576.5k, avg=256838.86, stdev=250357.23
 lat (msec): min=76, max=3637, avg=276.09, stdev=259.82
 clat percentiles (msec):
 | 1.00th=[ 132], 5.00th=[ 159], 10.00th=[ 174], 20.00th=[ 192],
 | 30.00th=[ 205], 40.00th=[ 218], 50.00th=[ 228], 60.00th=[ 241],
 | 70.00th=[ 255], 80.00th=[ 271], 90.00th=[ 300], 95.00th=[ 330],
 | 99.00th=[ 793], 99.50th=[ 2869], 99.90th=[ 3205], 99.95th=[ 3473],
 | 99.99th=[ 3540]
 bw ( KiB/s): min= 7, max= 255, per=26.30%, avg=148.62, stdev=39.97, samples=1700
 iops : min= 1, max= 63, avg=36.48, stdev= 9.99, samples=1700
 write: IOPS=23, BW=94.3KiB/s (96.5kB/s)(82.9MiB/900753msec); 0 zone resets
 slat (usec): min=3, max=2186.3k, avg=13572.91, stdev=44869.81
 clat (msec): min=71, max=3604, avg=251.70, stdev=233.77
 lat (msec): min=78, max=3604, avg=265.28, stdev=241.19
 clat percentiles (msec):
 | 1.00th=[ 129], 5.00th=[ 157], 10.00th=[ 171], 20.00th=[ 190],
 | 30.00th=[ 203], 40.00th=[ 215], 50.00th=[ 226], 60.00th=[ 239],
 | 70.00th=[ 253], 80.00th=[ 271], 90.00th=[ 296], 95.00th=[ 326],
 | 99.00th=[ 625], 99.50th=[ 2735], 99.90th=[ 3171], 99.95th=[ 3440],
 | 99.99th=[ 3540]
 bw ( KiB/s): min= 7, max= 215, per=26.22%, avg=99.10, stdev=33.88, samples=1698
 iops : min= 1, max= 53, avg=24.10, stdev= 8.47, samples=1698
 lat (usec) : 20=0.01%
 lat (msec) : 100=0.08%, 250=67.58%, 500=30.91%, 750=0.48%, 1000=0.04%
 lat (msec) : 2000=0.16%, >=2000=0.76%
 cpu : usr=0.09%, sys=0.90%, ctx=67957, majf=0, minf=20
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=31788,21231,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=116101: Wed Apr 15 03:01:43 2020
 read: IOPS=35, BW=142KiB/s (145kB/s)(125MiB/901171msec)
 slat (usec): min=4, max=2093.4k, avg=18729.42, stdev=40433.47
 clat (msec): min=70, max=3583, avg=255.23, stdev=242.69
 lat (msec): min=85, max=3596, avg=273.96, stdev=250.06
 clat percentiles (msec):
 | 1.00th=[ 130], 5.00th=[ 157], 10.00th=[ 171], 20.00th=[ 190],
 | 30.00th=[ 205], 40.00th=[ 218], 50.00th=[ 230], 60.00th=[ 243],
 | 70.00th=[ 255], 80.00th=[ 271], 90.00th=[ 300], 95.00th=[ 330],
 | 99.00th=[ 642], 99.50th=[ 2802], 99.90th=[ 3239], 99.95th=[ 3473],
 | 99.99th=[ 3540]
 bw ( KiB/s): min= 7, max= 271, per=26.38%, avg=149.04, stdev=40.51, samples=1702
 iops : min= 1, max= 67, avg=36.58, stdev=10.12, samples=1702
 write: IOPS=23, BW=94.3KiB/s (96.6kB/s)(83.0MiB/901171msec); 0 zone resets
 slat (usec): min=4, max=2162.2k, avg=14248.26, stdev=57466.27
 clat (usec): min=11, max=3581.3k, avg=252656.28, stdev=247111.54
 lat (msec): min=82, max=3602, avg=266.91, stdev=257.81
 clat percentiles (msec):
 | 1.00th=[ 127], 5.00th=[ 153], 10.00th=[ 169], 20.00th=[ 188],
 | 30.00th=[ 201], 40.00th=[ 215], 50.00th=[ 226], 60.00th=[ 239],
 | 70.00th=[ 253], 80.00th=[ 271], 90.00th=[ 296], 95.00th=[ 321],
 | 99.00th=[ 701], 99.50th=[ 2836], 99.90th=[ 3205], 99.95th=[ 3339],
 | 99.99th=[ 3540]
 bw ( KiB/s): min= 7, max= 223, per=26.29%, avg=99.37, stdev=34.59, samples=1695
 iops : min= 1, max= 55, avg=24.16, stdev= 8.64, samples=1695
 lat (usec) : 20=0.01%
 lat (msec) : 100=0.11%, 250=67.39%, 500=30.99%, 750=0.56%, 1000=0.05%
 lat (msec) : 2000=0.17%, >=2000=0.73%
 cpu : usr=0.09%, sys=0.89%, ctx=68014, majf=0, minf=20
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=31912,21253,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=116102: Wed Apr 15 03:01:43 2020
 read: IOPS=35, BW=141KiB/s (144kB/s)(124MiB/901172msec)
 slat (usec): min=2, max=2894.9k, avg=19180.06, stdev=54568.73
 clat (usec): min=11, max=3696.7k, avg=255785.90, stdev=247754.36
 lat (msec): min=71, max=3709, avg=274.97, stdev=257.15
 clat percentiles (msec):
 | 1.00th=[ 133], 5.00th=[ 159], 10.00th=[ 174], 20.00th=[ 192],
 | 30.00th=[ 205], 40.00th=[ 218], 50.00th=[ 228], 60.00th=[ 241],
 | 70.00th=[ 255], 80.00th=[ 271], 90.00th=[ 300], 95.00th=[ 330],
 | 99.00th=[ 617], 99.50th=[ 2836], 99.90th=[ 3205], 99.95th=[ 3473],
 | 99.99th=[ 3641]
 bw ( KiB/s): min= 7, max= 279, per=26.23%, avg=148.18, stdev=41.04, samples=1702
 iops : min= 1, max= 69, avg=36.37, stdev=10.25, samples=1702
 write: IOPS=23, BW=95.4KiB/s (97.7kB/s)(83.9MiB/901172msec); 0 zone resets
 slat (usec): min=3, max=2066.5k, avg=13586.55, stdev=41539.32
 clat (msec): min=58, max=3694, avg=251.30, stdev=240.94
 lat (msec): min=63, max=3725, avg=264.88, stdev=248.29
 clat percentiles (msec):
 | 1.00th=[ 129], 5.00th=[ 155], 10.00th=[ 169], 20.00th=[ 188],
 | 30.00th=[ 201], 40.00th=[ 213], 50.00th=[ 226], 60.00th=[ 239],
 | 70.00th=[ 251], 80.00th=[ 268], 90.00th=[ 296], 95.00th=[ 326],
 | 99.00th=[ 667], 99.50th=[ 2769], 99.90th=[ 3205], 99.95th=[ 3440],
 | 99.99th=[ 3641]
 bw ( KiB/s): min= 7, max= 223, per=26.61%, avg=100.59, stdev=34.36, samples=1693
 iops : min= 1, max= 55, avg=24.47, stdev= 8.59, samples=1693
 lat (usec) : 20=0.01%
 lat (msec) : 100=0.09%, 250=67.94%, 500=30.62%, 750=0.40%, 1000=0.05%
 lat (msec) : 2000=0.16%, >=2000=0.73%
 cpu : usr=0.09%, sys=0.89%, ctx=67816, majf=0, minf=19
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=31730,21487,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
 READ: bw=565KiB/s (579kB/s), 141KiB/s-142KiB/s (144kB/s-145kB/s), io=498MiB (522MB), run=900753-901172msec
 WRITE: bw=378KiB/s (387kB/s), 94.3KiB/s-95.4KiB/s (96.5kB/s-97.7kB/s), io=333MiB (349MB), run=900753-901172msec

Disk stats (read/write):
 dm-3: ios=123568/85178, merge=0/0, ticks=1814445/107073064, in_queue=108887509, util=20.35%, aggrios=123568/85178, aggrmerge=0/0, aggrticks=1815844/107585903, aggrin_queue=109401747, aggrutil=20.41%
 dm-0: ios=123568/85178, merge=0/0, ticks=1815844/107585903, in_queue=109401747, util=20.41%, aggrios=123574/84872, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
 md127: ios=123574/84872, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=61787/42433, aggrmerge=0/0, aggrticks=902035/1902456, aggrin_queue=2752314, aggrutil=14.40%
 sdb: ios=61589/42295, merge=0/0, ticks=896372/1921815, in_queue=2766186, util=14.25%
 sda: ios=61985/42572, merge=0/0, ticks=907698/1883097, in_queue=2738442, util=14.40%

machine 4:
NOTE: tests on this machine done under some light io load which cannot be avoided, so they are not very accurate

storage:
SanDisk SDSSDH3 500G (ssd)

storage layout:
ssd: cryptsetup luks with aes-xts-plain64 + lvm

raw linear read io performance with hdparm -Tt (to understand bus limitations):
ssd:

Timing cached reads: 4814 MB in 2.00 seconds = 2408.90 MB/sec
 Timing buffered disk reads: 1354 MB in 3.00 seconds = 451.18 MB/sec

crypto performance:

[h1]Tests are approximate using memory only (no storage IO).[/h1]

[h1]Algorithm | Key | Encryption | Decryption[/h1]

twofish-xts 512b 125.8 MiB/s 129.4 MiB/s

[h1]Tests are approximate using memory only (no storage IO).[/h1]

[h1]Algorithm | Key | Encryption | Decryption[/h1]

serpent-xts 512b 168.2 MiB/s 171.9 MiB/s

[h1]Tests are approximate using memory only (no storage IO).[/h1]

[h1]Algorithm | Key | Encryption | Decryption[/h1]

aes-xts 512b 738.9 MiB/s 721.8 MiB/s

[h1]Tests are approximate using memory only (no storage IO).[/h1]

[h1]Algorithm | Key | Encryption | Decryption[/h1]

camellia-xts 512b 183.5 MiB/s 184.7 MiB/s

[h1]Tests are approximate using memory only (no storage IO).[/h1]

PBKDF2-ripemd160 70167 iterations per second for 512-bit key

[h1]Tests are approximate using memory only (no storage IO).[/h1]

PBKDF2-sha256 110329 iterations per second for 512-bit key

[h1]Tests are approximate using memory only (no storage IO).[/h1]

PBKDF2-sha512 188321 iterations per second for 512-bit key

[h1]Tests are approximate using memory only (no storage IO).[/h1]

PBKDF2-whirlpool 107789 iterations per second for 512-bit key

xfs filesystem tests:

fio-rand-RW.fio:

ssd array:

file1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.18
Starting 4 processes
file1: Laying out IO file (1 file / 10240MiB)
Jobs: 4 (f=4): [m(4)][100.0%][r=104MiB/s,w=69.6MiB/s][r=26.7k,w=17.8k IOPS][eta 00m:00s]
file1: (groupid=0, jobs=1): err= 0: pid=864500: Wed Apr 15 02:04:11 2020
 read: IOPS=4388, BW=17.1MiB/s (17.0MB/s)(15.1GiB/900001msec)
 slat (usec): min=3, max=2554.5k, avg=119.35, stdev=1869.84
 clat (usec): min=4, max=16401k, avg=2058.41, stdev=31678.60
 lat (usec): min=14, max=16401k, avg=2178.06, stdev=31763.13
 clat percentiles (usec):
 | 1.00th=[ 186], 5.00th=[ 221], 10.00th=[ 249], 20.00th=[ 363],
 | 30.00th=[ 644], 40.00th=[ 988], 50.00th=[ 1352], 60.00th=[ 1713],
 | 70.00th=[ 2114], 80.00th=[ 2671], 90.00th=[ 3621], 95.00th=[ 5080],
 | 99.00th=[ 12649], 99.50th=[ 20579], 99.90th=[ 54264], 99.95th=[ 83362],
 | 99.99th=[154141]
 bw ( KiB/s): min= 7, max=148095, per=26.09%, avg=18286.59, stdev=19853.55, samples=1723
 iops : min= 1, max=37023, avg=4571.29, stdev=4963.39, samples=1723
 write: IOPS=2926, BW=11.4MiB/s (11.0MB/s)(10.0GiB/900001msec); 0 zone resets
 slat (usec): min=4, max=1758.8k, avg=141.70, stdev=1462.20
 clat (usec): min=18, max=16401k, avg=2042.26, stdev=39682.13
 lat (usec): min=49, max=16401k, avg=2184.29, stdev=39771.47
 clat percentiles (usec):
 | 1.00th=[ 186], 5.00th=[ 221], 10.00th=[ 247], 20.00th=[ 351],
 | 30.00th=[ 619], 40.00th=[ 947], 50.00th=[ 1287], 60.00th=[ 1663],
 | 70.00th=[ 2073], 80.00th=[ 2606], 90.00th=[ 3556], 95.00th=[ 4883],
 | 99.00th=[ 12125], 99.50th=[ 20055], 99.90th=[ 52691], 99.95th=[ 80217],
 | 99.99th=[149947]
 bw ( KiB/s): min= 7, max=98794, per=26.11%, avg=12200.11, stdev=13247.13, samples=1722
 iops : min= 1, max=24698, avg=3049.65, stdev=3311.77, samples=1722
 lat (usec) : 10=0.01%, 20=0.01%, 100=0.01%, 250=10.22%, 500=15.13%
 lat (usec) : 750=8.36%, 1000=7.15%
 lat (msec) : 2=26.74%, 4=24.52%, 10=6.51%, 20=0.85%, 50=0.40%
 lat (msec) : 100=0.08%, 250=0.03%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.01%
 cpu : usr=2.96%, sys=19.35%, ctx=4143437, majf=0, minf=24
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=3949966,2633725,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=864501: Wed Apr 15 02:04:11 2020
 read: IOPS=4371, BW=17.1MiB/s (17.9MB/s)(15.0GiB/900001msec)
 slat (usec): min=3, max=447445, avg=117.01, stdev=754.98
 clat (usec): min=78, max=15782k, avg=2093.16, stdev=36392.78
 lat (usec): min=112, max=15782k, avg=2210.48, stdev=36425.14
 clat percentiles (usec):
 | 1.00th=[ 159], 5.00th=[ 212], 10.00th=[ 241], 20.00th=[ 338],
 | 30.00th=[ 619], 40.00th=[ 971], 50.00th=[ 1352], 60.00th=[ 1729],
 | 70.00th=[ 2147], 80.00th=[ 2704], 90.00th=[ 3687], 95.00th=[ 5145],
 | 99.00th=[ 12780], 99.50th=[ 20841], 99.90th=[ 55313], 99.95th=[ 82314],
 | 99.99th=[160433]
 bw ( KiB/s): min= 7, max=187361, per=25.97%, avg=18202.14, stdev=20885.08, samples=1724
 iops : min= 1, max=46840, avg=4550.17, stdev=5221.28, samples=1724
 write: IOPS=2914, BW=11.4MiB/s (11.9MB/s)(10.0GiB/900001msec); 0 zone resets
 slat (usec): min=4, max=2554.6k, avg=146.89, stdev=2574.23
 clat (usec): min=3, max=15791k, avg=2010.83, stdev=29391.42
 lat (usec): min=18, max=15791k, avg=2158.06, stdev=29591.43
 clat percentiles (usec):
 | 1.00th=[ 159], 5.00th=[ 212], 10.00th=[ 241], 20.00th=[ 330],
 | 30.00th=[ 594], 40.00th=[ 930], 50.00th=[ 1303], 60.00th=[ 1680],
 | 70.00th=[ 2089], 80.00th=[ 2638], 90.00th=[ 3589], 95.00th=[ 5014],
 | 99.00th=[ 12256], 99.50th=[ 20055], 99.90th=[ 52691], 99.95th=[ 80217],
 | 99.99th=[154141]
 bw ( KiB/s): min= 7, max=123968, per=25.98%, avg=12137.94, stdev=13935.60, samples=1724
 iops : min= 1, max=30992, avg=3034.11, stdev=3483.90, samples=1724
 lat (usec) : 4=0.01%, 50=0.01%, 100=0.01%, 250=11.59%, 500=14.77%
 lat (usec) : 750=8.02%, 1000=6.88%
 lat (msec) : 2=25.91%, 4=24.69%, 10=6.75%, 20=0.88%, 50=0.41%
 lat (msec) : 100=0.08%, 250=0.03%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.01%
 cpu : usr=2.99%, sys=19.20%, ctx=4148391, majf=0, minf=24
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=3933948,2623346,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=864502: Wed Apr 15 02:04:11 2020
 read: IOPS=4379, BW=17.1MiB/s (17.9MB/s)(15.0GiB/900001msec)
 slat (usec): min=3, max=2206.4k, avg=118.59, stdev=1655.52
 clat (usec): min=5, max=16019k, avg=2076.84, stdev=34200.33
 lat (usec): min=111, max=16019k, avg=2195.74, stdev=34296.39
 clat percentiles (usec):
 | 1.00th=[ 182], 5.00th=[ 219], 10.00th=[ 247], 20.00th=[ 351],
 | 30.00th=[ 635], 40.00th=[ 979], 50.00th=[ 1352], 60.00th=[ 1729],
 | 70.00th=[ 2147], 80.00th=[ 2671], 90.00th=[ 3654], 95.00th=[ 5080],
 | 99.00th=[ 12649], 99.50th=[ 20841], 99.90th=[ 55313], 99.95th=[ 83362],
 | 99.99th=[162530]
 bw ( KiB/s): min= 15, max=172047, per=26.02%, avg=18238.54, stdev=20148.48, samples=1724
 iops : min= 3, max=43011, avg=4559.27, stdev=5037.11, samples=1724
 write: IOPS=2918, BW=11.4MiB/s (11.0MB/s)(10.0GiB/900001msec); 0 zone resets
 slat (usec): min=4, max=2554.6k, avg=143.84, stdev=1820.61
 clat (usec): min=105, max=16010k, avg=2026.25, stdev=34670.41
 lat (usec): min=111, max=16010k, avg=2170.42, stdev=34749.00
 clat percentiles (usec):
 | 1.00th=[ 182], 5.00th=[ 217], 10.00th=[ 245], 20.00th=[ 343],
 | 30.00th=[ 603], 40.00th=[ 938], 50.00th=[ 1303], 60.00th=[ 1663],
 | 70.00th=[ 2089], 80.00th=[ 2606], 90.00th=[ 3556], 95.00th=[ 4948],
 | 99.00th=[ 12256], 99.50th=[ 20055], 99.90th=[ 52691], 99.95th=[ 80217],
 | 99.99th=[160433]
 bw ( KiB/s): min= 7, max=116630, per=26.00%, avg=12147.34, stdev=13451.00, samples=1725
 iops : min= 1, max=29157, avg=3036.47, stdev=3362.75, samples=1725
 lat (usec) : 10=0.01%, 250=10.65%, 500=15.18%, 750=8.20%, 1000=7.01%
 lat (msec) : 2=26.33%, 4=24.64%, 10=6.61%, 20=0.87%, 50=0.40%
 lat (msec) : 100=0.08%, 250=0.03%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.01%
 cpu : usr=2.89%, sys=19.29%, ctx=4154263, majf=0, minf=24
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=3941858,2626995,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=864503: Wed Apr 15 02:04:11 2020
 read: IOPS=4385, BW=17.1MiB/s (17.0MB/s)(15.1GiB/900001msec)
 slat (usec): min=3, max=700760, avg=118.74, stdev=889.52
 clat (usec): min=115, max=16070k, avg=2068.03, stdev=32558.52
 lat (usec): min=119, max=16070k, avg=2187.07, stdev=32600.79
 clat percentiles (usec):
 | 1.00th=[ 186], 5.00th=[ 219], 10.00th=[ 247], 20.00th=[ 359],
 | 30.00th=[ 635], 40.00th=[ 979], 50.00th=[ 1352], 60.00th=[ 1713],
 | 70.00th=[ 2147], 80.00th=[ 2671], 90.00th=[ 3654], 95.00th=[ 5080],
 | 99.00th=[ 12780], 99.50th=[ 20579], 99.90th=[ 54789], 99.95th=[ 81265],
 | 99.99th=[156238]
 bw ( KiB/s): min= 7, max=152247, per=26.07%, avg=18271.75, stdev=20053.83, samples=1723
 iops : min= 1, max=38061, avg=4567.57, stdev=5013.43, samples=1723
 write: IOPS=2920, BW=11.4MiB/s (11.0MB/s)(10.0GiB/900001msec); 0 zone resets
 slat (usec): min=4, max=2554.6k, avg=143.21, stdev=2515.70
 clat (usec): min=3, max=16029k, avg=2034.65, stdev=36777.34
 lat (usec): min=121, max=16029k, avg=2178.19, stdev=36931.23
 clat percentiles (usec):
 | 1.00th=[ 184], 5.00th=[ 219], 10.00th=[ 245], 20.00th=[ 347],
 | 30.00th=[ 611], 40.00th=[ 938], 50.00th=[ 1287], 60.00th=[ 1663],
 | 70.00th=[ 2073], 80.00th=[ 2606], 90.00th=[ 3556], 95.00th=[ 4948],
 | 99.00th=[ 12125], 99.50th=[ 20055], 99.90th=[ 53216], 99.95th=[ 80217],
 | 99.99th=[156238]
 bw ( KiB/s): min= 7, max=101357, per=26.06%, avg=12174.06, stdev=13378.53, samples=1722
 iops : min= 1, max=25339, avg=3043.14, stdev=3344.63, samples=1722
 lat (usec) : 4=0.01%, 250=10.55%, 500=15.05%, 750=8.30%, 1000=7.10%
 lat (msec) : 2=26.45%, 4=24.60%, 10=6.55%, 20=0.88%, 50=0.40%
 lat (msec) : 100=0.08%, 250=0.03%, 500=0.01%, 750=0.01%, 1000=0.01%
 lat (msec) : 2000=0.01%, >=2000=0.01%
 cpu : usr=2.98%, sys=19.22%, ctx=4133616, majf=0, minf=22
 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
 submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
 complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
 issued rwts: total=3946881,2628140,0,0 short=0,0,0,0 dropped=0,0,0,0
 latency : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
 READ: bw=68.5MiB/s (71.8MB/s), 17.1MiB/s-17.1MiB/s (17.9MB/s-17.0MB/s), io=60.2GiB (64.6GB), run=900001-900001msec
 WRITE: bw=45.6MiB/s (47.8MB/s), 11.4MiB/s-11.4MiB/s (11.9MB/s-11.0MB/s), io=40.1GiB (43.1GB), run=900001-900001msec

Disk stats (read/write):
 dm-5: ios=4095694/8382399, merge=0/0, ticks=944686/590705838, in_queue=591655521, util=98.08%, aggrios=4097011/8385705, aggrmerge=0/0, aggrticks=933147/590913671, aggrin_queue=591851814, aggrutil=98.14%
 dm-0: ios=4097011/8385705, merge=0/0, ticks=933147/590913671, in_queue=591851814, util=98.14%, aggrios=4097011/7853564, aggrmerge=0/532163, aggrticks=787654/937480, aggrin_queue=148997, aggrutil=98.17%
 sdc: ios=4097011/7853564, merge=0/532163, ticks=787654/937480, in_queue=148997, util=98.17%

fs_mark on same machines:

btrfs filesystem:

ssd arryy:

[h1]fs_mark -D 1024 -N 1024 -L 100 -n 1024 -p 128 -r 32 -s 4096 -t 64 -d .[/h1]

[h1]Version 3.3, 64 thread(s) starting at Thu Apr 16 23:14:11 2020[/h1]

[h1]Sync method: INBAND FSYNC: fsync() per file in write loop.[/h1]

[h1]Directories: Round Robin between directories across 1024 subdirectories with 1024 files per subdirectory.[/h1]

[h1]File names: 128 bytes long, (96 initial bytes of time stamp with 32 random bytes at end of name)[/h1]

[h1]Files info: size 4096 bytes, written with an IO size of 16384 bytes per write[/h1]

[h1]App overhead is time in microseconds spent in the test not doing file writing related system calls.[/h1]

#
FSUse% Count Size Files/sec App Overhead
 1 65536 4096 10310.7 724416
 1 131072 4096 10780.5 727404
 1 196608 4096 10841.3 727947
 1 262144 4096 9799.0 747258
 1 327680 4096 7313.5 824792
 1 393216 4096 4994.7 902028
 1 458752 4096 4586.5 968847
 1 524288 4096 4771.6 875701
 1 589824 4096 4902.5 964081
 1 655360 4096 4981.5 922469
 1 720896 4096 4953.7 953278
 1 786432 4096 4940.7 970283
 1 851968 4096 4905.0 960853
 1 917504 4096 4866.4 975005
 1 983040 4096 4904.3 961047
 1 1048576 4096 4470.7 971468
 1 1114112 4096 4881.0 928539
 1 1179648 4096 4898.9 985251
 1 1245184 4096 4426.8 998318
 1 1310720 4096 4717.5 1024192
 1 1376256 4096 4795.5 1010882
 1 1441792 4096 4768.4 948396
 1 1507328 4096 4881.9 930210
 1 1572864 4096 4706.8 926804
 1 1638400 4096 4362.5 1014378
 1 1703936 4096 4165.5 1017512
 1 1769472 4096 4454.5 979459
 1 1835008 4096 4414.1 910763
 1 1900544 4096 4584.9 935274
 1 1966080 4096 4245.9 974326
 1 2031616 4096 4192.8 940469
 1 2097152 4096 4390.3 940586
 1 2162688 4096 4618.7 929184
 1 2228224 4096 4399.2 926809
 1 2293760 4096 4120.1 1197906
 1 2359296 4096 4321.8 975645
 1 2424832 4096 4282.7 946189
 1 2490368 4096 4406.6 998897
 1 2555904 4096 4323.5 970225
 1 2621440 4096 4433.8 929924
 1 2686976 4096 3859.8 972714
 1 2752512 4096 3277.5 959557
 1 2818048 4096 3333.6 961802
 1 2883584 4096 3387.0 929832
 1 2949120 4096 3329.5 947360
 1 3014656 4096 3342.3 952607
 1 3080192 4096 3352.9 951114
 1 3145728 4096 3364.2 1045857
 1 3211264 4096 3262.2 993562
 1 3276800 4096 3347.9 984814
 1 3342336 4096 3311.6 1007202
 1 3407872 4096 3302.7 1014171
 1 3473408 4096 3339.7 986689
 1 3538944 4096 3327.9 1179439
 1 3604480 4096 3297.4 1012885
 1 3670016 4096 3315.7 955640
 1 3735552 4096 3273.4 991597
 1 3801088 4096 3271.2 949219
 1 3866624 4096 3187.1 1085862
 1 3932160 4096 3306.2 1034104
 1 3997696 4096 3243.0 1086646
 1 4063232 4096 3118.9 1064811
 1 4128768 4096 3153.9 985299
 1 4194304 4096 3169.8 1043782
 1 4259840 4096 3151.4 1005640
 1 4325376 4096 2933.7 1152361
 1 4390912 4096 3182.4 985131
 1 4456448 4096 3104.6 1024866
 1 4521984 4096 3049.8 975983
 1 4587520 4096 3011.5 1061210
 1 4653056 4096 2894.0 1071062
 1 4718592 4096 3019.0 986460
 1 4784128 4096 3084.8 1041762
 1 4849664 4096 3151.8 1062671
 1 4915200 4096 3152.9 994083
 1 4980736 4096 2962.9 1082241
 1 5046272 4096 3207.6 974949
 1 5111808 4096 2941.4 1251538
 1 5177344 4096 3095.8 1104318
 1 5242880 4096 3130.1 1037710
 1 5308416 4096 3056.7 1010320
 1 5373952 4096 2874.9 1013132
 1 5439488 4096 3155.1 1036078
 1 5505024 4096 3194.9 932772
 1 5570560 4096 3162.1 1013478
 1 5636096 4096 3027.3 1000038
 1 5701632 4096 3064.4 976337
 1 5767168 4096 3114.2 998867
 1 5832704 4096 3038.4 1022219
 1 5898240 4096 3135.7 1063486
 1 5963776 4096 3113.2 974104
 1 6029312 4096 2926.5 1044568
 1 6094848 4096 3022.6 938138
 1 6160384 4096 3080.7 946747
 1 6225920 4096 3035.1 1041103
 1 6291456 4096 2757.0 1070905
 1 6356992 4096 3095.7 1002116
 1 6422528 4096 2960.9 996828
 1 6488064 4096 2834.2 1067899
 1 6553600 4096 3015.3 1025490

xfs filesystem:

ssd array:

[h1]fs_mark -D 1024 -N 1024 -L 100 -n 1024 -p 128 -r 32 -s 4096 -t 64 -d test[/h1]

[h1]Version 3.3, 64 thread(s) starting at Thu Apr 16 23:48:42 2020[/h1]

[h1]Sync method: INBAND FSYNC: fsync() per file in write loop.[/h1]

[h1]Directories: Round Robin between directories across 1024 subdirectories with 1024 files per subdirectory.[/h1]

[h1]File names: 128 bytes long, (96 initial bytes of time stamp with 32 random bytes at end of name)[/h1]

[h1]Files info: size 4096 bytes, written with an IO size of 16384 bytes per write[/h1]

[h1]App overhead is time in microseconds spent in the test not doing file writing related system calls.[/h1]

#
FSUse% Count Size Files/sec App Overhead
 54 65536 4096 7999.0 648553
 54 131072 4096 7017.5 669987
 54 196608 4096 4130.0 800848
 54 262144 4096 3797.9 843377
 54 327680 4096 3129.0 847898
 54 393216 4096 3077.2 873509
 54 458752 4096 2738.3 927161
 54 524288 4096 2994.4 861687
 55 589824 4096 3123.8 822056
 55 655360 4096 3166.8 851053
 55 720896 4096 3080.1 814271
 55 786432 4096 2995.3 874025
 55 851968 4096 3134.7 825648
 55 917504 4096 3108.9 838706
 55 983040 4096 2922.5 882262
 55 1048576 4096 3082.5 844851
 55 1114112 4096 2960.5 939395
 56 1179648 4096 2942.2 873126
 56 1245184 4096 2891.3 932275
 56 1310720 4096 2938.7 868676
 56 1376256 4096 3028.6 823515
 56 1441792 4096 2754.5 865692
 56 1507328 4096 2971.4 846978
 56 1572864 4096 2919.7 862855
 56 1638400 4096 2937.6 884359
 56 1703936 4096 2809.8 920798
 57 1769472 4096 2917.3 851571
 57 1835008 4096 2804.6 887153
 57 1900544 4096 2707.8 889684
 57 1966080 4096 2894.6 910081
 57 2031616 4096 2930.0 973586
 57 2097152 4096 2736.2 873193
 57 2162688 4096 2596.3 1074422
 58 2228224 4096 2586.6 1037621
 58 2293760 4096 2646.0 907799
 58 2359296 4096 2746.2 909818
 58 2424832 4096 2897.5 874643
 58 2490368 4096 2884.1 889218
 58 2555904 4096 2889.7 829827
 58 2621440 4096 2723.1 980979
 59 2686976 4096 2884.7 849886
 59 2752512 4096 2804.5 892372
 59 2818048 4096 2821.7 824137
 59 2883584 4096 2837.9 855938
 59 2949120 4096 2874.3 925673
 59 3014656 4096 2882.0 867375
 59 3080192 4096 2662.0 904706
 59 3145728 4096 2909.3 852527
 59 3211264 4096 2821.0 866250
 60 3276800 4096 2979.5 853112
 60 3342336 4096 2882.4 957123
 59 3407872 4096 2735.5 870543
 60 3473408 4096 2784.8 859071
 60 3538944 4096 2957.5 911676
 60 3604480 4096 2775.3 885856
 60 3670016 4096 2778.3 940815
 60 3735552 4096 2989.9 867906
 60 3801088 4096 2880.0 910774
 61 3866624 4096 2833.5 861072
 61 3932160 4096 2783.7 910543
 61 3997696 4096 2740.0 916634
 61 4063232 4096 2692.5 928860
 61 4128768 4096 2688.5 935526
 61 4194304 4096 2691.0 907043
 61 4259840 4096 2741.5 916414
 61 4325376 4096 2787.0 951843
 61 4390912 4096 2798.5 898106
 62 4456448 4096 2878.6 870864
 62 4521984 4096 2927.3 912141
 62 4587520 4096 2868.6 976210
 62 4653056 4096 2923.0 877059
 62 4718592 4096 2739.4 899465
 62 4784128 4096 2693.2 1005320
 62 4849664 4096 2614.0 965127
 62 4915200 4096 2826.6 895421
 62 4980736 4096 2867.8 865318
 63 5046272 4096 2833.4 896990
 63 5111808 4096 2823.4 896109
 63 5177344 4096 2763.7 880317
 63 5242880 4096 2802.1 865805
 63 5308416 4096 2828.5 874843
 63 5373952 4096 2808.4 855479
 63 5439488 4096 2890.9 871179
 63 5505024 4096 2820.4 958657
 63 5570560 4096 2878.4 842219
 64 5636096 4096 2793.5 832474
 64 5701632 4096 2783.6 872805
 64 5767168 4096 2653.2 874041
 64 5832704 4096 2661.8 905106
 64 5898240 4096 2859.8 864755
 64 5963776 4096 2863.5 879432
 64 6029312 4096 2899.7 862990
 64 6094848 4096 2641.9 929778
 64 6160384 4096 2675.6 851607
 65 6225920 4096 2671.1 935284
 65 6291456 4096 2759.3 865565
 65 6356992 4096 2637.7 904031
 65 6422528 4096 2460.4 941102
 65 6488064 4096 2738.5 895132
 65 6553600 4096 2543.7 969335

ext4 filesystem:

ssd array:

[h1]fs_mark -D 1024 -N 1024 -L 100 -n 1024 -p 128 -r 32 -s 4096 -t 64 -d test[/h1]

[h1]Version 3.3, 64 thread(s) starting at Fri Apr 17 01:09:51 2020[/h1]

[h1]Sync method: INBAND FSYNC: fsync() per file in write loop.[/h1]

[h1]Directories: Round Robin between directories across 1024 subdirectories with 1024 files per subdirectory.[/h1]

[h1]File names: 128 bytes long, (96 initial bytes of time stamp with 32 random bytes at end of name)[/h1]

[h1]Files info: size 4096 bytes, written with an IO size of 16384 bytes per write[/h1]

[h1]App overhead is time in microseconds spent in the test not doing file writing related system calls.[/h1]

#
FSUse% Count Size Files/sec App Overhead
 5 65536 4096 7864.5 552086
 5 131072 4096 6016.0 659293
 5 196608 4096 3132.8 830952
 5 262144 4096 3111.3 868417
 5 327680 4096 3024.1 856514
 5 393216 4096 3054.6 855310
 5 458752 4096 2965.3 855742
 5 524288 4096 2939.1 848230
 5 589824 4096 2954.9 996805
 5 655360 4096 2777.9 902412
 5 720896 4096 2888.6 862273
 5 786432 4096 2896.1 824348
 5 851968 4096 2913.1 845055
 5 917504 4096 2890.0 880318
 5 983040 4096 2894.7 857438
 5 1048576 4096 2801.2 816060
 5 1114112 4096 2911.2 841817
 5 1179648 4096 2713.7 844881
 5 1245184 4096 2906.6 838485
 5 1310720 4096 2680.9 887251
 5 1376256 4096 2699.2 878854
 5 1441792 4096 2853.2 833365
 5 1507328 4096 2607.5 920782
 5 1572864 4096 2696.1 851208
 5 1638400 4096 2855.7 843803
 5 1703936 4096 2481.7 992642
 5 1769472 4096 2775.0 826382
 5 1835008 4096 2882.5 839825
 5 1900544 4096 2781.9 833644
 5 1966080 4096 2847.1 837430
 5 2031616 4096 2425.9 949966
 5 2097152 4096 2765.9 861270
 5 2162688 4096 2645.1 861435
 5 2228224 4096 2643.1 872495
 5 2293760 4096 2696.8 865095
 5 2359296 4096 2788.9 821588
 5 2424832 4096 2701.5 854663
 5 2490368 4096 2720.7 845920
 5 2555904 4096 2715.9 887622
 5 2621440 4096 2677.9 911478
 5 2686976 4096 2769.2 849543
 5 2752512 4096 2724.9 904973
 5 2818048 4096 2750.3 844592
 5 2883584 4096 2754.1 825710
 5 2949120 4096 2773.4 834391
 5 3014656 4096 2494.8 926544
 5 3080192 4096 2646.0 856238
 5 3145728 4096 2618.4 833934
 5 3211264 4096 2773.9 839651
 5 3276800 4096 2589.9 850756
 5 3342336 4096 2657.2 831601
 5 3407872 4096 2370.0 960856
 5 3473408 4096 2706.7 817512
 5 3538944 4096 2797.4 797210
 5 3604480 4096 2529.4 899022
 5 3670016 4096 2596.7 815678
 5 3735552 4096 2711.9 819621
 5 3801088 4096 2631.6 851691
 5 3866624 4096 2751.4 842720
 5 3932160 4096 2565.7 855631
 5 3997696 4096 2778.3 795482
 5 4063232 4096 2723.5 889595
 5 4128768 4096 2658.2 861042
 5 4194304 4096 2699.9 819071
 5 4259840 4096 2644.2 852887
 5 4325376 4096 2754.3 807878
 5 4390912 4096 2692.8 840796
 5 4456448 4096 2819.3 807838
 5 4521984 4096 2690.4 835833
 5 4587520 4096 2778.1 846772
 5 4653056 4096 2629.5 930968
 5 4718592 4096 2708.7 831528
 5 4784128 4096 2558.6 846661
 5 4849664 4096 2663.4 898813
 5 4915200 4096 2591.5 895358
 5 4980736 4096 2765.3 853465
 5 5046272 4096 2539.0 875678
 5 5111808 4096 2720.3 864972
 5 5177344 4096 2757.4 834596
 5 5242880 4096 2640.7 842025
 5 5308416 4096 2686.0 837971
 5 5373952 4096 2642.4 869985
 5 5439488 4096 2744.4 800594
 5 5505024 4096 2736.5 941328
 5 5570560 4096 2649.0 850955
 5 5636096 4096 2729.2 821149
 5 5701632 4096 2648.9 861357
 5 5767168 4096 2548.3 876694
 5 5832704 4096 2525.8 893559
 5 5898240 4096 2703.9 848639
 5 5963776 4096 2553.9 864792
 5 6029312 4096 2548.8 874867
 5 6094848 4096 2592.4 884055
 5 6160384 4096 2548.6 868029
 5 6225920 4096 2667.0 828056
 5 6291456 4096 2559.1 881193
 5 6356992 4096 2661.5 847813
 5 6422528 4096 2501.0 838245
 5 6488064 4096 2674.3 829347
 5 6553600 4096 2566.9 860537

linux filesystems xfs btrfs ext4 lvm raid mdadm luks cryptsetup benchmark

installation

Windows: (simple enough, most can handle it without contacting paid support)

Linux (and other unix systems): (impossible)

Initial system configuration

Windows: (simple, straightforward, easy to understand)

Linux (and other unix systems): (very hard, nearly impossible)

Troubleshooting

Windows: (simple, straightforward, easy to understand)

Conclusion: