so, after using linux since early 200x as my main os for work i can say what linux is shit, hard to understand, not user friendly, complicated shit, i found windows much more simple and user friendly, especially now-days, let me explain in details:
here we have two options
1) but pc with windows preinstalled (simplest solution)
- to use this method you need to go to nearby pc/notebook/phone store and just by what you like, that's it, you done installation and may use your windows now.
2) manual install (for advanced users)
- you need to by installation optical media
- you need to put installation optical media to your optical drive
- you need to boot from your optical drive (this is most complicated part, you may even need to go to uefi/bios to achieve this)
- you will boot into graphical shell where you need few time press "next" "accept" or "ok" button near right bottom corner of screen
- installation must be done, if it is not, or machine does not boot into windows you can use installation method 1, or buy new pc,
- that's all. it is as simple as this
3) advanced installation methods
to be more user friendly and simple windows does not support advanced installation methods and does not support unusual configurations such as network boot, boot from encrypted media with external key and all other unneeded complicated things, with exceptions for server editions which users does not need and do not licensed and do not wanted to use.
windows installation is simple enough for normal user, to not overthink yourself you got just two simple options too choose from.
with linux you have problems right after a moment of time you decided to install linux, first of all it have a LOT methods of installation, and all is bad and complicated, most does not work at all, many involving undesired brain activity called "learning", can you imagine it ?, so, let us proceed to installation methods explanation
1) buy pc with linux preinstalled (hard)
surprisingly somewhere near 2010+ this options become available too, but it is relatively hard to find pc with linux preinstalled in store, and even if you find one, most probably large part of bought pc's hardware will not work at all or work incorrectly, system configuration and problems resolving will be explained later.
2) manual install using installation media (very hard)
a big variety of installation media exists, it may be optical media, regular block device image for booting from block devices or even network boot with pxe, most normal users does not even understand this heresy typed above.
- you need to choose what you want to install, as explained above it is hard, and even involving undesired brain activities called "thinking", "learning"
- because of linux does not have unified graphical installer which used for every distro and have ability to use non-trivial complicated setups next steps are unpredictable and may vary, also involving undesired brain activities such as "learning", "reading", "thinking" which is unacceptable for normal users
3) redeploying existing installed system to new pc (impossible, myth)
tales and legends heard from time to time about some strange mythical thing like no installing at all but just moving installed system to new pc done by powerful unix adepts and other unholy mythical persons, it's scary just think about it.
i will not explain further as it may permanently damage brain of normal sane user, but even this little part should be enough to understand what linux installation is a pain and does not worth it.
1) drivers
- in case of manual installation you most probably need to install drivers for your pc, but do not be scared, you do not need to disturb your precious brain for this task. most probably you have some optical disks which come with your pc, you need to do following steps for each of your disks bundled with pc: a) insert disk in your optical drive. b) run every executable file with admistrator privileges, if some windows appears you need to press following buttons "next" "accept" "ok" "reboot", each small operation most probably require reboot or two, it's perfectly ok, if during process you pc hang (freeze) just hit reset button or hold power button of your notebook, it is also perfectly ok for windows to solve problems this way, you may need to repeat it like 10 times and problem should go away. c) if after this operations your system still lacks some drivers or does not behave properly you just need to buy new pc and repeat beginning from installation, it is as simple as that.
- if during previous actions your system start to constantly boot into mode called "BSOD" (blue screen with some strange numbers) it is also perfectly ok, just do installation from the beginning and do driver installation in different order, yes solving problems called "constant BSOD during boot" is as simple as this, just do full reinstall and it will gone, if not, buy new PC.
2) necessary programs
- everything you need is already included, also you can use a lot of services from microsoft online with your microsoft account, so you do not need install everything at all, in soce rare case microsoft does not provide needed service, in this case you can use online service from google with your google account for your task, but in very rare case where something missed, windows 10 introduced windows store where you can by everything, in very rare and very special case if you need some very special software for work, you can just contact software vendors and buy it directly, i suggest buy from vendors like adobe, you can call them from cell phone, yes, it's just as simple as this.
1) boot
often it is impossible to install system at all, and after you do all perfectly fine, your system just does not boot at all, you will need to figure out what wrong all by yourself, which will for sure involve a lot of undesired brain activities like "learning", "thinking", "understanding", you may even need to interact with unix communities where unholly unix adepts will most probably laught at you and not help at all, will say some unholy meaningless words like "man", "bootloader", "grub", "efi" and a lot of other nonsense, belive me, it's meaningless words impossible to understand and they just making fun of you, they may even try to force you read large complicated technical articles, documents and manuals, but reading does not help you at all, they just making fun of you again, also you can't use windows way to solve all and every problem called "reset button" or "full reinstall", here it does not solve anything, so if you are not lucky you will stuck here, kill a lot of time before you finally understand what you just need to go windows way.
2) boot?
if in very rare case after you turn on you pc something is happening, often instead of your desktop you see some nonsense on screen, a lot of text fast runing through screan and after few seconds just empty screen with some text ending with "login:" it's absolutely meaningless and nothing you can do about it, you should try to install different linux version and pray to have desktop after this hard and comlicated install process or you can try contact unix community, but as mentioned earlier they will just make fun of you with sensles words and large compplicated sensless texts, so better just go windows way before it's too late.
3) boot??
it is exists a very tiny possibility what after installation your desktop will appear.
4) necessary programs
but wait, what is this ?, where to input your microsoft account login and password ?, where is my start menu ?, all is worng, you do not have your programs, you can't check what drivers installed, drivers from optical discs does not installing at all, you can't use your adobe photoshop bought for a large price, it's just not installing at all, you do not have your favorite windows media player here or even internet explorer or microsoft edge, you can't connect to network, all your precious software downloaded from torrents does not installs at all, and not only pirated, but paid software too, so now you are realized what linux is shit yes ?, just go windows way already. you do not need more pain.
1) performance problems (simple)
- if you have problems with performance, reboot you pc (this may be needed up to 10 or more times, it's perfectly ok), yes, it's that simple
- if step above does not solve your issue, do full system reinstall, it should resolve your issue, yes. it's as simple as this.
- if step above does not solve your issue, buy new pc, yes, just as simple,
2) stability problems (simple, straightforward, easy to understand)
- solutions are exactly same as for performance problems, yes, it is that simple, one set of solutions for every kind of problem to keep users mind free.
Linux (and other unix systems): (impossible)
1) performance problems (impossible)
- if you have performance problems with linux nor reboot nor reinstall does nothing, just does not help, you can try buy new pc but it will also most probably not help
- in linux used hard complicated approach to solving any kind of problems, you will be force to do a lot of undesired brain activites such as "reading", "learning", "understanding", "thinking"m this is a real hell belive me, in worst cases you may even need to talk with linux developers directly, you will be forced to understand how system works and how problemstic parts of system works, you will be forced to spend your precious time which is dedicated for porno, games and slacking on lot of undesired brain activities instead, and when you think problem is finally solved, another problem will appears, and you will see no end for this, ever, in worst cases you will be forced to learn programming and scripting languages instead of drinking beer with your friends, this is a hell with no way out, belive me and go windows way !
2) stability problems (impossible)
- same as above, hitting reset and syste, reinstalls does not helping at all.
- same as above
soo....., just a friendly advice, before you ruined your life like me, just turn around and run, run..., run..., run windows way.
if something goes wrong you can always hit reset button few times to solve any kind of problems
in case problem not solved you can always reinstall windows
if reinstalling also not solved problem, you can always buy new pc with windows preinstalled.
do not ruin your life, it's time to reinstall windows !
installation
Windows: (simple enough, most can handle it without contacting paid support)
here we have two options
1) but pc with windows preinstalled (simplest solution)
- to use this method you need to go to nearby pc/notebook/phone store and just by what you like, that's it, you done installation and may use your windows now.
2) manual install (for advanced users)
- you need to by installation optical media
- you need to put installation optical media to your optical drive
- you need to boot from your optical drive (this is most complicated part, you may even need to go to uefi/bios to achieve this)
- you will boot into graphical shell where you need few time press "next" "accept" or "ok" button near right bottom corner of screen
- installation must be done, if it is not, or machine does not boot into windows you can use installation method 1, or buy new pc,
- that's all. it is as simple as this
3) advanced installation methods
to be more user friendly and simple windows does not support advanced installation methods and does not support unusual configurations such as network boot, boot from encrypted media with external key and all other unneeded complicated things, with exceptions for server editions which users does not need and do not licensed and do not wanted to use.
windows installation is simple enough for normal user, to not overthink yourself you got just two simple options too choose from.
Linux (and other unix systems): (impossible)
with linux you have problems right after a moment of time you decided to install linux, first of all it have a LOT methods of installation, and all is bad and complicated, most does not work at all, many involving undesired brain activity called "learning", can you imagine it ?, so, let us proceed to installation methods explanation
1) buy pc with linux preinstalled (hard)
surprisingly somewhere near 2010+ this options become available too, but it is relatively hard to find pc with linux preinstalled in store, and even if you find one, most probably large part of bought pc's hardware will not work at all or work incorrectly, system configuration and problems resolving will be explained later.
2) manual install using installation media (very hard)
a big variety of installation media exists, it may be optical media, regular block device image for booting from block devices or even network boot with pxe, most normal users does not even understand this heresy typed above.
- you need to choose what you want to install, as explained above it is hard, and even involving undesired brain activities called "thinking", "learning"
- because of linux does not have unified graphical installer which used for every distro and have ability to use non-trivial complicated setups next steps are unpredictable and may vary, also involving undesired brain activities such as "learning", "reading", "thinking" which is unacceptable for normal users
3) redeploying existing installed system to new pc (impossible, myth)
tales and legends heard from time to time about some strange mythical thing like no installing at all but just moving installed system to new pc done by powerful unix adepts and other unholy mythical persons, it's scary just think about it.
i will not explain further as it may permanently damage brain of normal sane user, but even this little part should be enough to understand what linux installation is a pain and does not worth it.
Initial system configuration
Windows: (simple, straightforward, easy to understand)
1) drivers
- in case of manual installation you most probably need to install drivers for your pc, but do not be scared, you do not need to disturb your precious brain for this task. most probably you have some optical disks which come with your pc, you need to do following steps for each of your disks bundled with pc: a) insert disk in your optical drive. b) run every executable file with admistrator privileges, if some windows appears you need to press following buttons "next" "accept" "ok" "reboot", each small operation most probably require reboot or two, it's perfectly ok, if during process you pc hang (freeze) just hit reset button or hold power button of your notebook, it is also perfectly ok for windows to solve problems this way, you may need to repeat it like 10 times and problem should go away. c) if after this operations your system still lacks some drivers or does not behave properly you just need to buy new pc and repeat beginning from installation, it is as simple as that.
- if during previous actions your system start to constantly boot into mode called "BSOD" (blue screen with some strange numbers) it is also perfectly ok, just do installation from the beginning and do driver installation in different order, yes solving problems called "constant BSOD during boot" is as simple as this, just do full reinstall and it will gone, if not, buy new PC.
2) necessary programs
- everything you need is already included, also you can use a lot of services from microsoft online with your microsoft account, so you do not need install everything at all, in soce rare case microsoft does not provide needed service, in this case you can use online service from google with your google account for your task, but in very rare case where something missed, windows 10 introduced windows store where you can by everything, in very rare and very special case if you need some very special software for work, you can just contact software vendors and buy it directly, i suggest buy from vendors like adobe, you can call them from cell phone, yes, it's just as simple as this.
Linux (and other unix systems): (very hard, nearly impossible)
1) boot
often it is impossible to install system at all, and after you do all perfectly fine, your system just does not boot at all, you will need to figure out what wrong all by yourself, which will for sure involve a lot of undesired brain activities like "learning", "thinking", "understanding", you may even need to interact with unix communities where unholly unix adepts will most probably laught at you and not help at all, will say some unholy meaningless words like "man", "bootloader", "grub", "efi" and a lot of other nonsense, belive me, it's meaningless words impossible to understand and they just making fun of you, they may even try to force you read large complicated technical articles, documents and manuals, but reading does not help you at all, they just making fun of you again, also you can't use windows way to solve all and every problem called "reset button" or "full reinstall", here it does not solve anything, so if you are not lucky you will stuck here, kill a lot of time before you finally understand what you just need to go windows way.
2) boot?
if in very rare case after you turn on you pc something is happening, often instead of your desktop you see some nonsense on screen, a lot of text fast runing through screan and after few seconds just empty screen with some text ending with "login:" it's absolutely meaningless and nothing you can do about it, you should try to install different linux version and pray to have desktop after this hard and comlicated install process or you can try contact unix community, but as mentioned earlier they will just make fun of you with sensles words and large compplicated sensless texts, so better just go windows way before it's too late.
3) boot??
it is exists a very tiny possibility what after installation your desktop will appear.
4) necessary programs
but wait, what is this ?, where to input your microsoft account login and password ?, where is my start menu ?, all is worng, you do not have your programs, you can't check what drivers installed, drivers from optical discs does not installing at all, you can't use your adobe photoshop bought for a large price, it's just not installing at all, you do not have your favorite windows media player here or even internet explorer or microsoft edge, you can't connect to network, all your precious software downloaded from torrents does not installs at all, and not only pirated, but paid software too, so now you are realized what linux is shit yes ?, just go windows way already. you do not need more pain.
Troubleshooting
Windows: (simple, straightforward, easy to understand)
1) performance problems (simple)
- if you have problems with performance, reboot you pc (this may be needed up to 10 or more times, it's perfectly ok), yes, it's that simple
- if step above does not solve your issue, do full system reinstall, it should resolve your issue, yes. it's as simple as this.
- if step above does not solve your issue, buy new pc, yes, just as simple,
2) stability problems (simple, straightforward, easy to understand)
- solutions are exactly same as for performance problems, yes, it is that simple, one set of solutions for every kind of problem to keep users mind free.
Linux (and other unix systems): (impossible)
1) performance problems (impossible)
- if you have performance problems with linux nor reboot nor reinstall does nothing, just does not help, you can try buy new pc but it will also most probably not help
- in linux used hard complicated approach to solving any kind of problems, you will be force to do a lot of undesired brain activites such as "reading", "learning", "understanding", "thinking"m this is a real hell belive me, in worst cases you may even need to talk with linux developers directly, you will be forced to understand how system works and how problemstic parts of system works, you will be forced to spend your precious time which is dedicated for porno, games and slacking on lot of undesired brain activities instead, and when you think problem is finally solved, another problem will appears, and you will see no end for this, ever, in worst cases you will be forced to learn programming and scripting languages instead of drinking beer with your friends, this is a hell with no way out, belive me and go windows way !
2) stability problems (impossible)
- same as above, hitting reset and syste, reinstalls does not helping at all.
- same as above
Conclusion:
soo....., just a friendly advice, before you ruined your life like me, just turn around and run, run..., run..., run windows way.
if something goes wrong you can always hit reset button few times to solve any kind of problems
in case problem not solved you can always reinstall windows
if reinstalling also not solved problem, you can always buy new pc with windows preinstalled.
do not ruin your life, it's time to reinstall windows !
small linux block devices performance test
#linux #filesystems #xfs #btrfs #ext4 #lvm #raid #mdadm #luks #cryptsetup
linux io test on different storage and diferent filesystems with fio:
NOTES:
tests done on different hw configurations, on devices wrapped by mdadm raid0, cryptsetup luks crypto and lvm, parameters, crypto and raw io performance will be shown for each test.
machine 1:
storage:
2 x Samsung SSD 860 EVO 500GB
3 x WDC WD40PURZ-85TTDY0
storage layout:
ssd array: 2 x ssd mdadm raid 0 + cryptsetup luks with aes-xts-plain64 + lvm
hdd array: 3 x hdd mdadm raid 0 + cryptsetup luks with camellia-xts-plain64 + lvm
raw linear read io performance with hdparm -Tt (to understand bus limitations):
ssd array:
hdd array:
crypto performance:
xfs filesystem tests:
fio-rand-RW.fio:
ssd array:
NOTE: filesystem BLOCK_SIZE=”512”
hdd array:
ssd-test.fio:
ssd array:
NOTE: filesystem BLOCK_SIZE=”512”
fio-seq-RW.fio:
ssd array:
NOTE: filesystem BLOCK_SIZE=”512”
btrfs filesystem tests:
fio-rand-RW.fio:
ssd array:
NOTE: deleting 10gb test file tooke like one minute or more on btrfs.
ssd-test.fio:
ssd array:
fio-seq-RW.fio:
ssd array:
ext4 filesystem tests:
fio-rand-RW.fio:
ssd array:
ssd-test.fio:
ssd array:
fio-seq-RW.fio:
ssd array:
machine 2:
storage: WDC WDS500G2B0A-00SM50 (ssd)
storage layout:
cryptsetup luks with camellia-xts-plain64 + lvm
raw linear read io performance with hdparm -Tt (to understand bus limitations):
for some reason very low speed here, probably because of external sata3 card
02:00.0 SATA controller: ASMedia Technology Inc. Device 0622 (rev 01)
smartctl show 6gb link speed here:
SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
anyway, let’s continue…
crypto performance:
obvously camellia is not the best choice here for crypto, probably i should switch to twofish or serpent in future
xfs filesystem tests:
fio-rand-RW.fio:
machine 3:
storage:
WDC WDS500G2B0A-00SM50 (ssd)
2 x WDC WD10EZRX-00L4HB0
storage layout:
ssd: cryptsetup luks with aes-xts-plain64 + lvm
hdd array: 2x hdd mdadm raid0 + cryptsetup luks with camellia-xts-plain64 + lvm
raw linear read io performance with hdparm -Tt (to understand bus limitations):
ssd:
hdd array:
crypto performance:
xfs filesystem tests:
fio-rand-RW.fio:
ssd:
hdd array:
machine 4:
NOTE: tests on this machine done under some light io load which cannot be avoided, so they are not very accurate
storage:
SanDisk SDSSDH3 500G (ssd)
storage layout:
ssd: cryptsetup luks with aes-xts-plain64 + lvm
raw linear read io performance with hdparm -Tt (to understand bus limitations):
ssd:
crypto performance:
xfs filesystem tests:
fio-rand-RW.fio:
ssd array:
fs_mark on same machines:
btrfs filesystem:
ssd arryy:
xfs filesystem:
ssd array:
ext4 filesystem:
ssd array:
#linux #filesystems #xfs #btrfs #ext4 #lvm #raid #mdadm #luks #cryptsetup
linux io test on different storage and diferent filesystems with fio:
NOTES:
tests done on different hw configurations, on devices wrapped by mdadm raid0, cryptsetup luks crypto and lvm, parameters, crypto and raw io performance will be shown for each test.
machine 1:
storage:
2 x Samsung SSD 860 EVO 500GB
3 x WDC WD40PURZ-85TTDY0
storage layout:
ssd array: 2 x ssd mdadm raid 0 + cryptsetup luks with aes-xts-plain64 + lvm
hdd array: 3 x hdd mdadm raid 0 + cryptsetup luks with camellia-xts-plain64 + lvm
raw linear read io performance with hdparm -Tt (to understand bus limitations):
ssd array:
Timing cached reads: 22102 MB in 2.00 seconds = 11064.06 MB/sec
Timing buffered disk reads: 3108 MB in 3.00 seconds = 1035.40 MB/sec
hdd array:
Timing cached reads: 23258 MB in 2.00 seconds = 11643.60 MB/sec
Timing buffered disk reads: 1522 MB in 3.00 seconds = 506.96 MB/sec
crypto performance:
[h1]Tests are approximate using memory only (no storage IO).[/h1]
[h1]Algorithm | Key | Encryption | Decryption[/h1]
twofish-xts 512b 353.1 MiB/s 357.0 MiB/s
[h1]Tests are approximate using memory only (no storage IO).[/h1]
[h1]Algorithm | Key | Encryption | Decryption[/h1]
serpent-xts 512b 612.3 MiB/s 609.4 MiB/s
[h1]Tests are approximate using memory only (no storage IO).[/h1]
[h1]Algorithm | Key | Encryption | Decryption[/h1]
aes-xts 512b 2681.4 MiB/s 2723.0 MiB/s
[h1]Tests are approximate using memory only (no storage IO).[/h1]
[h1]Algorithm | Key | Encryption | Decryption[/h1]
camellia-xts 512b 740.9 MiB/s 755.7 MiB/s
[h1]Tests are approximate using memory only (no storage IO).[/h1]
PBKDF2-ripemd160 200722 iterations per second for 512-bit key
[h1]Tests are approximate using memory only (no storage IO).[/h1]
PBKDF2-sha256 586451 iterations per second for 512-bit key
[h1]Tests are approximate using memory only (no storage IO).[/h1]
PBKDF2-sha512 781353 iterations per second for 512-bit key
[h1]Tests are approximate using memory only (no storage IO).[/h1]
PBKDF2-whirlpool 369737 iterations per second for 512-bit key
xfs filesystem tests:
fio-rand-RW.fio:
ssd array:
NOTE: filesystem BLOCK_SIZE=”512”
file1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.18
Starting 4 processes
file1: Laying out IO file (1 file / 10240MiB)
Jobs: 4 (f=4): [m(4)][100.0%][r=125MiB/s,w=83.3MiB/s][r=31.0k,w=21.3k IOPS][eta 00m:00s]
file1: (groupid=0, jobs=1): err= 0: pid=3927: Wed Apr 15 01:25:51 2020
read: IOPS=14.2k, BW=55.4MiB/s (58.1MB/s)(48.7GiB/900001msec)
slat (nsec): min=611, max=15747M, avg=30942.93, stdev=10093578.28
clat (nsec): min=1002, max=22729M, avg=647744.82, stdev=56483253.23
lat (usec): min=24, max=22729k, avg=678.75, stdev=57420.76
clat percentiles (usec):
| 1.00th=[ 53], 5.00th=[ 62], 10.00th=[ 67], 20.00th=[ 73],
| 30.00th=[ 80], 40.00th=[ 91], 50.00th=[ 198], 60.00th=[ 265],
| 70.00th=[ 404], 80.00th=[ 611], 90.00th=[ 1057], 95.00th=[ 1647],
| 99.00th=[ 3097], 99.50th=[ 3687], 99.90th=[ 5145], 99.95th=[ 5866],
| 99.99th=[10290]
bw ( KiB/s): min= 8, max=542264, per=36.81%, avg=83508.11, stdev=106473.54, samples=1222
iops : min= 2, max=135566, avg=20877.03, stdev=26618.40, samples=1222
write: IOPS=9453, BW=36.9MiB/s (38.7MB/s)(32.5GiB/900001msec); 0 zone resets
slat (nsec): min=1012, max=17920M, avg=51399.19, stdev=13365178.34
clat (usec): min=24, max=22729k, avg=615.97, stdev=53329.84
lat (usec): min=26, max=22729k, avg=667.44, stdev=55364.63
clat percentiles (usec):
| 1.00th=[ 52], 5.00th=[ 62], 10.00th=[ 67], 20.00th=[ 73],
| 30.00th=[ 80], 40.00th=[ 89], 50.00th=[ 186], 60.00th=[ 245],
| 70.00th=[ 379], 80.00th=[ 578], 90.00th=[ 1012], 95.00th=[ 1582],
| 99.00th=[ 3032], 99.50th=[ 3621], 99.90th=[ 5014], 99.95th=[ 5735],
| 99.99th=[ 9372]
bw ( KiB/s): min= 7, max=364976, per=36.96%, avg=55910.94, stdev=71077.22, samples=1217
iops : min= 1, max=91244, avg=13977.74, stdev=17769.31, samples=1217
lat (usec) : 2=0.01%, 50=0.70%, 100=44.30%, 250=14.47%, 500=16.34%
lat (usec) : 750=8.82%, 1000=4.80%
lat (msec) : 2=7.22%, 4=3.01%, 10=0.32%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%, >=2000=0.01%
cpu : usr=2.22%, sys=17.10%, ctx=7748150, majf=0, minf=23
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=12759994,8508306,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=3928: Wed Apr 15 01:25:51 2020
read: IOPS=14.2k, BW=55.4MiB/s (58.1MB/s)(48.7GiB/900001msec)
slat (nsec): min=632, max=14055M, avg=28707.68, stdev=8367525.63
clat (nsec): min=1262, max=22585M, avg=641342.80, stdev=54006164.38
lat (usec): min=25, max=22585k, avg=670.12, stdev=54889.64
clat percentiles (usec):
| 1.00th=[ 52], 5.00th=[ 62], 10.00th=[ 67], 20.00th=[ 74],
| 30.00th=[ 81], 40.00th=[ 91], 50.00th=[ 198], 60.00th=[ 269],
| 70.00th=[ 404], 80.00th=[ 611], 90.00th=[ 1057], 95.00th=[ 1647],
| 99.00th=[ 3097], 99.50th=[ 3687], 99.90th=[ 5145], 99.95th=[ 5866],
| 99.99th=[10028]
bw ( KiB/s): min= 8, max=534096, per=36.83%, avg=83552.06, stdev=106239.96, samples=1221
iops : min= 2, max=133524, avg=20888.01, stdev=26559.98, samples=1221
write: IOPS=9459, BW=36.0MiB/s (38.7MB/s)(32.5GiB/900001msec); 0 zone resets
slat (nsec): min=1002, max=17920M, avg=54725.10, stdev=15045267.29
clat (usec): min=24, max=22585k, avg=625.45, stdev=55917.05
lat (usec): min=26, max=22585k, avg=680.25, stdev=58145.95
clat percentiles (usec):
| 1.00th=[ 51], 5.00th=[ 62], 10.00th=[ 67], 20.00th=[ 73],
| 30.00th=[ 80], 40.00th=[ 89], 50.00th=[ 186], 60.00th=[ 245],
| 70.00th=[ 379], 80.00th=[ 578], 90.00th=[ 1012], 95.00th=[ 1582],
| 99.00th=[ 3032], 99.50th=[ 3589], 99.90th=[ 5014], 99.95th=[ 5735],
| 99.99th=[ 9765]
bw ( KiB/s): min= 7, max=359128, per=37.01%, avg=55994.86, stdev=71016.78, samples=1216
iops : min= 1, max=89782, avg=13998.72, stdev=17754.19, samples=1216
lat (usec) : 2=0.01%, 50=0.85%, 100=44.08%, 250=14.51%, 500=16.41%
lat (usec) : 750=8.79%, 1000=4.79%
lat (msec) : 2=7.23%, 4=3.02%, 10=0.32%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%, >=2000=0.01%
cpu : usr=2.24%, sys=17.08%, ctx=7664942, majf=0, minf=24
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=12756139,8513933,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=3929: Wed Apr 15 01:25:51 2020
read: IOPS=14.2k, BW=55.4MiB/s (58.1MB/s)(48.7GiB/900001msec)
slat (nsec): min=641, max=17920M, avg=30274.08, stdev=10110966.15
clat (usec): min=22, max=22670k, avg=649.54, stdev=55193.99
lat (usec): min=23, max=22670k, avg=679.88, stdev=56204.10
clat percentiles (usec):
| 1.00th=[ 51], 5.00th=[ 62], 10.00th=[ 67], 20.00th=[ 74],
| 30.00th=[ 81], 40.00th=[ 91], 50.00th=[ 198], 60.00th=[ 265],
| 70.00th=[ 404], 80.00th=[ 611], 90.00th=[ 1057], 95.00th=[ 1647],
| 99.00th=[ 3097], 99.50th=[ 3687], 99.90th=[ 5145], 99.95th=[ 5932],
| 99.99th=[10159]
bw ( KiB/s): min= 8, max=537344, per=36.82%, avg=83527.72, stdev=106442.07, samples=1222
iops : min= 2, max=134336, avg=20881.93, stdev=26610.52, samples=1222
write: IOPS=9457, BW=36.9MiB/s (38.7MB/s)(32.5GiB/900001msec); 0 zone resets
slat (nsec): min=991, max=15747M, avg=52380.86, stdev=13341407.56
clat (nsec): min=1192, max=22670M, avg=612775.08, stdev=55147764.50
lat (usec): min=23, max=22670k, avg=665.23, stdev=57123.82
clat percentiles (usec):
| 1.00th=[ 49], 5.00th=[ 61], 10.00th=[ 67], 20.00th=[ 73],
| 30.00th=[ 80], 40.00th=[ 90], 50.00th=[ 186], 60.00th=[ 245],
| 70.00th=[ 375], 80.00th=[ 578], 90.00th=[ 1012], 95.00th=[ 1582],
| 99.00th=[ 3032], 99.50th=[ 3589], 99.90th=[ 5014], 99.95th=[ 5800],
| 99.99th=[ 9503]
bw ( KiB/s): min= 8, max=358760, per=37.03%, avg=56023.15, stdev=71031.30, samples=1215
iops : min= 2, max=89690, avg=14005.79, stdev=17757.82, samples=1215
lat (usec) : 2=0.01%, 50=1.01%, 100=43.83%, 250=14.60%, 500=16.42%
lat (usec) : 750=8.80%, 1000=4.79%
lat (msec) : 2=7.21%, 4=3.02%, 10=0.32%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%, >=2000=0.01%
cpu : usr=2.25%, sys=17.08%, ctx=7573034, majf=0, minf=23
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=12763389,8511685,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=3930: Wed Apr 15 01:25:51 2020
read: IOPS=14.2k, BW=55.4MiB/s (58.1MB/s)(48.7GiB/900001msec)
slat (nsec): min=602, max=15251M, avg=26931.70, stdev=8058751.14
clat (usec): min=24, max=22701k, avg=647.36, stdev=56304.54
lat (usec): min=25, max=22701k, avg=674.36, stdev=56896.75
clat percentiles (usec):
| 1.00th=[ 54], 5.00th=[ 62], 10.00th=[ 67], 20.00th=[ 73],
| 30.00th=[ 80], 40.00th=[ 91], 50.00th=[ 198], 60.00th=[ 269],
| 70.00th=[ 404], 80.00th=[ 611], 90.00th=[ 1057], 95.00th=[ 1647],
| 99.00th=[ 3097], 99.50th=[ 3654], 99.90th=[ 5080], 99.95th=[ 5866],
| 99.99th=[10290]
bw ( KiB/s): min= 8, max=538712, per=36.89%, avg=83694.31, stdev=106111.60, samples=1220
iops : min= 2, max=134678, avg=20923.57, stdev=26527.89, samples=1220
write: IOPS=9449, BW=36.9MiB/s (38.7MB/s)(32.4GiB/900001msec); 0 zone resets
slat (nsec): min=982, max=17920M, avg=57355.87, stdev=15301847.63
clat (nsec): min=1393, max=22701M, avg=616232.01, stdev=53793093.23
lat (usec): min=26, max=22701k, avg=673.66, stdev=56413.37
clat percentiles (usec):
| 1.00th=[ 53], 5.00th=[ 62], 10.00th=[ 67], 20.00th=[ 73],
| 30.00th=[ 80], 40.00th=[ 89], 50.00th=[ 188], 60.00th=[ 245],
| 70.00th=[ 375], 80.00th=[ 578], 90.00th=[ 1004], 95.00th=[ 1582],
| 99.00th=[ 2999], 99.50th=[ 3589], 99.90th=[ 5014], 99.95th=[ 5735],
| 99.99th=[10028]
bw ( KiB/s): min= 8, max=358920, per=36.94%, avg=55889.27, stdev=70681.64, samples=1217
iops : min= 2, max=89730, avg=13972.32, stdev=17670.41, samples=1217
lat (usec) : 2=0.01%, 50=0.60%, 100=44.09%, 250=14.70%, 500=16.51%
lat (usec) : 750=8.80%, 1000=4.77%
lat (msec) : 2=7.21%, 4=3.00%, 10=0.31%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%, >=2000=0.01%
cpu : usr=2.22%, sys=17.14%, ctx=7555022, majf=0, minf=23
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=12767350,8504713,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=222MiB/s (232MB/s), 55.4MiB/s-55.4MiB/s (58.1MB/s-58.1MB/s), io=195GiB (209GB), run=900001-900001msec
WRITE: bw=148MiB/s (155MB/s), 36.9MiB/s-36.0MiB/s (38.7MB/s-38.7MB/s), io=130GiB (139GB), run=900001-900001msec
Disk stats (read/write):
dm-3: ios=4275603/17609796, merge=0/0, ticks=1415389/1013248346, in_queue=1014664460, util=98.65%, aggrios=4277523/17609907, aggrmerge=0/0, aggrticks=1411929/1013681205, aggrin_queue=1015093858, aggrutil=98.65%
dm-0: ios=4277523/17609907, merge=0/0, ticks=1411929/1013681205, in_queue=1015093858, util=98.65%, aggrios=4278851/17673756, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
md128: ios=4278851/17673756, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=2139263/6387625, aggrmerge=160/2449587, aggrticks=294633/673499, aggrin_queue=32241, aggrutil=97.61%
sdd: ios=2139623/6390350, merge=75/2451332, ticks=299397/894150, in_queue=43498, util=91.78%
sda: ios=2138904/6384901, merge=245/2447843, ticks=289869/452848, in_queue=20984, util=97.61%
hdd array:
file1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.18
Starting 4 processes
file1: Laying out IO file (1 file / 10240MiB)
Jobs: 4 (f=4): [m(4)][100.0%][r=534KiB/s,w=403KiB/s][r=133,w=100 IOPS][eta 00m:00s]
file1: (groupid=0, jobs=1): err= 0: pid=6427: Wed Apr 15 01:45:56 2020
read: IOPS=40, BW=163KiB/s (167kB/s)(143MiB/900027msec)
slat (nsec): min=1212, max=878192k, avg=16018904.58, stdev=21699428.23
clat (msec): min=32, max=2822, avg=221.78, stdev=153.22
lat (msec): min=40, max=2862, avg=237.80, stdev=161.48
clat percentiles (msec):
| 1.00th=[ 89], 5.00th=[ 113], 10.00th=[ 126], 20.00th=[ 142],
| 30.00th=[ 155], 40.00th=[ 165], 50.00th=[ 178], 60.00th=[ 192],
| 70.00th=[ 211], 80.00th=[ 257], 90.00th=[ 384], 95.00th=[ 481],
| 99.00th=[ 860], 99.50th=[ 1083], 99.90th=[ 1687], 99.95th=[ 2005],
| 99.99th=[ 2366]
bw ( KiB/s): min= 8, max= 328, per=25.35%, avg=163.77, stdev=73.33, samples=1789
iops : min= 2, max= 82, avg=40.92, stdev=18.35, samples=1789
write: IOPS=27, BW=109KiB/s (111kB/s)(95.4MiB/900027msec); 0 zone resets
slat (nsec): min=1673, max=797254k, avg=12810828.53, stdev=20531401.74
clat (nsec): min=1874, max=2770.1M, avg=219834449.76, stdev=158340065.25
lat (msec): min=34, max=2870, avg=232.65, stdev=165.75
clat percentiles (msec):
| 1.00th=[ 85], 5.00th=[ 108], 10.00th=[ 123], 20.00th=[ 138],
| 30.00th=[ 150], 40.00th=[ 163], 50.00th=[ 174], 60.00th=[ 188],
| 70.00th=[ 209], 80.00th=[ 257], 90.00th=[ 380], 95.00th=[ 477],
| 99.00th=[ 902], 99.50th=[ 1167], 99.90th=[ 1670], 99.95th=[ 2039],
| 99.99th=[ 2500]
bw ( KiB/s): min= 7, max= 288, per=25.33%, avg=109.69, stdev=53.25, samples=1780
iops : min= 1, max= 72, avg=27.40, stdev=13.32, samples=1780
lat (usec) : 2=0.01%
lat (msec) : 50=0.03%, 100=2.56%, 250=76.40%, 500=16.74%, 750=2.84%
lat (msec) : 1000=0.74%, 2000=0.64%, >=2000=0.05%
cpu : usr=0.02%, sys=0.70%, ctx=77997, majf=0, minf=19
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=36642,24424,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=6428: Wed Apr 15 01:45:56 2020
read: IOPS=39, BW=159KiB/s (163kB/s)(140MiB/900027msec)
slat (nsec): min=1242, max=753470k, avg=16238498.10, stdev=22017584.22
clat (msec): min=50, max=2980, avg=226.92, stdev=161.24
lat (msec): min=58, max=3103, avg=243.15, stdev=169.55
clat percentiles (msec):
| 1.00th=[ 93], 5.00th=[ 114], 10.00th=[ 128], 20.00th=[ 144],
| 30.00th=[ 157], 40.00th=[ 167], 50.00th=[ 180], 60.00th=[ 194],
| 70.00th=[ 215], 80.00th=[ 264], 90.00th=[ 388], 95.00th=[ 493],
| 99.00th=[ 936], 99.50th=[ 1167], 99.90th=[ 1720], 99.95th=[ 2056],
| 99.99th=[ 2802]
bw ( KiB/s): min= 7, max= 336, per=24.79%, avg=160.17, stdev=72.25, samples=1790
iops : min= 1, max= 84, avg=40.02, stdev=18.07, samples=1790
write: IOPS=26, BW=106KiB/s (109kB/s)(93.6MiB/900027msec); 0 zone resets
slat (nsec): min=1603, max=576628k, avg=13256187.30, stdev=20718121.87
clat (nsec): min=1543, max=3015.3M, avg=223824877.06, stdev=163028368.52
lat (msec): min=34, max=3088, avg=237.08, stdev=170.93
clat percentiles (msec):
| 1.00th=[ 89], 5.00th=[ 112], 10.00th=[ 125], 20.00th=[ 140],
| 30.00th=[ 153], 40.00th=[ 165], 50.00th=[ 178], 60.00th=[ 190],
| 70.00th=[ 211], 80.00th=[ 262], 90.00th=[ 384], 95.00th=[ 485],
| 99.00th=[ 944], 99.50th=[ 1200], 99.90th=[ 1770], 99.95th=[ 2265],
| 99.99th=[ 2668]
bw ( KiB/s): min= 7, max= 264, per=24.85%, avg=107.59, stdev=53.57, samples=1780
iops : min= 1, max= 66, avg=26.87, stdev=13.40, samples=1780
lat (usec) : 2=0.01%
lat (msec) : 50=0.01%, 100=2.08%, 250=76.35%, 500=16.82%, 750=3.20%
lat (msec) : 1000=0.66%, 2000=0.83%, >=2000=0.06%
cpu : usr=0.02%, sys=0.60%, ctx=77652, majf=0, minf=19
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=35859,23955,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=6429: Wed Apr 15 01:45:56 2020
read: IOPS=40, BW=163KiB/s (167kB/s)(143MiB/900027msec)
slat (nsec): min=1443, max=766424k, avg=15892940.95, stdev=21352833.36
clat (nsec): min=1884, max=2955.3M, avg=221958326.00, stdev=162678407.80
lat (msec): min=44, max=3029, avg=237.85, stdev=171.07
clat percentiles (msec):
| 1.00th=[ 91], 5.00th=[ 113], 10.00th=[ 125], 20.00th=[ 140],
| 30.00th=[ 153], 40.00th=[ 165], 50.00th=[ 176], 60.00th=[ 190],
| 70.00th=[ 209], 80.00th=[ 255], 90.00th=[ 380], 95.00th=[ 485],
| 99.00th=[ 911], 99.50th=[ 1167], 99.90th=[ 1854], 99.95th=[ 2232],
| 99.99th=[ 2534]
bw ( KiB/s): min= 8, max= 384, per=25.40%, avg=164.08, stdev=74.91, samples=1789
iops : min= 2, max= 96, avg=40.99, stdev=18.74, samples=1789
write: IOPS=27, BW=109KiB/s (112kB/s)(95.9MiB/900027msec); 0 zone resets
slat (nsec): min=1583, max=589310k, avg=12886125.28, stdev=20885914.54
clat (msec): min=31, max=2931, avg=217.75, stdev=155.35
lat (msec): min=41, max=3035, avg=230.64, stdev=162.83
clat percentiles (msec):
| 1.00th=[ 87], 5.00th=[ 109], 10.00th=[ 122], 20.00th=[ 138],
| 30.00th=[ 148], 40.00th=[ 161], 50.00th=[ 174], 60.00th=[ 186],
| 70.00th=[ 207], 80.00th=[ 255], 90.00th=[ 380], 95.00th=[ 481],
| 99.00th=[ 869], 99.50th=[ 1099], 99.90th=[ 1636], 99.95th=[ 2005],
| 99.99th=[ 2903]
bw ( KiB/s): min= 7, max= 304, per=25.57%, avg=110.72, stdev=53.36, samples=1773
iops : min= 1, max= 76, avg=27.65, stdev=13.35, samples=1773
lat (usec) : 2=0.01%
lat (msec) : 50=0.02%, 100=2.46%, 250=77.00%, 500=16.15%, 750=2.86%
lat (msec) : 1000=0.75%, 2000=0.68%, >=2000=0.07%
cpu : usr=0.03%, sys=0.59%, ctx=77951, majf=0, minf=18
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=36713,24552,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=6430: Wed Apr 15 01:45:56 2020
read: IOPS=40, BW=162KiB/s (165kB/s)(142MiB/900027msec)
slat (nsec): min=1322, max=824107k, avg=15921847.02, stdev=21301902.36
clat (usec): min=2, max=2898.1k, avg=223385.40, stdev=158901.08
lat (msec): min=27, max=3079, avg=239.31, stdev=167.29
clat percentiles (msec):
| 1.00th=[ 92], 5.00th=[ 113], 10.00th=[ 126], 20.00th=[ 142],
| 30.00th=[ 155], 40.00th=[ 165], 50.00th=[ 178], 60.00th=[ 192],
| 70.00th=[ 211], 80.00th=[ 259], 90.00th=[ 388], 95.00th=[ 485],
| 99.00th=[ 894], 99.50th=[ 1133], 99.90th=[ 1770], 99.95th=[ 2072],
| 99.99th=[ 2836]
bw ( KiB/s): min= 7, max= 344, per=25.16%, avg=162.53, stdev=73.59, samples=1788
iops : min= 1, max= 86, avg=40.61, stdev=18.41, samples=1788
write: IOPS=27, BW=109KiB/s (112kB/s)(95.8MiB/900027msec); 0 zone resets
slat (nsec): min=1583, max=656166k, avg=13099102.70, stdev=21441180.89
clat (msec): min=29, max=2542, avg=219.31, stdev=158.69
lat (msec): min=32, max=2556, avg=232.41, stdev=165.99
clat percentiles (msec):
| 1.00th=[ 88], 5.00th=[ 110], 10.00th=[ 122], 20.00th=[ 138],
| 30.00th=[ 150], 40.00th=[ 161], 50.00th=[ 174], 60.00th=[ 188],
| 70.00th=[ 207], 80.00th=[ 255], 90.00th=[ 384], 95.00th=[ 481],
| 99.00th=[ 919], 99.50th=[ 1133], 99.90th=[ 1720], 99.95th=[ 2039],
| 99.99th=[ 2433]
bw ( KiB/s): min= 7, max= 304, per=25.52%, avg=110.50, stdev=54.44, samples=1774
iops : min= 1, max= 76, avg=27.60, stdev=13.62, samples=1774
lat (usec) : 4=0.01%
lat (msec) : 50=0.03%, 100=2.30%, 250=76.78%, 500=16.50%, 750=2.89%
lat (msec) : 1000=0.74%, 2000=0.69%, >=2000=0.07%
cpu : usr=0.02%, sys=0.63%, ctx=77963, majf=0, minf=18
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=36345,24519,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=647KiB/s (662kB/s), 159KiB/s-163KiB/s (163kB/s-167kB/s), io=569MiB (596MB), run=900027-900027msec
WRITE: bw=433KiB/s (443kB/s), 106KiB/s-109KiB/s (109kB/s-112kB/s), io=381MiB (399MB), run=900027-900027msec
Disk stats (read/write):
dm-2: ios=147608/115656, merge=0/0, ticks=1852342/56226343, in_queue=58078685, util=21.65%, aggrios=147616/115661, aggrmerge=0/0, aggrticks=1852266/56226297, aggrin_queue=58078563, aggrutil=21.66%
dm-1: ios=147616/115661, merge=0/0, ticks=1852266/56226297, in_queue=58078563, util=21.66%, aggrios=147617/115710, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
md127: ios=147617/115710, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=49206/38650, aggrmerge=0/17, aggrticks=615920/1354928, aggrin_queue=1927653, aggrutil=12.05%
sdb: ios=49620/38693, merge=0/23, ticks=675653/1576403, in_queue=2208415, util=12.05%
sde: ios=49050/38640, merge=0/17, ticks=590031/1274911, in_queue=1821844, util=12.01%
sdc: ios=48949/38617, merge=0/12, ticks=582078/1213470, in_queue=1752702, util=11.99%
ssd-test.fio:
ssd array:
NOTE: filesystem BLOCK_SIZE=”512”
seq-read: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
rand-read: (g=1): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
seq-write: (g=2): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
rand-write: (g=3): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
fio-3.18
Starting 4 processes
seq-read: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [_(3),w(1)][97.1%][w=144MiB/s][w=36.0k IOPS][eta 00m:07s]
seq-read: (groupid=0, jobs=1): err= 0: pid=493789: Wed Apr 15 09:36:28 2020
read: IOPS=47.3k, BW=185MiB/s (194MB/s)(10.0GiB/55432msec)
slat (nsec): min=1734, max=7910.4k, avg=6939.26, stdev=6107.82
clat (usec): min=18, max=18890, avg=77.00, stdev=58.03
lat (usec): min=39, max=18894, avg=84.06, stdev=57.46
clat percentiles (usec):
| 1.00th=[ 46], 5.00th=[ 55], 10.00th=[ 57], 20.00th=[ 60],
| 30.00th=[ 61], 40.00th=[ 63], 50.00th=[ 73], 60.00th=[ 87],
| 70.00th=[ 91], 80.00th=[ 93], 90.00th=[ 95], 95.00th=[ 96],
| 99.00th=[ 176], 99.50th=[ 212], 99.90th=[ 330], 99.95th=[ 379],
| 99.99th=[ 2245]
bw ( KiB/s): min=165624, max=198600, per=100.00%, avg=189438.94, stdev=5253.06, samples=110
iops : min=41406, max=49650, avg=47359.73, stdev=1313.24, samples=110
lat (usec) : 20=0.01%, 50=2.23%, 100=95.33%, 250=2.25%, 500=0.16%
lat (usec) : 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
cpu : usr=5.21%, sys=33.67%, ctx=1870277, majf=0, minf=12
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2621440,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
rand-read: (groupid=1, jobs=1): err= 0: pid=493941: Wed Apr 15 09:36:28 2020
read: IOPS=21.4k, BW=83.7MiB/s (87.8MB/s)(5021MiB/60001msec)
slat (nsec): min=1993, max=698901, avg=5083.29, stdev=2238.84
clat (usec): min=10, max=5835, avg=180.83, stdev=107.98
lat (usec): min=64, max=5841, avg=186.03, stdev=107.39
clat percentiles (usec):
| 1.00th=[ 85], 5.00th=[ 87], 10.00th=[ 89], 20.00th=[ 92],
| 30.00th=[ 96], 40.00th=[ 113], 50.00th=[ 133], 60.00th=[ 178],
| 70.00th=[ 249], 80.00th=[ 265], 90.00th=[ 343], 95.00th=[ 363],
| 99.00th=[ 457], 99.50th=[ 519], 99.90th=[ 676], 99.95th=[ 881],
| 99.99th=[ 2343]
bw ( KiB/s): min=75832, max=90616, per=100.00%, avg=85828.91, stdev=2325.89, samples=119
iops : min=18958, max=22654, avg=21457.24, stdev=581.46, samples=119
lat (usec) : 20=0.01%, 100=35.33%, 250=34.90%, 500=29.15%, 750=0.54%
lat (usec) : 1000=0.03%
lat (msec) : 2=0.03%, 4=0.01%, 10=0.01%
cpu : usr=2.97%, sys=13.06%, ctx=1000451, majf=0, minf=11
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1285489,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
seq-write: (groupid=2, jobs=1): err= 0: pid=494174: Wed Apr 15 09:36:28 2020
write: IOPS=41.5k, BW=162MiB/s (170MB/s)(9724MiB/60001msec); 0 zone resets
slat (nsec): min=1302, max=505648, avg=3797.37, stdev=1015.23
clat (usec): min=9, max=85537, avg=91.98, stdev=119.30
lat (usec): min=42, max=85543, avg=95.89, stdev=119.29
clat percentiles (usec):
| 1.00th=[ 58], 5.00th=[ 72], 10.00th=[ 76], 20.00th=[ 78],
| 30.00th=[ 79], 40.00th=[ 80], 50.00th=[ 87], 60.00th=[ 96],
| 70.00th=[ 100], 80.00th=[ 104], 90.00th=[ 108], 95.00th=[ 111],
| 99.00th=[ 204], 99.50th=[ 239], 99.90th=[ 379], 99.95th=[ 553],
| 99.99th=[ 1237]
bw ( KiB/s): min=139312, max=174680, per=100.00%, avg=166184.06, stdev=5209.64, samples=119
iops : min=34828, max=43670, avg=41546.03, stdev=1302.41, samples=119
lat (usec) : 10=0.01%, 20=0.01%, 50=0.23%, 100=69.39%, 250=29.98%
lat (usec) : 500=0.34%, 750=0.03%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=5.76%, sys=19.09%, ctx=1840937, majf=0, minf=13
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,2489393,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
rand-write: (groupid=3, jobs=1): err= 0: pid=494628: Wed Apr 15 09:36:28 2020
write: IOPS=39.1k, BW=153MiB/s (160MB/s)(9155MiB/60001msec); 0 zone resets
slat (nsec): min=1373, max=1069.0k, avg=3878.94, stdev=1213.13
clat (usec): min=10, max=647688, avg=97.77, stdev=994.90
lat (usec): min=41, max=647745, avg=101.76, stdev=994.94
clat percentiles (usec):
| 1.00th=[ 45], 5.00th=[ 47], 10.00th=[ 49], 20.00th=[ 52],
| 30.00th=[ 56], 40.00th=[ 60], 50.00th=[ 70], 60.00th=[ 86],
| 70.00th=[ 115], 80.00th=[ 129], 90.00th=[ 161], 95.00th=[ 180],
| 99.00th=[ 281], 99.50th=[ 363], 99.90th=[ 1860], 99.95th=[ 2900],
| 99.99th=[ 7308]
bw ( KiB/s): min=58584, max=185840, per=100.00%, avg=157849.32, stdev=21750.44, samples=118
iops : min=14646, max=46460, avg=39462.38, stdev=5437.63, samples=118
lat (usec) : 20=0.01%, 50=15.62%, 100=49.24%, 250=33.66%, 500=1.20%
lat (usec) : 750=0.09%, 1000=0.04%
lat (msec) : 2=0.06%, 4=0.06%, 10=0.03%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
cpu : usr=6.20%, sys=18.91%, ctx=2049357, majf=0, minf=13
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,2343796,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=185MiB/s (194MB/s), 185MiB/s-185MiB/s (194MB/s-194MB/s), io=10.0GiB (10.7GB), run=55432-55432msec
Run status group 1 (all jobs):
READ: bw=83.7MiB/s (87.8MB/s), 83.7MiB/s-83.7MiB/s (87.8MB/s-87.8MB/s), io=5021MiB (5265MB), run=60001-60001msec
Run status group 2 (all jobs):
WRITE: bw=162MiB/s (170MB/s), 162MiB/s-162MiB/s (170MB/s-170MB/s), io=9724MiB (10.2GB), run=60001-60001msec
Run status group 3 (all jobs):
WRITE: bw=153MiB/s (160MB/s), 153MiB/s-153MiB/s (160MB/s-160MB/s), io=9155MiB (9600MB), run=60001-60001msec
Disk stats (read/write):
dm-3: ios=3920958/4851512, merge=0/0, ticks=445073/1856767, in_queue=2302040, util=99.14%, aggrios=3920967/4854312, aggrmerge=0/0, aggrticks=442211/1947582, aggrin_queue=2389993, aggrutil=99.23%
dm-0: ios=3920967/4854312, merge=0/0, ticks=442211/1947582, in_queue=2389993, util=99.23%, aggrios=3923477/4857404, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
md128: ios=3923477/4857404, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=1638145/2116515, aggrmerge=323580/311709, aggrticks=181144/169117, aggrin_queue=11661, aggrutil=87.44%
sdd: ios=1962398/2416430, merge=341/10761, ticks=141224/141199, in_queue=13519, util=80.41%
sda: ios=1313892/1816601, merge=646819/612657, ticks=221064/197036, in_queue=9804, util=87.44%
fio-seq-RW.fio:
ssd array:
NOTE: filesystem BLOCK_SIZE=”512”
file1: (g=0): rw=rw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=16
...
fio-3.18
Starting 4 processes
file1: Laying out IO file (1 file / 10240MiB)
Jobs: 4 (f=4): [M(4)][100.0%][eta 00m:00s]
file1: (groupid=0, jobs=1): err= 0: pid=42891: Thu Apr 16 22:22:20 2020
read: IOPS=1814, BW=454MiB/s (476MB/s)(400GiB/902747msec)
slat (usec): min=13, max=2717.6k, avg=184.83, stdev=5610.28
clat (usec): min=10, max=9105.8k, avg=5027.03, stdev=103760.79
lat (usec): min=501, max=9106.0k, avg=5212.05, stdev=103913.63
clat percentiles (usec):
| 1.00th=[ 1106], 5.00th=[ 1418], 10.00th=[ 1614],
| 20.00th=[ 1876], 30.00th=[ 2114], 40.00th=[ 2343],
| 50.00th=[ 2573], 60.00th=[ 2900], 70.00th=[ 3359],
| 80.00th=[ 4113], 90.00th=[ 5473], 95.00th=[ 6915],
| 99.00th=[ 11207], 99.50th=[ 14222], 99.90th=[ 25297],
| 99.95th=[ 40109], 99.99th=[5737808]
bw ( KiB/s): min= 1024, max=1362944, per=36.93%, avg=686251.06, stdev=248132.23, samples=1222
iops : min= 4, max= 5324, avg=2680.66, stdev=969.26, samples=1222
write: IOPS=1209, BW=302MiB/s (317MB/s)(267GiB/902747msec); 0 zone resets
slat (usec): min=26, max=3839.1k, avg=279.70, stdev=3882.85
clat (usec): min=481, max=9105.6k, avg=4859.48, stdev=100268.29
lat (usec): min=540, max=9105.6k, avg=5139.40, stdev=100344.72
clat percentiles (usec):
| 1.00th=[ 1090], 5.00th=[ 1401], 10.00th=[ 1598],
| 20.00th=[ 1844], 30.00th=[ 2073], 40.00th=[ 2278],
| 50.00th=[ 2540], 60.00th=[ 2835], 70.00th=[ 3261],
| 80.00th=[ 3949], 90.00th=[ 5342], 95.00th=[ 6718],
| 99.00th=[ 10945], 99.50th=[ 13829], 99.90th=[ 24511],
| 99.95th=[ 34341], 99.99th=[5268046]
bw ( KiB/s): min= 505, max=936960, per=36.89%, avg=456976.76, stdev=166758.28, samples=1223
iops : min= 1, max= 3660, avg=1785.06, stdev=651.40, samples=1223
lat (usec) : 20=0.01%, 500=0.01%, 750=0.13%, 1000=0.38%
lat (msec) : 2=25.57%, 4=53.44%, 10=19.08%, 20=1.21%, 50=0.14%
lat (msec) : 100=0.01%, 250=0.01%, 750=0.01%, 1000=0.01%, 2000=0.01%
lat (msec) : >=2000=0.03%
cpu : usr=1.10%, sys=51.43%, ctx=14324594, majf=0, minf=21
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1638401,1091922,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=42892: Thu Apr 16 22:22:20 2020
read: IOPS=1814, BW=454MiB/s (476MB/s)(400GiB/902746msec)
slat (usec): min=11, max=3839.3k, avg=184.51, stdev=6310.31
clat (usec): min=8, max=8994.8k, avg=5096.20, stdev=105688.68
lat (usec): min=570, max=8994.9k, avg=5280.91, stdev=105880.31
clat percentiles (usec):
| 1.00th=[ 1123], 5.00th=[ 1434], 10.00th=[ 1614],
| 20.00th=[ 1876], 30.00th=[ 2114], 40.00th=[ 2311],
| 50.00th=[ 2573], 60.00th=[ 2900], 70.00th=[ 3326],
| 80.00th=[ 4080], 90.00th=[ 5473], 95.00th=[ 6849],
| 99.00th=[ 11207], 99.50th=[ 14222], 99.90th=[ 25297],
| 99.95th=[ 40109], 99.99th=[5536482]
bw ( KiB/s): min= 512, max=1246720, per=37.08%, avg=689074.87, stdev=243035.95, samples=1217
iops : min= 2, max= 4870, avg=2691.69, stdev=949.36, samples=1217
write: IOPS=1209, BW=302MiB/s (317MB/s)(267GiB/902746msec); 0 zone resets
slat (usec): min=24, max=1319.9k, avg=278.72, stdev=1498.99
clat (usec): min=536, max=8994.7k, avg=4753.99, stdev=98651.40
lat (usec): min=599, max=8994.8k, avg=5032.93, stdev=98661.83
clat percentiles (usec):
| 1.00th=[ 1123], 5.00th=[ 1418], 10.00th=[ 1598],
| 20.00th=[ 1860], 30.00th=[ 2073], 40.00th=[ 2278],
| 50.00th=[ 2540], 60.00th=[ 2835], 70.00th=[ 3228],
| 80.00th=[ 3916], 90.00th=[ 5276], 95.00th=[ 6718],
| 99.00th=[ 10945], 99.50th=[ 13829], 99.90th=[ 24511],
| 99.95th=[ 33162], 99.99th=[5469373]
bw ( KiB/s): min= 505, max=813056, per=37.09%, avg=459402.89, stdev=162443.85, samples=1217
iops : min= 1, max= 3176, avg=1794.53, stdev=634.55, samples=1217
lat (usec) : 10=0.01%, 750=0.04%, 1000=0.39%
lat (msec) : 2=25.53%, 4=53.81%, 10=18.85%, 20=1.17%, 50=0.16%
lat (msec) : 100=0.01%, 250=0.01%, 750=0.01%, 1000=0.01%, 2000=0.01%
lat (msec) : >=2000=0.03%
cpu : usr=1.09%, sys=51.99%, ctx=14033638, majf=0, minf=21
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1638401,1092319,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=42893: Thu Apr 16 22:22:20 2020
read: IOPS=1814, BW=454MiB/s (476MB/s)(400GiB/902745msec)
slat (usec): min=13, max=2572.8k, avg=178.75, stdev=4798.26
clat (usec): min=9, max=8953.0k, avg=4957.37, stdev=100759.70
lat (usec): min=424, max=8953.0k, avg=5136.31, stdev=100874.82
clat percentiles (usec):
| 1.00th=[ 1106], 5.00th=[ 1418], 10.00th=[ 1614],
| 20.00th=[ 1876], 30.00th=[ 2089], 40.00th=[ 2311],
| 50.00th=[ 2573], 60.00th=[ 2900], 70.00th=[ 3359],
| 80.00th=[ 4113], 90.00th=[ 5473], 95.00th=[ 6849],
| 99.00th=[ 11207], 99.50th=[ 14222], 99.90th=[ 25560],
| 99.95th=[ 38536], 99.99th=[5402264]
bw ( KiB/s): min= 512, max=1317376, per=36.90%, avg=685699.53, stdev=248396.06, samples=1223
iops : min= 2, max= 5146, avg=2678.50, stdev=970.29, samples=1223
write: IOPS=1211, BW=303MiB/s (317MB/s)(267GiB/902745msec); 0 zone resets
slat (usec): min=22, max=3839.1k, avg=287.84, stdev=5264.77
clat (usec): min=415, max=8952.5k, avg=4957.54, stdev=103987.50
lat (usec): min=483, max=8952.6k, avg=5245.60, stdev=104124.31
clat percentiles (usec):
| 1.00th=[ 1090], 5.00th=[ 1401], 10.00th=[ 1598],
| 20.00th=[ 1844], 30.00th=[ 2073], 40.00th=[ 2278],
| 50.00th=[ 2540], 60.00th=[ 2835], 70.00th=[ 3261],
| 80.00th=[ 3949], 90.00th=[ 5342], 95.00th=[ 6718],
| 99.00th=[ 10814], 99.50th=[ 13698], 99.90th=[ 25035],
| 99.95th=[ 36963], 99.99th=[5469373]
bw ( KiB/s): min= 512, max=848896, per=36.97%, avg=457954.36, stdev=164997.43, samples=1222
iops : min= 2, max= 3316, avg=1788.87, stdev=644.52, samples=1222
lat (usec) : 10=0.01%, 500=0.01%, 750=0.22%, 1000=0.39%
lat (msec) : 2=25.59%, 4=53.40%, 10=19.04%, 20=1.15%, 50=0.16%
lat (msec) : 100=0.01%, 250=0.01%, 750=0.01%, 1000=0.01%, 2000=0.01%
lat (msec) : >=2000=0.03%
cpu : usr=1.09%, sys=51.41%, ctx=14245479, majf=0, minf=21
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1638401,1093352,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=42894: Thu Apr 16 22:22:20 2020
read: IOPS=1814, BW=454MiB/s (476MB/s)(400GiB/902746msec)
slat (usec): min=12, max=3839.2k, avg=203.77, stdev=6385.25
clat (usec): min=9, max=9027.8k, avg=5012.82, stdev=103448.02
lat (usec): min=501, max=9027.0k, avg=5216.78, stdev=103648.60
clat percentiles (usec):
| 1.00th=[ 996], 5.00th=[ 1418], 10.00th=[ 1614],
| 20.00th=[ 1893], 30.00th=[ 2114], 40.00th=[ 2376],
| 50.00th=[ 2638], 60.00th=[ 2933], 70.00th=[ 3392],
| 80.00th=[ 4113], 90.00th=[ 5473], 95.00th=[ 6849],
| 99.00th=[ 11207], 99.50th=[ 14222], 99.90th=[ 25560],
| 99.95th=[ 39060], 99.99th=[5402264]
bw ( KiB/s): min= 512, max=1742848, per=36.90%, avg=685701.93, stdev=248867.68, samples=1223
iops : min= 2, max= 6808, avg=2678.51, stdev=972.14, samples=1223
write: IOPS=1208, BW=302MiB/s (317MB/s)(266GiB/902746msec); 0 zone resets
slat (usec): min=20, max=953698, avg=253.24, stdev=1060.72
clat (usec): min=462, max=9027.6k, avg=4886.78, stdev=100816.16
lat (usec): min=506, max=9027.9k, avg=5140.24, stdev=100821.59
clat percentiles (usec):
| 1.00th=[ 988], 5.00th=[ 1385], 10.00th=[ 1598],
| 20.00th=[ 1860], 30.00th=[ 2089], 40.00th=[ 2311],
| 50.00th=[ 2573], 60.00th=[ 2868], 70.00th=[ 3294],
| 80.00th=[ 3949], 90.00th=[ 5276], 95.00th=[ 6652],
| 99.00th=[ 10814], 99.50th=[ 13829], 99.90th=[ 24511],
| 99.95th=[ 36963], 99.99th=[5268046]
bw ( KiB/s): min= 505, max=1180160, per=36.85%, avg=456424.24, stdev=166213.07, samples=1223
iops : min= 1, max= 4610, avg=1782.90, stdev=649.27, samples=1223
lat (usec) : 10=0.01%, 500=0.01%, 750=0.59%, 1000=0.45%
lat (msec) : 2=23.97%, 4=54.39%, 10=19.25%, 20=1.17%, 50=0.14%
lat (msec) : 100=0.01%, 250=0.01%, 750=0.01%, 1000=0.01%, 2000=0.01%
lat (msec) : >=2000=0.03%
cpu : usr=1.12%, sys=52.21%, ctx=14216879, majf=0, minf=23
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1638401,1090585,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=1815MiB/s (1903MB/s), 454MiB/s-454MiB/s (476MB/s-476MB/s), io=1600GiB (1718GB), run=902745-902747msec
WRITE: bw=1210MiB/s (1268MB/s), 302MiB/s-303MiB/s (317MB/s-317MB/s), io=1066GiB (1145GB), run=902745-902747msec
Disk stats (read/write):
dm-3: ios=3862919/26552569, merge=0/0, ticks=12686654/1044094996, in_queue=1056784029, util=76.85%, aggrios=3862919/26620232, aggrmerge=0/0, aggrticks=12688443/1281930380, aggrin_queue=1294621201, aggrutil=79.59%
dm-0: ios=3862919/26620232, merge=0/0, ticks=12688443/1281930380, in_queue=1294621201, util=79.59%, aggrios=4339895/26927913, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
md128: ios=4339895/26927913, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=2048871/356161, aggrmerge=101818/13108098, aggrticks=6068873/720584, aggrin_queue=5628966, aggrutil=70.02%
sdd: ios=2105652/369456, merge=57410/13091934, ticks=3621523/872452, in_queue=3320778, util=67.30%
sda: ios=1992090/342866, merge=146227/13124263, ticks=8516224/568716, in_queue=7937154, util=70.02%
btrfs filesystem tests:
fio-rand-RW.fio:
ssd array:
file1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.18
Starting 4 processes
file1: Laying out IO file (1 file / 10240MiB)
Jobs: 4 (f=4): [m(4)][100.0%][r=174MiB/s,w=117MiB/s][r=44.5k,w=29.9k IOPS][eta 00m:00s]
file1: (groupid=0, jobs=1): err= 0: pid=488554: Wed Apr 15 09:15:29 2020
read: IOPS=6112, BW=23.9MiB/s (25.0MB/s)(20.0GiB/900103msec)
slat (nsec): min=571, max=1947.5M, avg=67475.51, stdev=2205656.85
clat (usec): min=22, max=3821.9k, avg=1499.71, stdev=30697.76
lat (usec): min=23, max=3821.9k, avg=1567.27, stdev=30783.88
clat percentiles (usec):
| 1.00th=[ 46], 5.00th=[ 61], 10.00th=[ 75],
| 20.00th=[ 99], 30.00th=[ 127], 40.00th=[ 172],
| 50.00th=[ 245], 60.00th=[ 396], 70.00th=[ 627],
| 80.00th=[ 963], 90.00th=[ 1516], 95.00th=[ 2278],
| 99.00th=[ 5473], 99.50th=[ 7308], 99.90th=[ 37487],
| 99.95th=[ 968885], 99.99th=[1367344]
bw ( KiB/s): min= 16, max=151848, per=35.00%, avg=34250.51, stdev=31158.93, samples=1285
iops : min= 4, max=37962, avg=8562.63, stdev=7789.73, samples=1285
write: IOPS=4075, BW=15.9MiB/s (16.7MB/s)(13.0GiB/900103msec); 0 zone resets
slat (usec): min=2, max=1557.9k, avg=140.13, stdev=11835.59
clat (nsec): min=1853, max=3821.8M, avg=1432078.82, stdev=29974401.79
lat (usec): min=26, max=3821.8k, avg=1572.28, stdev=32228.56
clat percentiles (usec):
| 1.00th=[ 46], 5.00th=[ 61], 10.00th=[ 75],
| 20.00th=[ 100], 30.00th=[ 128], 40.00th=[ 172],
| 50.00th=[ 245], 60.00th=[ 396], 70.00th=[ 627],
| 80.00th=[ 963], 90.00th=[ 1516], 95.00th=[ 2278],
| 99.00th=[ 5473], 99.50th=[ 7242], 99.90th=[ 29754],
| 99.95th=[ 918553], 99.99th=[1350566]
bw ( KiB/s): min= 24, max=101080, per=35.05%, avg=22854.64, stdev=20765.75, samples=1284
iops : min= 6, max=25270, avg=5713.66, stdev=5191.44, samples=1284
lat (usec) : 2=0.01%, 50=1.89%, 100=18.49%, 250=30.01%, 500=14.57%
lat (usec) : 750=9.24%, 1000=6.71%
lat (msec) : 2=12.98%, 4=4.01%, 10=1.84%, 20=0.13%, 50=0.03%
lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.02%
lat (msec) : 2000=0.05%, >=2000=0.01%
cpu : usr=0.95%, sys=7.46%, ctx=1312493, majf=0, minf=22
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=5501661,3668295,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=488555: Wed Apr 15 09:15:29 2020
read: IOPS=6112, BW=23.9MiB/s (25.0MB/s)(20.0GiB/900104msec)
slat (nsec): min=561, max=1465.5M, avg=67391.92, stdev=2092061.09
clat (usec): min=21, max=3823.5k, avg=1510.35, stdev=31053.86
lat (usec): min=22, max=3823.5k, avg=1577.82, stdev=31126.44
clat percentiles (usec):
| 1.00th=[ 46], 5.00th=[ 61], 10.00th=[ 74],
| 20.00th=[ 98], 30.00th=[ 126], 40.00th=[ 169],
| 50.00th=[ 245], 60.00th=[ 396], 70.00th=[ 627],
| 80.00th=[ 963], 90.00th=[ 1516], 95.00th=[ 2278],
| 99.00th=[ 5538], 99.50th=[ 7373], 99.90th=[ 36439],
| 99.95th=[ 977273], 99.99th=[1367344]
bw ( KiB/s): min= 16, max=148960, per=34.95%, avg=34200.94, stdev=31141.73, samples=1287
iops : min= 4, max=37240, avg=8550.23, stdev=7785.43, samples=1287
write: IOPS=4076, BW=15.9MiB/s (16.7MB/s)(13.0GiB/900104msec); 0 zone resets
slat (usec): min=2, max=1947.7k, avg=140.52, stdev=11867.77
clat (usec): min=4, max=3823.5k, avg=1415.34, stdev=29048.97
lat (usec): min=27, max=3823.5k, avg=1555.92, stdev=31388.25
clat percentiles (usec):
| 1.00th=[ 45], 5.00th=[ 61], 10.00th=[ 75],
| 20.00th=[ 99], 30.00th=[ 128], 40.00th=[ 172],
| 50.00th=[ 247], 60.00th=[ 396], 70.00th=[ 627],
| 80.00th=[ 963], 90.00th=[ 1516], 95.00th=[ 2311],
| 99.00th=[ 5538], 99.50th=[ 7373], 99.90th=[ 29230],
| 99.95th=[ 910164], 99.99th=[1350566]
bw ( KiB/s): min= 8, max=100416, per=35.00%, avg=22827.97, stdev=20768.03, samples=1286
iops : min= 2, max=25104, avg=5706.99, stdev=5192.01, samples=1286
lat (usec) : 10=0.01%, 50=1.97%, 100=18.55%, 250=29.86%, 500=14.57%
lat (usec) : 750=9.25%, 1000=6.73%
lat (msec) : 2=12.94%, 4=4.03%, 10=1.84%, 20=0.14%, 50=0.03%
lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.02%
lat (msec) : 2000=0.05%, >=2000=0.01%
cpu : usr=0.97%, sys=7.30%, ctx=1311783, majf=0, minf=23
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=5502258,3669714,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=488556: Wed Apr 15 09:15:29 2020
read: IOPS=6116, BW=23.9MiB/s (25.1MB/s)(21.0GiB/900102msec)
slat (nsec): min=541, max=1557.7M, avg=68480.10, stdev=2406561.16
clat (usec): min=21, max=3848.9k, avg=1499.25, stdev=30707.96
lat (usec): min=22, max=3848.9k, avg=1567.81, stdev=30804.46
clat percentiles (usec):
| 1.00th=[ 45], 5.00th=[ 61], 10.00th=[ 74],
| 20.00th=[ 98], 30.00th=[ 126], 40.00th=[ 169],
| 50.00th=[ 245], 60.00th=[ 396], 70.00th=[ 627],
| 80.00th=[ 963], 90.00th=[ 1516], 95.00th=[ 2278],
| 99.00th=[ 5538], 99.50th=[ 7373], 99.90th=[ 38011],
| 99.95th=[ 968885], 99.99th=[1367344]
bw ( KiB/s): min= 32, max=153216, per=34.98%, avg=34222.23, stdev=31198.49, samples=1287
iops : min= 8, max=38304, avg=8555.56, stdev=7799.62, samples=1287
write: IOPS=4075, BW=15.9MiB/s (16.7MB/s)(13.0GiB/900102msec); 0 zone resets
slat (usec): min=2, max=1947.5k, avg=138.80, stdev=11780.17
clat (usec): min=8, max=3848.7k, avg=1431.08, stdev=29641.13
lat (usec): min=26, max=3848.8k, avg=1569.95, stdev=31905.49
clat percentiles (usec):
| 1.00th=[ 45], 5.00th=[ 61], 10.00th=[ 75],
| 20.00th=[ 99], 30.00th=[ 127], 40.00th=[ 172],
| 50.00th=[ 245], 60.00th=[ 396], 70.00th=[ 627],
| 80.00th=[ 963], 90.00th=[ 1516], 95.00th=[ 2278],
| 99.00th=[ 5538], 99.50th=[ 7308], 99.90th=[ 28705],
| 99.95th=[ 926942], 99.99th=[1350566]
bw ( KiB/s): min= 32, max=101216, per=34.96%, avg=22801.94, stdev=20801.58, samples=1287
iops : min= 8, max=25304, avg=5700.48, stdev=5200.39, samples=1287
lat (usec) : 10=0.01%, 50=2.01%, 100=18.63%, 250=29.77%, 500=14.52%
lat (usec) : 750=9.25%, 1000=6.72%
lat (msec) : 2=13.00%, 4=3.98%, 10=1.85%, 20=0.14%, 50=0.03%
lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.02%
lat (msec) : 2000=0.05%, >=2000=0.01%
cpu : usr=0.95%, sys=7.35%, ctx=1308964, majf=0, minf=23
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=5505680,3668390,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=488557: Wed Apr 15 09:15:29 2020
read: IOPS=6120, BW=23.9MiB/s (25.1MB/s)(21.0GiB/900102msec)
slat (nsec): min=531, max=1520.5M, avg=68615.13, stdev=2477276.87
clat (usec): min=22, max=3761.5k, avg=1504.46, stdev=30983.41
lat (usec): min=23, max=3761.5k, avg=1573.15, stdev=31084.50
clat percentiles (usec):
| 1.00th=[ 45], 5.00th=[ 61], 10.00th=[ 74],
| 20.00th=[ 98], 30.00th=[ 126], 40.00th=[ 169],
| 50.00th=[ 245], 60.00th=[ 396], 70.00th=[ 627],
| 80.00th=[ 963], 90.00th=[ 1516], 95.00th=[ 2278],
| 99.00th=[ 5538], 99.50th=[ 7373], 99.90th=[ 37487],
| 99.95th=[ 968885], 99.99th=[1367344]
bw ( KiB/s): min= 40, max=152888, per=35.08%, avg=34321.05, stdev=31303.41, samples=1284
iops : min= 10, max=38222, avg=8580.26, stdev=7825.85, samples=1284
write: IOPS=4075, BW=15.9MiB/s (16.7MB/s)(13.0GiB/900102msec); 0 zone resets
slat (nsec): min=1953, max=1947.7M, avg=137847.28, stdev=11731710.62
clat (usec): min=3, max=3761.5k, avg=1421.93, stdev=29600.36
lat (usec): min=26, max=3761.5k, avg=1559.84, stdev=31849.23
clat percentiles (usec):
| 1.00th=[ 45], 5.00th=[ 61], 10.00th=[ 74],
| 20.00th=[ 99], 30.00th=[ 127], 40.00th=[ 172],
| 50.00th=[ 245], 60.00th=[ 396], 70.00th=[ 627],
| 80.00th=[ 963], 90.00th=[ 1516], 95.00th=[ 2278],
| 99.00th=[ 5473], 99.50th=[ 7308], 99.90th=[ 28181],
| 99.95th=[ 910164], 99.99th=[1350566]
bw ( KiB/s): min= 16, max=100128, per=35.05%, avg=22856.83, stdev=20864.27, samples=1284
iops : min= 4, max=25032, avg=5714.21, stdev=5216.07, samples=1284
lat (usec) : 4=0.01%, 50=2.04%, 100=18.75%, 250=29.65%, 500=14.54%
lat (usec) : 750=9.23%, 1000=6.73%
lat (msec) : 2=12.96%, 4=4.00%, 10=1.85%, 20=0.13%, 50=0.03%
lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.02%
lat (msec) : 2000=0.05%, >=2000=0.01%
cpu : usr=0.94%, sys=7.63%, ctx=1325019, majf=0, minf=22
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=5508710,3668641,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=95.6MiB/s (100MB/s), 23.9MiB/s-23.9MiB/s (25.0MB/s-25.1MB/s), io=83.0GiB (90.2GB), run=900102-900104msec
WRITE: bw=63.7MiB/s (66.8MB/s), 15.9MiB/s-15.9MiB/s (16.7MB/s-16.7MB/s), io=55.0GiB (60.1GB), run=900102-900104msec
NOTE: deleting 10gb test file tooke like one minute or more on btrfs.
ssd-test.fio:
ssd array:
seq-read: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
rand-read: (g=1): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
seq-write: (g=2): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
rand-write: (g=3): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
fio-3.18
Starting 4 processes
seq-read: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [_(3),w(1)][95.6%][w=145MiB/s][w=37.2k IOPS][eta 00m:11s]
seq-read: (groupid=0, jobs=1): err= 0: pid=491399: Wed Apr 15 09:23:03 2020
read: IOPS=47.3k, BW=185MiB/s (194MB/s)(10.0GiB/55385msec)
slat (usec): min=2, max=340, avg=10.66, stdev= 4.89
clat (usec): min=21, max=15591, avg=73.22, stdev=41.49
lat (usec): min=43, max=15630, avg=84.02, stdev=39.98
clat percentiles (usec):
| 1.00th=[ 44], 5.00th=[ 49], 10.00th=[ 52], 20.00th=[ 54],
| 30.00th=[ 55], 40.00th=[ 59], 50.00th=[ 74], 60.00th=[ 84],
| 70.00th=[ 86], 80.00th=[ 89], 90.00th=[ 93], 95.00th=[ 96],
| 99.00th=[ 178], 99.50th=[ 212], 99.90th=[ 326], 99.95th=[ 375],
| 99.99th=[ 1795]
bw ( KiB/s): min=171528, max=198360, per=100.00%, avg=189631.37, stdev=5075.54, samples=110
iops : min=42882, max=49590, avg=47407.82, stdev=1268.90, samples=110
lat (usec) : 50=6.37%, 100=90.79%, 250=2.65%, 500=0.16%, 750=0.01%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
cpu : usr=5.30%, sys=49.00%, ctx=971079, majf=0, minf=15
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2621440,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
rand-read: (groupid=1, jobs=1): err= 0: pid=491498: Wed Apr 15 09:23:03 2020
read: IOPS=21.3k, BW=83.1MiB/s (87.2MB/s)(4989MiB/60001msec)
slat (usec): min=3, max=767, avg= 8.89, stdev= 2.77
clat (usec): min=14, max=30037, avg=178.14, stdev=121.64
lat (usec): min=72, max=30045, avg=187.17, stdev=121.07
clat percentiles (usec):
| 1.00th=[ 88], 5.00th=[ 91], 10.00th=[ 93], 20.00th=[ 96],
| 30.00th=[ 103], 40.00th=[ 120], 50.00th=[ 137], 60.00th=[ 172],
| 70.00th=[ 231], 80.00th=[ 258], 90.00th=[ 326], 95.00th=[ 351],
| 99.00th=[ 445], 99.50th=[ 498], 99.90th=[ 660], 99.95th=[ 840],
| 99.99th=[ 2442]
bw ( KiB/s): min=76015, max=88776, per=100.00%, avg=85235.61, stdev=2436.96, samples=119
iops : min=19003, max=22194, avg=21308.89, stdev=609.28, samples=119
lat (usec) : 20=0.01%, 50=0.01%, 100=27.27%, 250=47.88%, 500=24.35%
lat (usec) : 750=0.43%, 1000=0.03%
lat (msec) : 2=0.02%, 4=0.02%, 10=0.01%, 20=0.01%, 50=0.01%
cpu : usr=3.30%, sys=20.80%, ctx=1025418, majf=0, minf=14
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1277191,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
seq-write: (groupid=2, jobs=1): err= 0: pid=491822: Wed Apr 15 09:23:03 2020
write: IOPS=39.7k, BW=155MiB/s (162MB/s)(9295MiB/60001msec); 0 zone resets
slat (usec): min=4, max=296, avg=11.64, stdev= 2.44
clat (usec): min=27, max=102728, avg=88.55, stdev=222.10
lat (usec): min=48, max=102736, avg=100.32, stdev=222.12
clat percentiles (usec):
| 1.00th=[ 57], 5.00th=[ 64], 10.00th=[ 67], 20.00th=[ 70],
| 30.00th=[ 71], 40.00th=[ 73], 50.00th=[ 84], 60.00th=[ 96],
| 70.00th=[ 99], 80.00th=[ 102], 90.00th=[ 105], 95.00th=[ 110],
| 99.00th=[ 204], 99.50th=[ 239], 99.90th=[ 379], 99.95th=[ 586],
| 99.99th=[ 2900]
bw ( KiB/s): min=76064, max=170440, per=100.00%, avg=158811.59, stdev=9809.61, samples=119
iops : min=19016, max=42610, avg=39702.91, stdev=2452.40, samples=119
lat (usec) : 50=0.23%, 100=74.24%, 250=25.14%, 500=0.33%, 750=0.02%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%
cpu : usr=6.06%, sys=48.71%, ctx=1685703, majf=0, minf=14
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,2379610,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
rand-write: (groupid=3, jobs=1): err= 0: pid=492071: Wed Apr 15 09:23:03 2020
write: IOPS=36.8k, BW=144MiB/s (151MB/s)(8626MiB/60002msec); 0 zone resets
slat (usec): min=5, max=160439, avg=14.43, stdev=232.09
clat (usec): min=26, max=160506, avg=93.44, stdev=486.35
lat (usec): min=46, max=179366, avg=108.00, stdev=541.71
clat percentiles (usec):
| 1.00th=[ 46], 5.00th=[ 50], 10.00th=[ 54], 20.00th=[ 61],
| 30.00th=[ 67], 40.00th=[ 70], 50.00th=[ 74], 60.00th=[ 94],
| 70.00th=[ 98], 80.00th=[ 103], 90.00th=[ 125], 95.00th=[ 163],
| 99.00th=[ 249], 99.50th=[ 302], 99.90th=[ 1045], 99.95th=[ 2024],
| 99.99th=[12387]
bw ( KiB/s): min=34888, max=163664, per=100.00%, avg=147452.25, stdev=19099.00, samples=119
iops : min= 8722, max=40916, avg=36863.08, stdev=4774.77, samples=119
lat (usec) : 50=5.95%, 100=68.34%, 250=24.72%, 500=0.84%, 750=0.04%
lat (usec) : 1000=0.02%
lat (msec) : 2=0.05%, 4=0.03%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%
cpu : usr=6.21%, sys=51.58%, ctx=1384080, majf=0, minf=12
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,2208289,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=185MiB/s (194MB/s), 185MiB/s-185MiB/s (194MB/s-194MB/s), io=10.0GiB (10.7GB), run=55385-55385msec
Run status group 1 (all jobs):
READ: bw=83.1MiB/s (87.2MB/s), 83.1MiB/s-83.1MiB/s (87.2MB/s-87.2MB/s), io=4989MiB (5231MB), run=60001-60001msec
Run status group 2 (all jobs):
WRITE: bw=155MiB/s (162MB/s), 155MiB/s-155MiB/s (162MB/s-162MB/s), io=9295MiB (9747MB), run=60001-60001msec
Run status group 3 (all jobs):
WRITE: bw=144MiB/s (151MB/s), 144MiB/s-144MiB/s (151MB/s-151MB/s), io=8626MiB (9045MB), run=60002-60002msec
fio-seq-RW.fio:
ssd array:
file1: (g=0): rw=rw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=16
...
fio-3.18
Starting 4 processes
file1: Laying out IO file (1 file / 10240MiB)
Jobs: 3 (f=3): [M(3),_(1)][100.0%][r=2813KiB/s,w=1278KiB/s][r=10,w=4 IOPS][eta 00m:00s]
file1: (groupid=0, jobs=1): err= 0: pid=46321: Thu Apr 16 23:02:16 2020
read: IOPS=1908, BW=477MiB/s (500MB/s)(420GiB/901390msec)
slat (usec): min=10, max=3292.9k, avg=204.17, stdev=6557.26
clat (usec): min=390, max=9498.3k, avg=4851.96, stdev=113854.72
lat (usec): min=418, max=9498.4k, avg=5056.25, stdev=114423.64
clat percentiles (usec):
| 1.00th=[ 685], 5.00th=[ 857], 10.00th=[ 955],
| 20.00th=[ 1074], 30.00th=[ 1188], 40.00th=[ 1287],
| 50.00th=[ 1418], 60.00th=[ 1631], 70.00th=[ 2180],
| 80.00th=[ 3032], 90.00th=[ 4752], 95.00th=[ 7832],
| 99.00th=[ 17171], 99.50th=[ 20055], 99.90th=[ 59507],
| 99.95th=[ 918553], 99.99th=[5737808]
bw ( KiB/s): min= 512, max=2663424, per=41.53%, avg=811549.54, stdev=498872.78, samples=1085
iops : min= 2, max=10404, avg=3170.11, stdev=1948.72, samples=1085
write: IOPS=1272, BW=318MiB/s (333MB/s)(280GiB/901390msec); 0 zone resets
slat (usec): min=17, max=7010.7k, avg=246.03, stdev=18576.42
clat (usec): min=7, max=9498.2k, avg=4513.35, stdev=108187.37
lat (usec): min=433, max=9498.3k, avg=4759.52, stdev=109821.97
clat percentiles (usec):
| 1.00th=[ 668], 5.00th=[ 848], 10.00th=[ 938],
| 20.00th=[ 1074], 30.00th=[ 1172], 40.00th=[ 1270],
| 50.00th=[ 1385], 60.00th=[ 1565], 70.00th=[ 1991],
| 80.00th=[ 2835], 90.00th=[ 4490], 95.00th=[ 7504],
| 99.00th=[ 16909], 99.50th=[ 19792], 99.90th=[ 50594],
| 99.95th=[ 350225], 99.99th=[5603591]
bw ( KiB/s): min= 512, max=1752064, per=41.84%, avg=544907.86, stdev=330632.85, samples=1077
iops : min= 2, max= 6844, avg=2128.55, stdev=1291.53, samples=1077
lat (usec) : 10=0.01%, 500=0.02%, 750=2.13%, 1000=11.57%
lat (msec) : 2=54.87%, 4=18.91%, 10=9.22%, 20=2.79%, 50=0.38%
lat (msec) : 100=0.03%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%, >=2000=0.04%
cpu : usr=0.89%, sys=52.00%, ctx=11350080, majf=0, minf=19
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1720322,1146597,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=46322: Thu Apr 16 23:02:16 2020
read: IOPS=1908, BW=477MiB/s (500MB/s)(420GiB/901390msec)
slat (usec): min=10, max=3293.5k, avg=200.37, stdev=6405.68
clat (usec): min=323, max=9485.2k, avg=4939.08, stdev=116035.25
lat (usec): min=344, max=9485.2k, avg=5139.56, stdev=116579.45
clat percentiles (usec):
| 1.00th=[ 652], 5.00th=[ 865], 10.00th=[ 963],
| 20.00th=[ 1090], 30.00th=[ 1188], 40.00th=[ 1303],
| 50.00th=[ 1434], 60.00th=[ 1631], 70.00th=[ 2147],
| 80.00th=[ 2999], 90.00th=[ 4621], 95.00th=[ 7635],
| 99.00th=[ 17171], 99.50th=[ 20055], 99.90th=[ 58983],
| 99.95th=[1216349], 99.99th=[5737808]
bw ( KiB/s): min= 512, max=2759680, per=42.15%, avg=823612.52, stdev=494622.50, samples=1069
iops : min= 2, max=10780, avg=3217.23, stdev=1932.12, samples=1069
write: IOPS=1272, BW=318MiB/s (334MB/s)(280GiB/901390msec); 0 zone resets
slat (usec): min=17, max=7010.9k, avg=246.65, stdev=18480.59
clat (usec): min=8, max=9484.8k, avg=4382.28, stdev=105404.90
lat (usec): min=356, max=9484.9k, avg=4629.06, stdev=107053.86
clat percentiles (usec):
| 1.00th=[ 644], 5.00th=[ 848], 10.00th=[ 955],
| 20.00th=[ 1074], 30.00th=[ 1172], 40.00th=[ 1270],
| 50.00th=[ 1401], 60.00th=[ 1565], 70.00th=[ 1958],
| 80.00th=[ 2802], 90.00th=[ 4424], 95.00th=[ 7439],
| 99.00th=[ 16909], 99.50th=[ 19530], 99.90th=[ 45876],
| 99.95th=[ 299893], 99.99th=[5670700]
bw ( KiB/s): min= 512, max=1901056, per=42.32%, avg=551189.49, stdev=329238.16, samples=1065
iops : min= 2, max= 7426, avg=2153.08, stdev=1286.09, samples=1065
lat (usec) : 10=0.01%, 500=0.12%, 750=2.12%, 1000=10.72%
lat (msec) : 2=56.17%, 4=18.62%, 10=9.04%, 20=2.73%, 50=0.38%
lat (msec) : 100=0.03%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%, >=2000=0.04%
cpu : usr=0.87%, sys=52.48%, ctx=11428364, majf=0, minf=21
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1720153,1146882,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=46323: Thu Apr 16 23:02:16 2020
read: IOPS=1907, BW=477MiB/s (500MB/s)(420GiB/901390msec)
slat (usec): min=10, max=3293.0k, avg=199.17, stdev=5943.95
clat (usec): min=426, max=9500.2k, avg=4796.43, stdev=112188.75
lat (usec): min=453, max=9500.2k, avg=4995.71, stdev=112701.60
clat percentiles (usec):
| 1.00th=[ 685], 5.00th=[ 865], 10.00th=[ 955],
| 20.00th=[ 1090], 30.00th=[ 1188], 40.00th=[ 1287],
| 50.00th=[ 1418], 60.00th=[ 1631], 70.00th=[ 2147],
| 80.00th=[ 3032], 90.00th=[ 4686], 95.00th=[ 7767],
| 99.00th=[ 16909], 99.50th=[ 19792], 99.90th=[ 56361],
| 99.95th=[ 759170], 99.99th=[5737808]
bw ( KiB/s): min= 512, max=2656256, per=41.86%, avg=817886.86, stdev=490288.18, samples=1076
iops : min= 2, max=10376, avg=3194.87, stdev=1915.19, samples=1076
write: IOPS=1272, BW=318MiB/s (334MB/s)(280GiB/901390msec); 0 zone resets
slat (usec): min=19, max=7010.9k, avg=252.68, stdev=18856.09
clat (usec): min=8, max=9500.1k, avg=4599.49, stdev=112152.04
lat (usec): min=466, max=9500.2k, avg=4852.30, stdev=113800.28
clat percentiles (usec):
| 1.00th=[ 676], 5.00th=[ 848], 10.00th=[ 947],
| 20.00th=[ 1074], 30.00th=[ 1172], 40.00th=[ 1270],
| 50.00th=[ 1385], 60.00th=[ 1565], 70.00th=[ 1975],
| 80.00th=[ 2835], 90.00th=[ 4490], 95.00th=[ 7439],
| 99.00th=[ 16909], 99.50th=[ 19792], 99.90th=[ 50594],
| 99.95th=[ 434111], 99.99th=[5737808]
bw ( KiB/s): min= 512, max=1813504, per=42.28%, avg=550665.49, stdev=324374.68, samples=1066
iops : min= 2, max= 7084, avg=2151.03, stdev=1267.09, samples=1066
lat (usec) : 10=0.01%, 500=0.01%, 750=1.98%, 1000=11.29%
lat (msec) : 2=55.58%, 4=18.75%, 10=9.13%, 20=2.79%, 50=0.37%
lat (msec) : 100=0.03%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%, >=2000=0.04%
cpu : usr=0.87%, sys=52.21%, ctx=11309634, majf=0, minf=22
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1719377,1146884,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=46324: Thu Apr 16 23:02:16 2020
read: IOPS=1910, BW=478MiB/s (501MB/s)(420GiB/900276msec)
slat (usec): min=10, max=3292.0k, avg=208.68, stdev=7003.33
clat (usec): min=7, max=9491.9k, avg=4823.59, stdev=112823.41
lat (usec): min=456, max=9491.0k, avg=5032.38, stdev=113451.79
clat percentiles (usec):
| 1.00th=[ 668], 5.00th=[ 857], 10.00th=[ 955],
| 20.00th=[ 1090], 30.00th=[ 1188], 40.00th=[ 1303],
| 50.00th=[ 1434], 60.00th=[ 1680], 70.00th=[ 2245],
| 80.00th=[ 3064], 90.00th=[ 4752], 95.00th=[ 7832],
| 99.00th=[ 17171], 99.50th=[ 20055], 99.90th=[ 58983],
| 99.95th=[ 692061], 99.99th=[5872026]
bw ( KiB/s): min= 512, max=2155008, per=41.53%, avg=811541.40, stdev=486199.95, samples=1085
iops : min= 2, max= 8418, avg=3170.08, stdev=1899.22, samples=1085
write: IOPS=1272, BW=318MiB/s (333MB/s)(280GiB/900276msec); 0 zone resets
slat (usec): min=19, max=7010.8k, avg=246.79, stdev=18222.38
clat (usec): min=406, max=9491.6k, avg=4546.45, stdev=108194.79
lat (usec): min=462, max=9491.7k, avg=4793.38, stdev=109812.04
clat percentiles (usec):
| 1.00th=[ 660], 5.00th=[ 848], 10.00th=[ 947],
| 20.00th=[ 1074], 30.00th=[ 1172], 40.00th=[ 1287],
| 50.00th=[ 1401], 60.00th=[ 1598], 70.00th=[ 2057],
| 80.00th=[ 2900], 90.00th=[ 4555], 95.00th=[ 7504],
| 99.00th=[ 16909], 99.50th=[ 19792], 99.90th=[ 52691],
| 99.95th=[ 358613], 99.99th=[5737808]
bw ( KiB/s): min= 512, max=1421824, per=41.56%, avg=541277.40, stdev=323462.89, samples=1083
iops : min= 2, max= 5554, avg=2114.36, stdev=1263.52, samples=1083
lat (usec) : 10=0.01%, 500=0.02%, 750=2.22%, 1000=10.98%
lat (msec) : 2=54.39%, 4=19.69%, 10=9.43%, 20=2.78%, 50=0.38%
lat (msec) : 100=0.03%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%, >=2000=0.04%
cpu : usr=0.87%, sys=51.56%, ctx=10983262, majf=0, minf=22
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1720324,1145310,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=1908MiB/s (2001MB/s), 477MiB/s-478MiB/s (500MB/s-501MB/s), io=1680GiB (1804GB), run=900276-901390msec
WRITE: bw=1272MiB/s (1334MB/s), 318MiB/s-318MiB/s (333MB/s-334MB/s), io=1120GiB (1202GB), run=900276-901390msec
ext4 filesystem tests:
fio-rand-RW.fio:
ssd array:
file1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.18
Starting 4 processes
file1: Laying out IO file (1 file / 10240MiB)
Jobs: 4 (f=4): [m(4)][100.0%][eta 00m:00s]
file1: (groupid=0, jobs=1): err= 0: pid=4187: Thu Apr 16 20:47:28 2020
read: IOPS=26.8k, BW=105MiB/s (110MB/s)(92.2GiB/902250msec)
slat (nsec): min=521, max=22937M, avg=24337.34, stdev=12851101.99
clat (usec): min=7, max=25704k, avg=336.13, stdev=58337.67
lat (usec): min=22, max=25704k, avg=360.52, stdev=59985.21
clat percentiles (usec):
| 1.00th=[ 26], 5.00th=[ 29], 10.00th=[ 31], 20.00th=[ 35],
| 30.00th=[ 38], 40.00th=[ 42], 50.00th=[ 46], 60.00th=[ 52],
| 70.00th=[ 72], 80.00th=[ 178], 90.00th=[ 351], 95.00th=[ 570],
| 99.00th=[ 1385], 99.50th=[ 1860], 99.90th=[ 2868], 99.95th=[ 3359],
| 99.99th=[ 5735]
bw ( KiB/s): min= 8, max=850920, per=54.82%, avg=234830.66, stdev=237231.04, samples=823
iops : min= 2, max=212730, avg=58707.67, stdev=59307.77, samples=823
write: IOPS=17.8k, BW=69.7MiB/s (73.1MB/s)(61.4GiB/902250msec); 0 zone resets
slat (nsec): min=751, max=19321M, avg=12303.51, stdev=10578386.16
clat (usec): min=20, max=25704k, avg=336.73, stdev=60134.65
lat (usec): min=22, max=25704k, avg=349.08, stdev=61123.80
clat percentiles (usec):
| 1.00th=[ 26], 5.00th=[ 29], 10.00th=[ 31], 20.00th=[ 35],
| 30.00th=[ 39], 40.00th=[ 42], 50.00th=[ 46], 60.00th=[ 52],
| 70.00th=[ 72], 80.00th=[ 178], 90.00th=[ 351], 95.00th=[ 570],
| 99.00th=[ 1385], 99.50th=[ 1860], 99.90th=[ 2868], 99.95th=[ 3392],
| 99.99th=[ 5800]
bw ( KiB/s): min= 8, max=564768, per=54.54%, avg=155777.15, stdev=158137.96, samples=827
iops : min= 2, max=141192, avg=38944.29, stdev=39534.49, samples=827
lat (usec) : 10=0.01%, 50=57.99%, 100=12.64%, 250=14.02%, 500=9.16%
lat (usec) : 750=2.97%, 1000=1.30%
lat (msec) : 2=1.50%, 4=0.37%, 10=0.03%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%, >=2000=0.01%
cpu : usr=3.06%, sys=18.16%, ctx=6486230, majf=0, minf=24
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=24158675,16103789,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=4188: Thu Apr 16 20:47:28 2020
read: IOPS=26.8k, BW=105MiB/s (110MB/s)(92.1GiB/902250msec)
slat (nsec): min=511, max=19324M, avg=24262.33, stdev=11823631.98
clat (usec): min=7, max=25701k, avg=331.05, stdev=56646.11
lat (usec): min=22, max=25701k, avg=355.36, stdev=58246.64
clat percentiles (usec):
| 1.00th=[ 26], 5.00th=[ 29], 10.00th=[ 31], 20.00th=[ 35],
| 30.00th=[ 38], 40.00th=[ 42], 50.00th=[ 46], 60.00th=[ 52],
| 70.00th=[ 72], 80.00th=[ 178], 90.00th=[ 355], 95.00th=[ 570],
| 99.00th=[ 1385], 99.50th=[ 1860], 99.90th=[ 2835], 99.95th=[ 3326],
| 99.99th=[ 5735]
bw ( KiB/s): min= 8, max=845840, per=54.28%, avg=232522.77, stdev=236563.19, samples=831
iops : min= 2, max=211460, avg=58130.68, stdev=59140.78, samples=831
write: IOPS=17.9k, BW=69.7MiB/s (73.1MB/s)(61.5GiB/902250msec); 0 zone resets
slat (nsec): min=751, max=18028M, avg=12467.36, stdev=10905218.24
clat (usec): min=20, max=25701k, avg=344.32, stdev=59589.00
lat (usec): min=23, max=25701k, avg=356.83, stdev=60581.24
clat percentiles (usec):
| 1.00th=[ 26], 5.00th=[ 29], 10.00th=[ 31], 20.00th=[ 35],
| 30.00th=[ 39], 40.00th=[ 42], 50.00th=[ 46], 60.00th=[ 52],
| 70.00th=[ 72], 80.00th=[ 178], 90.00th=[ 351], 95.00th=[ 570],
| 99.00th=[ 1385], 99.50th=[ 1860], 99.90th=[ 2868], 99.95th=[ 3359],
| 99.99th=[ 5735]
bw ( KiB/s): min= 8, max=564408, per=54.43%, avg=155455.95, stdev=157786.99, samples=829
iops : min= 2, max=141104, avg=38863.98, stdev=39446.74, samples=829
lat (usec) : 10=0.01%, 50=57.83%, 100=12.83%, 250=13.96%, 500=9.19%
lat (usec) : 750=2.98%, 1000=1.29%
lat (msec) : 2=1.50%, 4=0.37%, 10=0.03%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%, >=2000=0.01%
cpu : usr=3.09%, sys=18.11%, ctx=6347526, majf=0, minf=25
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=24153782,16109439,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=4189: Thu Apr 16 20:47:28 2020
read: IOPS=26.8k, BW=105MiB/s (110MB/s)(92.1GiB/902250msec)
slat (nsec): min=511, max=19321M, avg=22579.21, stdev=10573514.31
clat (usec): min=7, max=25718k, avg=333.62, stdev=57753.62
lat (usec): min=22, max=25718k, avg=356.25, stdev=59144.99
clat percentiles (usec):
| 1.00th=[ 26], 5.00th=[ 29], 10.00th=[ 31], 20.00th=[ 35],
| 30.00th=[ 38], 40.00th=[ 42], 50.00th=[ 46], 60.00th=[ 51],
| 70.00th=[ 71], 80.00th=[ 178], 90.00th=[ 351], 95.00th=[ 578],
| 99.00th=[ 1385], 99.50th=[ 1860], 99.90th=[ 2868], 99.95th=[ 3392],
| 99.99th=[ 5866]
bw ( KiB/s): min= 8, max=850224, per=54.47%, avg=233345.08, stdev=237852.27, samples=828
iops : min= 2, max=212556, avg=58336.27, stdev=59463.07, samples=828
write: IOPS=17.9k, BW=69.7MiB/s (73.1MB/s)(61.4GiB/902250msec); 0 zone resets
slat (nsec): min=742, max=18031M, avg=14999.88, stdev=12288734.93
clat (usec): min=21, max=25718k, avg=340.53, stdev=58896.28
lat (usec): min=22, max=25718k, avg=355.58, stdev=60247.68
clat percentiles (usec):
| 1.00th=[ 26], 5.00th=[ 29], 10.00th=[ 31], 20.00th=[ 35],
| 30.00th=[ 39], 40.00th=[ 42], 50.00th=[ 46], 60.00th=[ 52],
| 70.00th=[ 71], 80.00th=[ 178], 90.00th=[ 351], 95.00th=[ 570],
| 99.00th=[ 1385], 99.50th=[ 1860], 99.90th=[ 2868], 99.95th=[ 3359],
| 99.99th=[ 5800]
bw ( KiB/s): min= 8, max=569056, per=54.36%, avg=155259.24, stdev=158673.45, samples=830
iops : min= 2, max=142264, avg=38814.81, stdev=39668.36, samples=830
lat (usec) : 10=0.01%, 50=58.15%, 100=12.55%, 250=13.97%, 500=9.15%
lat (usec) : 750=2.97%, 1000=1.30%
lat (msec) : 2=1.50%, 4=0.38%, 10=0.03%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%, >=2000=0.01%
cpu : usr=3.08%, sys=18.10%, ctx=6328985, majf=0, minf=24
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=24151688,16108475,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=4190: Thu Apr 16 20:47:28 2020
read: IOPS=26.8k, BW=105MiB/s (110MB/s)(92.2GiB/902250msec)
slat (nsec): min=511, max=20650M, avg=23949.60, stdev=11590584.38
clat (usec): min=8, max=25693k, avg=338.63, stdev=58590.52
lat (usec): min=21, max=25693k, avg=362.64, stdev=60059.25
clat percentiles (usec):
| 1.00th=[ 26], 5.00th=[ 29], 10.00th=[ 31], 20.00th=[ 35],
| 30.00th=[ 38], 40.00th=[ 42], 50.00th=[ 46], 60.00th=[ 51],
| 70.00th=[ 71], 80.00th=[ 178], 90.00th=[ 351], 95.00th=[ 570],
| 99.00th=[ 1385], 99.50th=[ 1860], 99.90th=[ 2868], 99.95th=[ 3359],
| 99.99th=[ 5735]
bw ( KiB/s): min= 8, max=855224, per=54.49%, avg=233415.42, stdev=237744.80, samples=828
iops : min= 2, max=213806, avg=58353.86, stdev=59436.20, samples=828
write: IOPS=17.8k, BW=69.7MiB/s (73.1MB/s)(61.4GiB/902250msec); 0 zone resets
slat (nsec): min=741, max=19324M, avg=12901.57, stdev=11251956.87
clat (usec): min=20, max=25692k, avg=332.98, stdev=58218.59
lat (usec): min=22, max=25692k, avg=345.93, stdev=59474.78
clat percentiles (usec):
| 1.00th=[ 26], 5.00th=[ 29], 10.00th=[ 31], 20.00th=[ 35],
| 30.00th=[ 38], 40.00th=[ 42], 50.00th=[ 46], 60.00th=[ 52],
| 70.00th=[ 72], 80.00th=[ 178], 90.00th=[ 351], 95.00th=[ 570],
| 99.00th=[ 1385], 99.50th=[ 1860], 99.90th=[ 2868], 99.95th=[ 3392],
| 99.99th=[ 5866]
bw ( KiB/s): min= 8, max=572496, per=54.67%, avg=156138.08, stdev=158425.86, samples=825
iops : min= 2, max=143124, avg=39034.52, stdev=39606.47, samples=825
lat (usec) : 10=0.01%, 50=58.27%, 100=12.37%, 250=14.02%, 500=9.17%
lat (usec) : 750=2.98%, 1000=1.29%
lat (msec) : 2=1.50%, 4=0.38%, 10=0.03%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%, >=2000=0.01%
cpu : usr=2.99%, sys=18.21%, ctx=6419172, majf=0, minf=23
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=24158978,16102061,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=418MiB/s (439MB/s), 105MiB/s-105MiB/s (110MB/s-110MB/s), io=369GiB (396GB), run=902250-902250msec
WRITE: bw=279MiB/s (292MB/s), 69.7MiB/s-69.7MiB/s (73.1MB/s-73.1MB/s), io=246GiB (264GB), run=902250-902250msec
Disk stats (read/write):
dm-4: ios=5993954/14497630, merge=0/0, ticks=1634288/1037657607, in_queue=1039291895, util=97.78%, aggrios=5998052/14574477, aggrmerge=0/0, aggrticks=2123171/1462952472, aggrin_queue=1465076704, aggrutil=97.99%
dm-0: ios=5998052/14574477, merge=0/0, ticks=2123171/1462952472, in_queue=1465076704, util=97.99%, aggrios=5998766/14562384, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
md128: ios=5998766/14562384, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=2999228/5929551, aggrmerge=147/1352125, aggrticks=524887/1056422, aggrin_queue=47791, aggrutil=97.33%
sdd: ios=2997639/5937272, merge=110/1346600, ticks=387298/1487985, in_queue=87156, util=93.40%
sda: ios=3000817/5921831, merge=185/1357651, ticks=662477/624859, in_queue=8426, util=97.33%
ssd-test.fio:
ssd array:
seq-read: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
rand-read: (g=1): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
seq-write: (g=2): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
rand-write: (g=3): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
fio-3.18
Starting 4 processes
seq-read: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [_(3),w(1)][99.2%][w=169MiB/s][w=43.3k IOPS][eta 00m:02s]
seq-read: (groupid=0, jobs=1): err= 0: pid=25992: Thu Apr 16 20:59:39 2020
read: IOPS=47.4k, BW=185MiB/s (194MB/s)(10.0GiB/55281msec)
slat (nsec): min=1683, max=301896, avg=6826.06, stdev=3563.61
clat (usec): min=16, max=7686, avg=76.89, stdev=41.30
lat (usec): min=39, max=7693, avg=83.83, stdev=40.16
clat percentiles (usec):
| 1.00th=[ 46], 5.00th=[ 55], 10.00th=[ 57], 20.00th=[ 59],
| 30.00th=[ 61], 40.00th=[ 63], 50.00th=[ 75], 60.00th=[ 87],
| 70.00th=[ 90], 80.00th=[ 92], 90.00th=[ 95], 95.00th=[ 97],
| 99.00th=[ 182], 99.50th=[ 206], 99.90th=[ 289], 99.95th=[ 355],
| 99.99th=[ 1582]
bw ( KiB/s): min=162408, max=199288, per=100.00%, avg=189975.23, stdev=5129.09, samples=110
iops : min=40602, max=49822, avg=47493.80, stdev=1282.28, samples=110
lat (usec) : 20=0.01%, 50=2.25%, 100=94.94%, 250=2.66%, 500=0.13%
lat (usec) : 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%
cpu : usr=5.25%, sys=33.18%, ctx=1879641, majf=0, minf=12
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=2621440,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
rand-read: (groupid=1, jobs=1): err= 0: pid=26072: Thu Apr 16 20:59:39 2020
read: IOPS=21.1k, BW=82.6MiB/s (86.6MB/s)(4953MiB/60001msec)
slat (usec): min=2, max=339, avg= 5.07, stdev= 1.96
clat (usec): min=23, max=28131, avg=183.42, stdev=131.33
lat (usec): min=64, max=28139, avg=188.62, stdev=130.84
clat percentiles (usec):
| 1.00th=[ 85], 5.00th=[ 88], 10.00th=[ 90], 20.00th=[ 93],
| 30.00th=[ 96], 40.00th=[ 112], 50.00th=[ 135], 60.00th=[ 180],
| 70.00th=[ 253], 80.00th=[ 269], 90.00th=[ 347], 95.00th=[ 367],
| 99.00th=[ 461], 99.50th=[ 519], 99.90th=[ 660], 99.95th=[ 865],
| 99.99th=[ 2474]
bw ( KiB/s): min=74800, max=89048, per=100.00%, avg=84649.53, stdev=2408.84, samples=119
iops : min=18700, max=22262, avg=21162.38, stdev=602.20, samples=119
lat (usec) : 50=0.01%, 100=35.20%, 250=33.60%, 500=30.60%, 750=0.53%
lat (usec) : 1000=0.02%
lat (msec) : 2=0.02%, 4=0.02%, 10=0.01%, 20=0.01%, 50=0.01%
cpu : usr=3.11%, sys=12.85%, ctx=978765, majf=0, minf=12
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1268052,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
seq-write: (groupid=2, jobs=1): err= 0: pid=26145: Thu Apr 16 20:59:39 2020
write: IOPS=42.8k, BW=167MiB/s (175MB/s)(9.79GiB/60003msec); 0 zone resets
slat (nsec): min=1543, max=10207k, avg=4043.19, stdev=6419.44
clat (usec): min=35, max=82562, avg=88.87, stdev=139.06
lat (usec): min=44, max=82565, avg=93.03, stdev=139.21
clat percentiles (usec):
| 1.00th=[ 55], 5.00th=[ 74], 10.00th=[ 76], 20.00th=[ 78],
| 30.00th=[ 79], 40.00th=[ 80], 50.00th=[ 84], 60.00th=[ 95],
| 70.00th=[ 96], 80.00th=[ 97], 90.00th=[ 100], 95.00th=[ 102],
| 99.00th=[ 178], 99.50th=[ 212], 99.90th=[ 302], 99.95th=[ 359],
| 99.99th=[ 832]
bw ( KiB/s): min=145760, max=176824, per=100.00%, avg=171344.39, stdev=4452.16, samples=119
iops : min=36440, max=44206, avg=42836.09, stdev=1113.04, samples=119
lat (usec) : 50=0.31%, 100=91.40%, 250=8.09%, 500=0.17%, 750=0.01%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=5.71%, sys=20.67%, ctx=1905968, majf=0, minf=14
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,2566351,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
rand-write: (groupid=3, jobs=1): err= 0: pid=26269: Thu Apr 16 20:59:39 2020
write: IOPS=42.1k, BW=165MiB/s (173MB/s)(9871MiB/60001msec); 0 zone resets
slat (nsec): min=1653, max=154630, avg=4258.08, stdev=690.88
clat (usec): min=14, max=69914, avg=89.99, stdev=138.70
lat (usec): min=39, max=69919, avg=94.37, stdev=138.71
clat percentiles (usec):
| 1.00th=[ 44], 5.00th=[ 46], 10.00th=[ 47], 20.00th=[ 50],
| 30.00th=[ 53], 40.00th=[ 58], 50.00th=[ 67], 60.00th=[ 85],
| 70.00th=[ 115], 80.00th=[ 127], 90.00th=[ 161], 95.00th=[ 174],
| 99.00th=[ 260], 99.50th=[ 302], 99.90th=[ 420], 99.95th=[ 515],
| 99.99th=[ 2311]
bw ( KiB/s): min=136816, max=189304, per=100.00%, avg=168739.86, stdev=11097.33, samples=119
iops : min=34204, max=47326, avg=42184.97, stdev=2774.35, samples=119
lat (usec) : 20=0.01%, 50=22.63%, 100=42.21%, 250=33.94%, 500=1.16%
lat (usec) : 750=0.03%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=6.15%, sys=22.35%, ctx=2177658, majf=0, minf=12
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,2526915,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=185MiB/s (194MB/s), 185MiB/s-185MiB/s (194MB/s-194MB/s), io=10.0GiB (10.7GB), run=55281-55281msec
Run status group 1 (all jobs):
READ: bw=82.6MiB/s (86.6MB/s), 82.6MiB/s-82.6MiB/s (86.6MB/s-86.6MB/s), io=4953MiB (5194MB), run=60001-60001msec
Run status group 2 (all jobs):
WRITE: bw=167MiB/s (175MB/s), 167MiB/s-167MiB/s (175MB/s-175MB/s), io=9.79GiB (10.5GB), run=60003-60003msec
Run status group 3 (all jobs):
WRITE: bw=165MiB/s (173MB/s), 165MiB/s-165MiB/s (173MB/s-173MB/s), io=9871MiB (10.3GB), run=60001-60001msec
Disk stats (read/write):
dm-4: ios=3889492/5087822, merge=0/0, ticks=435735/446565, in_queue=882300, util=99.40%, aggrios=3890468/5113230, aggrmerge=0/0, aggrticks=433045/1974911, aggrin_queue=2408127, aggrutil=99.55%
dm-0: ios=3890468/5113230, merge=0/0, ticks=433045/1974911, in_queue=2408127, util=99.55%, aggrios=3890470/5113458, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
md128: ios=3890470/5113458, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=1621878/2239322, aggrmerge=323357/317244, aggrticks=175205/174238, aggrin_queue=1788, aggrutil=87.64%
sdd: ios=1945635/2553523, merge=123/2765, ticks=134406/145088, in_queue=3176, util=81.04%
sda: ios=1298121/1925121, merge=646591/631723, ticks=216004/203388, in_queue=400, util=87.64%
fio-seq-RW.fio:
ssd array:
file1: (g=0): rw=rw, bs=(R) 256KiB-256KiB, (W) 256KiB-256KiB, (T) 256KiB-256KiB, ioengine=libaio, iodepth=16
...
fio-3.18
Starting 4 processes
file1: Laying out IO file (1 file / 10240MiB)
Jobs: 4 (f=4): [M(4)][100.0%][r=4876MiB/s,w=3196MiB/s][r=19.5k,w=12.8k IOPS][eta 00m:00s]
file1: (groupid=0, jobs=1): err= 0: pid=31629: Thu Apr 16 22:02:18 2020
read: IOPS=1755, BW=439MiB/s (460MB/s)(386GiB/900001msec)
slat (usec): min=10, max=6001.5k, avg=243.42, stdev=13604.69
clat (usec): min=239, max=10819k, avg=5185.92, stdev=109885.26
lat (usec): min=281, max=10819k, avg=5429.46, stdev=111139.70
clat percentiles (usec):
| 1.00th=[ 889], 5.00th=[ 1221], 10.00th=[ 1418],
| 20.00th=[ 1680], 30.00th=[ 1893], 40.00th=[ 2114],
| 50.00th=[ 2376], 60.00th=[ 2737], 70.00th=[ 3261],
| 80.00th=[ 4015], 90.00th=[ 5276], 95.00th=[ 6783],
| 99.00th=[ 10945], 99.50th=[ 13304], 99.90th=[ 36963],
| 99.95th=[ 792724], 99.99th=[7281312]
bw ( KiB/s): min= 512, max=1683968, per=38.57%, avg=693499.08, stdev=320473.55, samples=1165
iops : min= 2, max= 6578, avg=2708.97, stdev=1251.85, samples=1165
write: IOPS=1170, BW=293MiB/s (307MB/s)(257GiB/900001msec); 0 zone resets
slat (usec): min=24, max=4547.7k, avg=272.29, stdev=10969.43
clat (nsec): min=1943, max=10818M, avg=5038702.95, stdev=110500052.27
lat (usec): min=238, max=10818k, avg=5311.12, stdev=111106.80
clat percentiles (usec):
| 1.00th=[ 865], 5.00th=[ 1188], 10.00th=[ 1385],
| 20.00th=[ 1631], 30.00th=[ 1844], 40.00th=[ 2040],
| 50.00th=[ 2278], 60.00th=[ 2540], 70.00th=[ 2999],
| 80.00th=[ 3720], 90.00th=[ 4948], 95.00th=[ 6521],
| 99.00th=[ 10683], 99.50th=[ 13042], 99.90th=[ 36963],
| 99.95th=[ 650118], 99.99th=[7281312]
bw ( KiB/s): min= 512, max=1103872, per=38.91%, avg=466225.56, stdev=210492.39, samples=1155
iops : min= 2, max= 4312, avg=1821.18, stdev=822.24, samples=1155
lat (usec) : 2=0.01%, 250=0.01%, 500=0.01%, 750=0.36%, 1000=1.58%
lat (msec) : 2=34.04%, 4=45.13%, 10=17.43%, 20=1.27%, 50=0.10%
lat (msec) : 100=0.02%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%, >=2000=0.04%
cpu : usr=0.82%, sys=56.44%, ctx=7701664, majf=0, minf=21
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1580114,1053170,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=31630: Thu Apr 16 22:02:18 2020
read: IOPS=1755, BW=439MiB/s (460MB/s)(386GiB/900001msec)
slat (usec): min=11, max=6001.3k, avg=236.29, stdev=12943.27
clat (usec): min=418, max=9439.3k, avg=5315.54, stdev=111617.09
lat (usec): min=435, max=9439.3k, avg=5551.96, stdev=112738.52
clat percentiles (usec):
| 1.00th=[ 906], 5.00th=[ 1237], 10.00th=[ 1434],
| 20.00th=[ 1713], 30.00th=[ 1926], 40.00th=[ 2147],
| 50.00th=[ 2409], 60.00th=[ 2737], 70.00th=[ 3228],
| 80.00th=[ 3982], 90.00th=[ 5211], 95.00th=[ 6783],
| 99.00th=[ 10945], 99.50th=[ 13173], 99.90th=[ 41157],
| 99.95th=[1132463], 99.99th=[7214203]
bw ( KiB/s): min= 512, max=1685504, per=38.81%, avg=697806.97, stdev=310838.43, samples=1158
iops : min= 2, max= 6584, avg=2725.80, stdev=1214.22, samples=1158
write: IOPS=1170, BW=293MiB/s (307MB/s)(257GiB/900001msec); 0 zone resets
slat (usec): min=27, max=4547.6k, avg=283.11, stdev=11936.31
clat (nsec): min=1353, max=9438.9M, avg=4841527.39, stdev=107041842.36
lat (usec): min=278, max=9438.0k, avg=5124.77, stdev=107745.97
clat percentiles (usec):
| 1.00th=[ 889], 5.00th=[ 1221], 10.00th=[ 1401],
| 20.00th=[ 1663], 30.00th=[ 1860], 40.00th=[ 2057],
| 50.00th=[ 2278], 60.00th=[ 2573], 70.00th=[ 2999],
| 80.00th=[ 3687], 90.00th=[ 4883], 95.00th=[ 6456],
| 99.00th=[ 10552], 99.50th=[ 12780], 99.90th=[ 34341],
| 99.95th=[ 170918], 99.99th=[7214203]
bw ( KiB/s): min= 512, max=1175552, per=38.88%, avg=465971.38, stdev=207316.31, samples=1156
iops : min= 2, max= 4592, avg=1820.19, stdev=809.83, samples=1156
lat (usec) : 2=0.01%, 500=0.01%, 750=0.31%, 1000=1.47%
lat (msec) : 2=33.17%, 4=46.48%, 10=17.13%, 20=1.25%, 50=0.11%
lat (msec) : 100=0.02%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%, >=2000=0.04%
cpu : usr=0.81%, sys=57.17%, ctx=7700112, majf=0, minf=24
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1580380,1053470,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=31631: Thu Apr 16 22:02:18 2020
read: IOPS=1755, BW=439MiB/s (460MB/s)(386GiB/900001msec)
slat (usec): min=10, max=6001.6k, avg=231.26, stdev=12228.15
clat (nsec): min=1082, max=9452.4M, avg=5205366.88, stdev=109985047.72
lat (usec): min=36, max=9452.4k, avg=5436.74, stdev=111086.16
clat percentiles (usec):
| 1.00th=[ 914], 5.00th=[ 1237], 10.00th=[ 1434],
| 20.00th=[ 1696], 30.00th=[ 1909], 40.00th=[ 2147],
| 50.00th=[ 2376], 60.00th=[ 2737], 70.00th=[ 3261],
| 80.00th=[ 4015], 90.00th=[ 5276], 95.00th=[ 6783],
| 99.00th=[ 10945], 99.50th=[ 13042], 99.90th=[ 37487],
| 99.95th=[ 884999], 99.99th=[7281312]
bw ( KiB/s): min= 512, max=1386496, per=38.87%, avg=698891.65, stdev=310544.20, samples=1156
iops : min= 2, max= 5416, avg=2730.03, stdev=1213.07, samples=1156
write: IOPS=1171, BW=293MiB/s (307MB/s)(257GiB/900001msec); 0 zone resets
slat (usec): min=29, max=4547.4k, avg=290.22, stdev=12847.83
clat (usec): min=303, max=9451.9k, avg=5004.77, stdev=108932.38
lat (usec): min=534, max=9452.1k, avg=5295.11, stdev=109724.17
clat percentiles (usec):
| 1.00th=[ 889], 5.00th=[ 1205], 10.00th=[ 1385],
| 20.00th=[ 1647], 30.00th=[ 1844], 40.00th=[ 2057],
| 50.00th=[ 2278], 60.00th=[ 2540], 70.00th=[ 2999],
| 80.00th=[ 3687], 90.00th=[ 4948], 95.00th=[ 6456],
| 99.00th=[ 10683], 99.50th=[ 12780], 99.90th=[ 36439],
| 99.95th=[ 734004], 99.99th=[7147095]
bw ( KiB/s): min= 512, max=912384, per=38.91%, avg=466288.45, stdev=207371.91, samples=1156
iops : min= 2, max= 3564, avg=1821.43, stdev=810.05, samples=1156
lat (usec) : 2=0.01%, 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%
lat (usec) : 750=0.33%, 1000=1.42%
lat (msec) : 2=33.79%, 4=45.70%, 10=17.30%, 20=1.27%, 50=0.10%
lat (msec) : 100=0.02%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%, >=2000=0.04%
cpu : usr=0.80%, sys=56.71%, ctx=7578493, majf=0, minf=24
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1580092,1054191,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=31632: Thu Apr 16 22:02:18 2020
read: IOPS=1755, BW=439MiB/s (460MB/s)(386GiB/900001msec)
slat (usec): min=10, max=7776.5k, avg=238.59, stdev=12921.86
clat (nsec): min=1242, max=9439.3M, avg=5241348.12, stdev=109996869.99
lat (usec): min=74, max=9439.4k, avg=5480.06, stdev=111197.22
clat percentiles (usec):
| 1.00th=[ 898], 5.00th=[ 1237], 10.00th=[ 1434],
| 20.00th=[ 1713], 30.00th=[ 1926], 40.00th=[ 2147],
| 50.00th=[ 2409], 60.00th=[ 2769], 70.00th=[ 3326],
| 80.00th=[ 4047], 90.00th=[ 5276], 95.00th=[ 6849],
| 99.00th=[ 10945], 99.50th=[ 13173], 99.90th=[ 33162],
| 99.95th=[ 884999], 99.99th=[7281312]
bw ( KiB/s): min= 512, max=1449984, per=38.38%, avg=689968.99, stdev=307834.81, samples=1171
iops : min= 2, max= 5664, avg=2695.18, stdev=1202.49, samples=1171
write: IOPS=1169, BW=292MiB/s (306MB/s)(257GiB/900001msec); 0 zone resets
slat (usec): min=29, max=4367.0k, avg=285.30, stdev=11879.82
clat (usec): min=75, max=9415.3k, avg=4960.03, stdev=107558.96
lat (usec): min=313, max=9415.4k, avg=5245.46, stdev=108213.90
clat percentiles (usec):
| 1.00th=[ 881], 5.00th=[ 1205], 10.00th=[ 1401],
| 20.00th=[ 1663], 30.00th=[ 1876], 40.00th=[ 2073],
| 50.00th=[ 2311], 60.00th=[ 2606], 70.00th=[ 3064],
| 80.00th=[ 3752], 90.00th=[ 5014], 95.00th=[ 6587],
| 99.00th=[ 10683], 99.50th=[ 12911], 99.90th=[ 32900],
| 99.95th=[ 450888], 99.99th=[7147095]
bw ( KiB/s): min= 512, max=935424, per=38.67%, avg=463388.66, stdev=201307.47, samples=1161
iops : min= 2, max= 3654, avg=1810.10, stdev=786.36, samples=1161
lat (usec) : 2=0.01%, 100=0.01%, 500=0.01%, 750=0.38%, 1000=1.43%
lat (msec) : 2=32.60%, 4=46.20%, 10=17.92%, 20=1.29%, 50=0.10%
lat (msec) : 100=0.02%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%, >=2000=0.04%
cpu : usr=0.80%, sys=56.16%, ctx=7520294, majf=0, minf=25
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1580181,1052150,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=1756MiB/s (1841MB/s), 439MiB/s-439MiB/s (460MB/s-460MB/s), io=1543GiB (1657GB), run=900001-900001msec
WRITE: bw=1170MiB/s (1227MB/s), 292MiB/s-293MiB/s (306MB/s-307MB/s), io=1029GiB (1104GB), run=900001-900001msec
Disk stats (read/write):
dm-4: ios=3563816/5044708, merge=0/0, ticks=13307671/18446744071674149170, in_queue=18446744071687456841, util=70.15%, aggrios=3565147/5185081, aggrmerge=0/0, aggrticks=13426178/18446744071890657751, aggrin_queue=18446744071904087582, aggrutil=73.91%
dm-0: ios=3565147/5185081, merge=0/0, ticks=13426178/18446744071890657751, in_queue=18446744071904087582, util=73.91%, aggrios=4097296/5789839, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
md128: ios=4097296/5789839, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=1864869/331459, aggrmerge=135781/2563833, aggrticks=4842566/833110, aggrin_queue=4628710, aggrutil=65.40%
sdd: ios=1933063/340559, merge=81504/2554693, ticks=2903166/1168077, in_queue=3010995, util=64.10%
sda: ios=1796675/322360, merge=190058/2572974, ticks=6781967/498143, in_queue=6246426, util=65.40%
machine 2:
storage: WDC WDS500G2B0A-00SM50 (ssd)
storage layout:
cryptsetup luks with camellia-xts-plain64 + lvm
raw linear read io performance with hdparm -Tt (to understand bus limitations):
/dev/sda:
Timing cached reads: 5738 MB in 2.00 seconds = 2870.00 MB/sec
Timing buffered disk reads: 226 MB in 3.02 seconds = 74.74 MB/sec
for some reason very low speed here, probably because of external sata3 card
02:00.0 SATA controller: ASMedia Technology Inc. Device 0622 (rev 01)
smartctl show 6gb link speed here:
SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
anyway, let’s continue…
crypto performance:
[h1]Tests are approximate using memory only (no storage IO).[/h1]
[h1]Algorithm | Key | Encryption | Decryption[/h1]
twofish-xts 512b 181.5 MiB/s 186.3 MiB/s
[h1]Tests are approximate using memory only (no storage IO).[/h1]
[h1]Algorithm | Key | Encryption | Decryption[/h1]
serpent-xts 512b 170.6 MiB/s 179.2 MiB/s
[h1]Tests are approximate using memory only (no storage IO).[/h1]
[h1]Algorithm | Key | Encryption | Decryption[/h1]
aes-xts 512b 122.8 MiB/s 125.9 MiB/s
[h1]Tests are approximate using memory only (no storage IO).[/h1]
[h1]Algorithm | Key | Encryption | Decryption[/h1]
camellia-xts 512b 167.3 MiB/s 173.4 MiB/s
[h1]Tests are approximate using memory only (no storage IO).[/h1]
PBKDF2-ripemd160 129007 iterations per second for 512-bit key
[h1]Tests are approximate using memory only (no storage IO).[/h1]
PBKDF2-sha256 220659 iterations per second for 512-bit key
[h1]Tests are approximate using memory only (no storage IO).[/h1]
PBKDF2-sha512 332670 iterations per second for 512-bit key
[h1]Tests are approximate using memory only (no storage IO).[/h1]
PBKDF2-whirlpool 199197 iterations per second for 512-bit key
obvously camellia is not the best choice here for crypto, probably i should switch to twofish or serpent in future
xfs filesystem tests:
fio-rand-RW.fio:
file1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.18
Starting 4 processes
file1: Laying out IO file (1 file / 10240MiB)
Jobs: 4 (f=4): [m(4)][100.0%][r=32.4MiB/s,w=22.3MiB/s][r=8306,w=5708 IOPS][eta 00m:00s]
file1: (groupid=0, jobs=1): err= 0: pid=621474: Wed Apr 15 02:06:48 2020
read: IOPS=1935, BW=7741KiB/s (7926kB/s)(6803MiB/900001msec)
slat (nsec): min=1891, max=5893.4M, avg=300566.15, stdev=10037311.50
clat (usec): min=66, max=6080.1k, avg=4678.04, stdev=40385.49
lat (usec): min=69, max=6080.1k, avg=4979.19, stdev=42253.34
clat percentiles (usec):
| 1.00th=[ 685], 5.00th=[ 1352], 10.00th=[ 1778],
| 20.00th=[ 2278], 30.00th=[ 2638], 40.00th=[ 2999],
| 50.00th=[ 3392], 60.00th=[ 3851], 70.00th=[ 4359],
| 80.00th=[ 5145], 90.00th=[ 6521], 95.00th=[ 8455],
| 99.00th=[ 22676], 99.50th=[ 30016], 99.90th=[ 49021],
| 99.95th=[ 64226], 99.99th=[2734687]
bw ( KiB/s): min= 7, max=15936, per=27.09%, avg=8355.01, stdev=3468.68, samples=1666
iops : min= 1, max= 3984, avg=2088.62, stdev=867.19, samples=1666
write: IOPS=1289, BW=5159KiB/s (5283kB/s)(4534MiB/900001msec); 0 zone resets
slat (usec): min=2, max=3014.6k, avg=312.79, stdev=6120.86
clat (usec): min=8, max=6053.9k, avg=4614.70, stdev=41890.22
lat (usec): min=70, max=6080.6k, avg=4928.04, stdev=42745.15
clat percentiles (usec):
| 1.00th=[ 660], 5.00th=[ 1303], 10.00th=[ 1713],
| 20.00th=[ 2212], 30.00th=[ 2573], 40.00th=[ 2966],
| 50.00th=[ 3326], 60.00th=[ 3785], 70.00th=[ 4293],
| 80.00th=[ 5080], 90.00th=[ 6390], 95.00th=[ 8356],
| 99.00th=[ 21627], 99.50th=[ 29230], 99.90th=[ 47973],
| 99.95th=[ 63701], 99.99th=[2902459]
bw ( KiB/s): min= 7, max=10672, per=27.23%, avg=5598.61, stdev=2288.35, samples=1657
iops : min= 1, max= 2668, avg=1399.51, stdev=572.10, samples=1657
lat (usec) : 10=0.01%, 100=0.02%, 250=0.06%, 500=0.38%, 750=0.85%
lat (usec) : 1000=1.21%
lat (msec) : 2=12.05%, 4=49.09%, 10=32.74%, 20=2.38%, 50=1.13%
lat (msec) : 100=0.06%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%, >=2000=0.01%
cpu : usr=1.59%, sys=8.53%, ctx=2707503, majf=2, minf=20
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1741637,1160773,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=621475: Wed Apr 15 02:06:48 2020
read: IOPS=1922, BW=7690KiB/s (7874kB/s)(6759MiB/900001msec)
slat (nsec): min=1916, max=5612.2M, avg=294667.69, stdev=8843280.94
clat (usec): min=5, max=6000.1k, avg=4705.97, stdev=40069.66
lat (usec): min=69, max=6000.1k, avg=5001.22, stdev=41468.73
clat percentiles (usec):
| 1.00th=[ 693], 5.00th=[ 1385], 10.00th=[ 1795],
| 20.00th=[ 2278], 30.00th=[ 2671], 40.00th=[ 3032],
| 50.00th=[ 3425], 60.00th=[ 3851], 70.00th=[ 4424],
| 80.00th=[ 5145], 90.00th=[ 6587], 95.00th=[ 8586],
| 99.00th=[ 22676], 99.50th=[ 30016], 99.90th=[ 49021],
| 99.95th=[ 65274], 99.99th=[2835350]
bw ( KiB/s): min= 7, max=14352, per=26.88%, avg=8290.15, stdev=3431.62, samples=1668
iops : min= 1, max= 3588, avg=2072.39, stdev=857.92, samples=1668
write: IOPS=1281, BW=5127KiB/s (5250kB/s)(4506MiB/900001msec); 0 zone resets
slat (usec): min=2, max=4509.8k, avg=327.40, stdev=8306.99
clat (usec): min=66, max=6000.1k, avg=4647.82, stdev=42006.68
lat (usec): min=69, max=6000.1k, avg=4975.78, stdev=43569.84
clat percentiles (usec):
| 1.00th=[ 668], 5.00th=[ 1336], 10.00th=[ 1729],
| 20.00th=[ 2212], 30.00th=[ 2606], 40.00th=[ 2966],
| 50.00th=[ 3359], 60.00th=[ 3818], 70.00th=[ 4359],
| 80.00th=[ 5145], 90.00th=[ 6521], 95.00th=[ 8586],
| 99.00th=[ 21890], 99.50th=[ 29230], 99.90th=[ 47973],
| 99.95th=[ 63177], 99.99th=[2701132]
bw ( KiB/s): min= 7, max= 9776, per=27.05%, avg=5560.95, stdev=2246.97, samples=1658
iops : min= 1, max= 2444, avg=1390.10, stdev=561.75, samples=1658
lat (usec) : 10=0.01%, 100=0.02%, 250=0.07%, 500=0.36%, 750=0.80%
lat (usec) : 1000=1.17%
lat (msec) : 2=11.68%, 4=49.07%, 10=33.14%, 20=2.45%, 50=1.14%
lat (msec) : 100=0.06%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%, >=2000=0.01%
cpu : usr=1.59%, sys=8.40%, ctx=2711432, majf=1, minf=23
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1730177,1153643,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=621476: Wed Apr 15 02:06:48 2020
read: IOPS=1925, BW=7701KiB/s (7886kB/s)(6769MiB/900001msec)
slat (nsec): min=1835, max=3014.6M, avg=293882.01, stdev=7975456.23
clat (usec): min=16, max=6081.5k, avg=4701.02, stdev=40545.51
lat (usec): min=65, max=6081.5k, avg=4995.47, stdev=41788.33
clat percentiles (usec):
| 1.00th=[ 676], 5.00th=[ 1385], 10.00th=[ 1795],
| 20.00th=[ 2278], 30.00th=[ 2671], 40.00th=[ 3032],
| 50.00th=[ 3425], 60.00th=[ 3851], 70.00th=[ 4424],
| 80.00th=[ 5145], 90.00th=[ 6521], 95.00th=[ 8586],
| 99.00th=[ 22676], 99.50th=[ 30278], 99.90th=[ 50070],
| 99.95th=[ 68682], 99.99th=[2868904]
bw ( KiB/s): min= 7, max=14626, per=26.97%, avg=8317.50, stdev=3430.16, samples=1665
iops : min= 1, max= 3656, avg=2079.23, stdev=857.57, samples=1665
write: IOPS=1284, BW=5137KiB/s (5260kB/s)(4515MiB/900001msec); 0 zone resets
slat (usec): min=2, max=5770.0k, avg=327.47, stdev=9706.72
clat (usec): min=62, max=6081.3k, avg=4635.87, stdev=41925.59
lat (usec): min=66, max=6081.6k, avg=4963.87, stdev=43686.92
clat percentiles (usec):
| 1.00th=[ 644], 5.00th=[ 1319], 10.00th=[ 1729],
| 20.00th=[ 2212], 30.00th=[ 2606], 40.00th=[ 2966],
| 50.00th=[ 3359], 60.00th=[ 3785], 70.00th=[ 4359],
| 80.00th=[ 5080], 90.00th=[ 6456], 95.00th=[ 8455],
| 99.00th=[ 21890], 99.50th=[ 29230], 99.90th=[ 47449],
| 99.95th=[ 63177], 99.99th=[2835350]
bw ( KiB/s): min= 8, max= 9720, per=27.12%, avg=5574.94, stdev=2265.45, samples=1657
iops : min= 2, max= 2430, avg=1393.59, stdev=566.38, samples=1657
lat (usec) : 20=0.01%, 100=0.02%, 250=0.07%, 500=0.38%, 750=0.86%
lat (usec) : 1000=1.17%
lat (msec) : 2=11.63%, 4=49.28%, 10=32.89%, 20=2.47%, 50=1.14%
lat (msec) : 100=0.06%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%, >=2000=0.01%
cpu : usr=1.60%, sys=8.41%, ctx=2726842, majf=0, minf=24
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1732741,1155858,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=621477: Wed Apr 15 02:06:48 2020
read: IOPS=1928, BW=7714KiB/s (7900kB/s)(6780MiB/900001msec)
slat (nsec): min=1882, max=4509.8M, avg=295620.68, stdev=8875316.44
clat (usec): min=2, max=6079.9k, avg=4726.13, stdev=43107.65
lat (usec): min=65, max=6079.9k, avg=5022.33, stdev=44581.37
clat percentiles (usec):
| 1.00th=[ 693], 5.00th=[ 1385], 10.00th=[ 1795],
| 20.00th=[ 2278], 30.00th=[ 2671], 40.00th=[ 3032],
| 50.00th=[ 3425], 60.00th=[ 3851], 70.00th=[ 4424],
| 80.00th=[ 5145], 90.00th=[ 6521], 95.00th=[ 8586],
| 99.00th=[ 22414], 99.50th=[ 29754], 99.90th=[ 47973],
| 99.95th=[ 64226], 99.99th=[2969568]
bw ( KiB/s): min= 7, max=14872, per=27.04%, avg=8341.89, stdev=3408.53, samples=1663
iops : min= 1, max= 3718, avg=2085.33, stdev=852.15, samples=1663
write: IOPS=1283, BW=5136KiB/s (5259kB/s)(4514MiB/900001msec); 0 zone resets
slat (usec): min=2, max=5611.0k, avg=324.11, stdev=8194.12
clat (usec): min=62, max=6063.8k, avg=4587.10, stdev=38392.47
lat (usec): min=66, max=6080.2k, avg=4911.76, stdev=39836.92
clat percentiles (usec):
| 1.00th=[ 668], 5.00th=[ 1336], 10.00th=[ 1729],
| 20.00th=[ 2212], 30.00th=[ 2606], 40.00th=[ 2966],
| 50.00th=[ 3359], 60.00th=[ 3785], 70.00th=[ 4359],
| 80.00th=[ 5080], 90.00th=[ 6456], 95.00th=[ 8455],
| 99.00th=[ 21890], 99.50th=[ 29230], 99.90th=[ 46400],
| 99.95th=[ 61080], 99.99th=[1870660]
bw ( KiB/s): min= 7, max= 9768, per=27.08%, avg=5566.76, stdev=2257.52, samples=1659
iops : min= 1, max= 2442, avg=1391.55, stdev=564.40, samples=1659
lat (usec) : 4=0.01%, 100=0.02%, 250=0.06%, 500=0.36%, 750=0.82%
lat (usec) : 1000=1.18%
lat (msec) : 2=11.73%, 4=49.18%, 10=32.94%, 20=2.47%, 50=1.15%
lat (msec) : 100=0.05%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%, >=2000=0.01%
cpu : usr=1.53%, sys=8.46%, ctx=2720607, majf=0, minf=24
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=1735751,1155553,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=30.1MiB/s (31.6MB/s), 7690KiB/s-7741KiB/s (7874kB/s-7926kB/s), io=26.5GiB (28.4GB), run=900001-900001msec
WRITE: bw=20.1MiB/s (21.1MB/s), 5127KiB/s-5159KiB/s (5250kB/s-5283kB/s), io=17.6GiB (18.9GB), run=900001-900001msec
Disk stats (read/write):
dm-1: ios=3685533/4166016, merge=0/0, ticks=1286849/119447523, in_queue=120737260, util=99.40%, aggrios=3685677/4166016, aggrmerge=0/0, aggrticks=1277080/119441923, aggrin_queue=120721887, aggrutil=99.40%
dm-0: ios=3685677/4166016, merge=0/0, ticks=1277080/119441923, in_queue=120721887, util=99.40%, aggrios=3685677/4160531, aggrmerge=0/5487, aggrticks=797504/1490474, aggrin_queue=73134, aggrutil=99.38%
sda: ios=3685677/4160531, merge=0/5487, ticks=797504/1490474, in_queue=73134, util=99.38%
machine 3:
storage:
WDC WDS500G2B0A-00SM50 (ssd)
2 x WDC WD10EZRX-00L4HB0
storage layout:
ssd: cryptsetup luks with aes-xts-plain64 + lvm
hdd array: 2x hdd mdadm raid0 + cryptsetup luks with camellia-xts-plain64 + lvm
raw linear read io performance with hdparm -Tt (to understand bus limitations):
ssd:
Timing cached reads: 8664 MB in 2.00 seconds = 4335.58 MB/sec
Timing buffered disk reads: 1248 MB in 3.00 seconds = 415.96 MB/sec
hdd array:
Timing cached reads: 8692 MB in 2.00 seconds = 4349.16 MB/sec
Timing buffered disk reads: 892 MB in 3.00 seconds = 297.28 MB/sec
crypto performance:
[h1]Tests are approximate using memory only (no storage IO).[/h1]
[h1]Algorithm | Key | Encryption | Decryption[/h1]
twofish-xts 512b 219.6 MiB/s 225.9 MiB/s
[h1]Tests are approximate using memory only (no storage IO).[/h1]
[h1]Algorithm | Key | Encryption | Decryption[/h1]
serpent-xts 512b 226.4 MiB/s 233.5 MiB/s
[h1]Tests are approximate using memory only (no storage IO).[/h1]
[h1]Algorithm | Key | Encryption | Decryption[/h1]
aes-xts 512b 1198.2 MiB/s 1102.4 MiB/s
[h1]Tests are approximate using memory only (no storage IO).[/h1]
[h1]Algorithm | Key | Encryption | Decryption[/h1]
camellia-xts 512b 259.6 MiB/s 267.7 MiB/s
[h1]Tests are approximate using memory only (no storage IO).[/h1]
PBKDF2-ripemd160 168473 iterations per second for 512-bit key
[h1]Tests are approximate using memory only (no storage IO).[/h1]
PBKDF2-sha256 230355 iterations per second for 512-bit key
[h1]Tests are approximate using memory only (no storage IO).[/h1]
PBKDF2-sha512 392431 iterations per second for 512-bit key
[h1]Tests are approximate using memory only (no storage IO).[/h1]
PBKDF2-whirlpool 215934 iterations per second for 512-bit key
xfs filesystem tests:
fio-rand-RW.fio:
ssd:
file1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.18
Starting 4 processes
file1: Laying out IO file (1 file / 10240MiB)
Jobs: 4 (f=4): [m(4)][100.0%][eta 00m:00s]
file1: (groupid=0, jobs=1): err= 0: pid=115788: Wed Apr 15 02:42:20 2020
read: IOPS=4398, BW=17.2MiB/s (18.0MB/s)(15.2GiB/904453msec)
slat (nsec): min=1713, max=19778M, avg=106136.70, stdev=11322677.73
clat (usec): min=52, max=27685k, avg=2068.36, stdev=71398.30
lat (usec): min=54, max=27685k, avg=2174.71, stdev=72597.41
clat percentiles (usec):
| 1.00th=[ 83], 5.00th=[ 100], 10.00th=[ 110], 20.00th=[ 165],
| 30.00th=[ 469], 40.00th=[ 775], 50.00th=[ 1090], 60.00th=[ 1467],
| 70.00th=[ 1909], 80.00th=[ 2474], 90.00th=[ 3392], 95.00th=[ 4359],
| 99.00th=[ 7570], 99.50th=[ 10683], 99.90th=[ 29230], 99.95th=[ 41157],
| 99.99th=[187696]
bw ( KiB/s): min= 8, max=340064, per=31.70%, avg=22272.22, stdev=33531.22, samples=1428
iops : min= 2, max=85016, avg=5567.98, stdev=8382.81, samples=1428
write: IOPS=2932, BW=11.5MiB/s (12.0MB/s)(10.1GiB/904453msec); 0 zone resets
slat (usec): min=2, max=12792k, avg=169.85, stdev=19676.64
clat (usec): min=4, max=27685k, avg=2015.24, stdev=70796.95
lat (usec): min=55, max=27685k, avg=2185.32, stdev=74041.06
clat percentiles (usec):
| 1.00th=[ 82], 5.00th=[ 99], 10.00th=[ 110], 20.00th=[ 159],
| 30.00th=[ 449], 40.00th=[ 734], 50.00th=[ 1045], 60.00th=[ 1401],
| 70.00th=[ 1844], 80.00th=[ 2409], 90.00th=[ 3326], 95.00th=[ 4293],
| 99.00th=[ 7439], 99.50th=[ 10290], 99.90th=[ 27657], 99.95th=[ 39584],
| 99.99th=[160433]
bw ( KiB/s): min= 8, max=228088, per=31.78%, avg=14883.21, stdev=22374.69, samples=1425
iops : min= 2, max=57022, avg=3720.72, stdev=5593.66, samples=1425
lat (usec) : 10=0.01%, 100=5.29%, 250=18.63%, 500=7.73%, 750=8.09%
lat (usec) : 1000=7.88%
lat (msec) : 2=24.60%, 4=21.55%, 10=5.69%, 20=0.34%, 50=0.17%
lat (msec) : 100=0.02%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%, >=2000=0.01%
cpu : usr=2.18%, sys=13.19%, ctx=3611418, majf=0, minf=23
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3977929,2652645,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=115789: Wed Apr 15 02:42:20 2020
read: IOPS=4387, BW=17.1MiB/s (17.0MB/s)(15.1GiB/904453msec)
slat (nsec): min=1492, max=9319.9M, avg=114899.83, stdev=10620345.06
clat (usec): min=9, max=27835k, avg=2066.00, stdev=71693.91
lat (usec): min=56, max=27835k, avg=2181.12, stdev=72886.66
clat percentiles (usec):
| 1.00th=[ 83], 5.00th=[ 100], 10.00th=[ 110], 20.00th=[ 163],
| 30.00th=[ 465], 40.00th=[ 766], 50.00th=[ 1090], 60.00th=[ 1467],
| 70.00th=[ 1926], 80.00th=[ 2507], 90.00th=[ 3425], 95.00th=[ 4359],
| 99.00th=[ 7570], 99.50th=[ 10683], 99.90th=[ 29230], 99.95th=[ 40109],
| 99.99th=[147850]
bw ( KiB/s): min= 8, max=347336, per=31.64%, avg=22231.52, stdev=33886.02, samples=1427
iops : min= 2, max=86834, avg=5557.80, stdev=8471.50, samples=1427
write: IOPS=2925, BW=11.4MiB/s (11.0MB/s)(10.1GiB/904453msec); 0 zone resets
slat (usec): min=2, max=19745k, avg=157.48, stdev=20211.37
clat (usec): min=54, max=27835k, avg=2031.82, stdev=73292.55
lat (usec): min=57, max=27835k, avg=2189.53, stdev=76492.99
clat percentiles (usec):
| 1.00th=[ 83], 5.00th=[ 99], 10.00th=[ 109], 20.00th=[ 159],
| 30.00th=[ 441], 40.00th=[ 734], 50.00th=[ 1045], 60.00th=[ 1418],
| 70.00th=[ 1860], 80.00th=[ 2442], 90.00th=[ 3359], 95.00th=[ 4293],
| 99.00th=[ 7439], 99.50th=[ 10290], 99.90th=[ 28181], 99.95th=[ 39060],
| 99.99th=[137364]
bw ( KiB/s): min= 8, max=231312, per=31.85%, avg=14918.22, stdev=22628.46, samples=1418
iops : min= 2, max=57828, avg=3729.47, stdev=5657.11, samples=1418
lat (usec) : 10=0.01%, 100=5.33%, 250=18.74%, 500=7.82%, 750=8.01%
lat (usec) : 1000=7.79%
lat (msec) : 2=24.35%, 4=21.61%, 10=5.80%, 20=0.35%, 50=0.17%
lat (msec) : 100=0.02%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%, >=2000=0.01%
cpu : usr=2.16%, sys=13.21%, ctx=3592523, majf=0, minf=23
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3967874,2645804,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=115790: Wed Apr 15 02:42:20 2020
read: IOPS=4393, BW=17.2MiB/s (17.0MB/s)(15.2GiB/904453msec)
slat (nsec): min=1681, max=12792M, avg=116265.18, stdev=13332344.36
clat (usec): min=53, max=28423k, avg=2089.26, stdev=73339.30
lat (usec): min=56, max=28423k, avg=2205.74, stdev=74861.83
clat percentiles (usec):
| 1.00th=[ 87], 5.00th=[ 100], 10.00th=[ 110], 20.00th=[ 172],
| 30.00th=[ 469], 40.00th=[ 775], 50.00th=[ 1090], 60.00th=[ 1467],
| 70.00th=[ 1909], 80.00th=[ 2474], 90.00th=[ 3392], 95.00th=[ 4359],
| 99.00th=[ 7570], 99.50th=[ 10552], 99.90th=[ 28967], 99.95th=[ 39584],
| 99.99th=[198181]
bw ( KiB/s): min= 8, max=342864, per=31.75%, avg=22311.96, stdev=33541.39, samples=1424
iops : min= 2, max=85716, avg=5577.91, stdev=8385.34, samples=1424
write: IOPS=2928, BW=11.4MiB/s (11.0MB/s)(10.1GiB/904453msec); 0 zone resets
slat (usec): min=2, max=19745k, avg=154.84, stdev=17365.04
clat (usec): min=10, max=28423k, avg=1990.12, stdev=69995.95
lat (usec): min=56, max=28423k, avg=2145.20, stdev=72725.24
clat percentiles (usec):
| 1.00th=[ 86], 5.00th=[ 100], 10.00th=[ 110], 20.00th=[ 163],
| 30.00th=[ 445], 40.00th=[ 734], 50.00th=[ 1045], 60.00th=[ 1401],
| 70.00th=[ 1844], 80.00th=[ 2409], 90.00th=[ 3326], 95.00th=[ 4293],
| 99.00th=[ 7439], 99.50th=[ 10290], 99.90th=[ 28181], 99.95th=[ 39060],
| 99.99th=[141558]
bw ( KiB/s): min= 8, max=229368, per=31.84%, avg=14913.15, stdev=22414.86, samples=1420
iops : min= 2, max=57342, avg=3728.20, stdev=5603.70, samples=1420
lat (usec) : 20=0.01%, 100=5.01%, 250=18.72%, 500=7.96%, 750=8.05%
lat (usec) : 1000=7.88%
lat (msec) : 2=24.63%, 4=21.49%, 10=5.71%, 20=0.35%, 50=0.17%
lat (msec) : 100=0.02%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%, >=2000=0.01%
cpu : usr=2.15%, sys=13.25%, ctx=3649760, majf=0, minf=23
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3973815,2648617,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=115791: Wed Apr 15 02:42:20 2020
read: IOPS=4387, BW=17.1MiB/s (17.0MB/s)(15.1GiB/904454msec)
slat (nsec): min=1743, max=19778M, avg=121148.63, stdev=16032951.75
clat (usec): min=53, max=27687k, avg=2051.97, stdev=66316.94
lat (usec): min=56, max=27687k, avg=2173.34, stdev=68621.44
clat percentiles (usec):
| 1.00th=[ 80], 5.00th=[ 99], 10.00th=[ 109], 20.00th=[ 165],
| 30.00th=[ 469], 40.00th=[ 775], 50.00th=[ 1106], 60.00th=[ 1483],
| 70.00th=[ 1926], 80.00th=[ 2507], 90.00th=[ 3425], 95.00th=[ 4359],
| 99.00th=[ 7570], 99.50th=[ 10683], 99.90th=[ 28443], 99.95th=[ 39584],
| 99.99th=[198181]
bw ( KiB/s): min= 8, max=349384, per=31.64%, avg=22234.08, stdev=33880.89, samples=1427
iops : min= 2, max=87346, avg=5558.44, stdev=8470.21, samples=1427
write: IOPS=2921, BW=11.4MiB/s (11.0MB/s)(10.1GiB/904454msec); 0 zone resets
slat (usec): min=2, max=12792k, avg=148.36, stdev=13818.41
clat (usec): min=10, max=27686k, avg=2055.01, stdev=77333.17
lat (usec): min=56, max=27686k, avg=2203.60, stdev=78954.75
clat percentiles (usec):
| 1.00th=[ 79], 5.00th=[ 99], 10.00th=[ 109], 20.00th=[ 159],
| 30.00th=[ 449], 40.00th=[ 742], 50.00th=[ 1057], 60.00th=[ 1418],
| 70.00th=[ 1860], 80.00th=[ 2442], 90.00th=[ 3359], 95.00th=[ 4293],
| 99.00th=[ 7439], 99.50th=[ 10290], 99.90th=[ 27919], 99.95th=[ 39060],
| 99.99th=[198181]
bw ( KiB/s): min= 8, max=232600, per=31.68%, avg=14837.29, stdev=22596.83, samples=1424
iops : min= 2, max=58150, avg=3709.24, stdev=5649.20, samples=1424
lat (usec) : 20=0.01%, 100=5.68%, 250=18.35%, 500=7.66%, 750=7.96%
lat (usec) : 1000=7.82%
lat (msec) : 2=24.51%, 4=21.71%, 10=5.78%, 20=0.35%, 50=0.16%
lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%, >=2000=0.01%
cpu : usr=2.18%, sys=13.17%, ctx=3600493, majf=0, minf=23
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3968340,2642604,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=68.6MiB/s (71.0MB/s), 17.1MiB/s-17.2MiB/s (17.0MB/s-18.0MB/s), io=60.6GiB (65.1GB), run=904453-904454msec
WRITE: bw=45.7MiB/s (47.0MB/s), 11.4MiB/s-11.5MiB/s (11.0MB/s-12.0MB/s), io=40.4GiB (43.4GB), run=904453-904454msec
Disk stats (read/write):
dm-2: ios=3315692/8009253, merge=0/0, ticks=1040390/18446744071668371254, in_queue=18446744071669412014, util=98.70%, aggrios=3315692/8009253, aggrmerge=0/0, aggrticks=1036954/18446744071689086446, aggrin_queue=18446744071690123769, aggrutil=98.70%
dm-1: ios=3315692/8009253, merge=0/0, ticks=1036954/18446744071689086446, in_queue=18446744071690123769, util=98.70%, aggrios=3315692/7323541, aggrmerge=0/447753, aggrticks=464244/1102433, aggrin_queue=46733, aggrutil=98.65%
sdc: ios=3315692/7323541, merge=0/447753, ticks=464244/1102433, in_queue=46733, util=98.65%
hdd array:
file1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.18
Starting 4 processes
file1: Laying out IO file (1 file / 10240MiB)
Jobs: 3 (f=3): [m(1),_(1),m(2)][50.1%][r=47KiB/s,w=15KiB/s][r=11,w=3 IOPS][eta 14m:59s]
file1: (groupid=0, jobs=1): err= 0: pid=116099: Wed Apr 15 03:01:43 2020
read: IOPS=35, BW=142KiB/s (145kB/s)(125MiB/901171msec)
slat (usec): min=4, max=2393.1k, avg=19205.65, stdev=52749.56
clat (msec): min=68, max=3719, avg=255.72, stdev=245.40
lat (msec): min=78, max=3735, avg=274.93, stdev=255.14
clat percentiles (msec):
| 1.00th=[ 131], 5.00th=[ 159], 10.00th=[ 174], 20.00th=[ 192],
| 30.00th=[ 205], 40.00th=[ 218], 50.00th=[ 230], 60.00th=[ 243],
| 70.00th=[ 255], 80.00th=[ 271], 90.00th=[ 300], 95.00th=[ 326],
| 99.00th=[ 651], 99.50th=[ 2836], 99.90th=[ 3205], 99.95th=[ 3473],
| 99.99th=[ 3708]
bw ( KiB/s): min= 7, max= 247, per=26.52%, avg=149.84, stdev=39.31, samples=1694
iops : min= 1, max= 61, avg=36.78, stdev= 9.82, samples=1694
write: IOPS=23, BW=94.3KiB/s (96.5kB/s)(82.0MiB/901171msec); 0 zone resets
slat (usec): min=3, max=2581.7k, avg=13522.18, stdev=44734.86
clat (usec): min=11, max=3695.6k, avg=251779.96, stdev=245770.99
lat (msec): min=71, max=3708, avg=265.30, stdev=252.69
clat percentiles (msec):
| 1.00th=[ 128], 5.00th=[ 155], 10.00th=[ 169], 20.00th=[ 188],
| 30.00th=[ 201], 40.00th=[ 213], 50.00th=[ 226], 60.00th=[ 239],
| 70.00th=[ 251], 80.00th=[ 268], 90.00th=[ 296], 95.00th=[ 326],
| 99.00th=[ 651], 99.50th=[ 2836], 99.90th=[ 3239], 99.95th=[ 3540],
| 99.99th=[ 3641]
bw ( KiB/s): min= 7, max= 215, per=26.32%, avg=99.49, stdev=33.86, samples=1692
iops : min= 1, max= 53, avg=24.20, stdev= 8.47, samples=1692
lat (usec) : 20=0.01%
lat (msec) : 100=0.12%, 250=67.49%, 500=30.99%, 750=0.45%, 1000=0.03%
lat (msec) : 2000=0.18%, >=2000=0.72%
cpu : usr=0.09%, sys=0.87%, ctx=68032, majf=0, minf=18
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=31933,21241,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=116100: Wed Apr 15 03:01:43 2020
read: IOPS=35, BW=141KiB/s (145kB/s)(124MiB/900753msec)
slat (usec): min=3, max=2189.7k, avg=19250.42, stdev=50318.66
clat (usec): min=12, max=3576.5k, avg=256838.86, stdev=250357.23
lat (msec): min=76, max=3637, avg=276.09, stdev=259.82
clat percentiles (msec):
| 1.00th=[ 132], 5.00th=[ 159], 10.00th=[ 174], 20.00th=[ 192],
| 30.00th=[ 205], 40.00th=[ 218], 50.00th=[ 228], 60.00th=[ 241],
| 70.00th=[ 255], 80.00th=[ 271], 90.00th=[ 300], 95.00th=[ 330],
| 99.00th=[ 793], 99.50th=[ 2869], 99.90th=[ 3205], 99.95th=[ 3473],
| 99.99th=[ 3540]
bw ( KiB/s): min= 7, max= 255, per=26.30%, avg=148.62, stdev=39.97, samples=1700
iops : min= 1, max= 63, avg=36.48, stdev= 9.99, samples=1700
write: IOPS=23, BW=94.3KiB/s (96.5kB/s)(82.9MiB/900753msec); 0 zone resets
slat (usec): min=3, max=2186.3k, avg=13572.91, stdev=44869.81
clat (msec): min=71, max=3604, avg=251.70, stdev=233.77
lat (msec): min=78, max=3604, avg=265.28, stdev=241.19
clat percentiles (msec):
| 1.00th=[ 129], 5.00th=[ 157], 10.00th=[ 171], 20.00th=[ 190],
| 30.00th=[ 203], 40.00th=[ 215], 50.00th=[ 226], 60.00th=[ 239],
| 70.00th=[ 253], 80.00th=[ 271], 90.00th=[ 296], 95.00th=[ 326],
| 99.00th=[ 625], 99.50th=[ 2735], 99.90th=[ 3171], 99.95th=[ 3440],
| 99.99th=[ 3540]
bw ( KiB/s): min= 7, max= 215, per=26.22%, avg=99.10, stdev=33.88, samples=1698
iops : min= 1, max= 53, avg=24.10, stdev= 8.47, samples=1698
lat (usec) : 20=0.01%
lat (msec) : 100=0.08%, 250=67.58%, 500=30.91%, 750=0.48%, 1000=0.04%
lat (msec) : 2000=0.16%, >=2000=0.76%
cpu : usr=0.09%, sys=0.90%, ctx=67957, majf=0, minf=20
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=31788,21231,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=116101: Wed Apr 15 03:01:43 2020
read: IOPS=35, BW=142KiB/s (145kB/s)(125MiB/901171msec)
slat (usec): min=4, max=2093.4k, avg=18729.42, stdev=40433.47
clat (msec): min=70, max=3583, avg=255.23, stdev=242.69
lat (msec): min=85, max=3596, avg=273.96, stdev=250.06
clat percentiles (msec):
| 1.00th=[ 130], 5.00th=[ 157], 10.00th=[ 171], 20.00th=[ 190],
| 30.00th=[ 205], 40.00th=[ 218], 50.00th=[ 230], 60.00th=[ 243],
| 70.00th=[ 255], 80.00th=[ 271], 90.00th=[ 300], 95.00th=[ 330],
| 99.00th=[ 642], 99.50th=[ 2802], 99.90th=[ 3239], 99.95th=[ 3473],
| 99.99th=[ 3540]
bw ( KiB/s): min= 7, max= 271, per=26.38%, avg=149.04, stdev=40.51, samples=1702
iops : min= 1, max= 67, avg=36.58, stdev=10.12, samples=1702
write: IOPS=23, BW=94.3KiB/s (96.6kB/s)(83.0MiB/901171msec); 0 zone resets
slat (usec): min=4, max=2162.2k, avg=14248.26, stdev=57466.27
clat (usec): min=11, max=3581.3k, avg=252656.28, stdev=247111.54
lat (msec): min=82, max=3602, avg=266.91, stdev=257.81
clat percentiles (msec):
| 1.00th=[ 127], 5.00th=[ 153], 10.00th=[ 169], 20.00th=[ 188],
| 30.00th=[ 201], 40.00th=[ 215], 50.00th=[ 226], 60.00th=[ 239],
| 70.00th=[ 253], 80.00th=[ 271], 90.00th=[ 296], 95.00th=[ 321],
| 99.00th=[ 701], 99.50th=[ 2836], 99.90th=[ 3205], 99.95th=[ 3339],
| 99.99th=[ 3540]
bw ( KiB/s): min= 7, max= 223, per=26.29%, avg=99.37, stdev=34.59, samples=1695
iops : min= 1, max= 55, avg=24.16, stdev= 8.64, samples=1695
lat (usec) : 20=0.01%
lat (msec) : 100=0.11%, 250=67.39%, 500=30.99%, 750=0.56%, 1000=0.05%
lat (msec) : 2000=0.17%, >=2000=0.73%
cpu : usr=0.09%, sys=0.89%, ctx=68014, majf=0, minf=20
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=31912,21253,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=116102: Wed Apr 15 03:01:43 2020
read: IOPS=35, BW=141KiB/s (144kB/s)(124MiB/901172msec)
slat (usec): min=2, max=2894.9k, avg=19180.06, stdev=54568.73
clat (usec): min=11, max=3696.7k, avg=255785.90, stdev=247754.36
lat (msec): min=71, max=3709, avg=274.97, stdev=257.15
clat percentiles (msec):
| 1.00th=[ 133], 5.00th=[ 159], 10.00th=[ 174], 20.00th=[ 192],
| 30.00th=[ 205], 40.00th=[ 218], 50.00th=[ 228], 60.00th=[ 241],
| 70.00th=[ 255], 80.00th=[ 271], 90.00th=[ 300], 95.00th=[ 330],
| 99.00th=[ 617], 99.50th=[ 2836], 99.90th=[ 3205], 99.95th=[ 3473],
| 99.99th=[ 3641]
bw ( KiB/s): min= 7, max= 279, per=26.23%, avg=148.18, stdev=41.04, samples=1702
iops : min= 1, max= 69, avg=36.37, stdev=10.25, samples=1702
write: IOPS=23, BW=95.4KiB/s (97.7kB/s)(83.9MiB/901172msec); 0 zone resets
slat (usec): min=3, max=2066.5k, avg=13586.55, stdev=41539.32
clat (msec): min=58, max=3694, avg=251.30, stdev=240.94
lat (msec): min=63, max=3725, avg=264.88, stdev=248.29
clat percentiles (msec):
| 1.00th=[ 129], 5.00th=[ 155], 10.00th=[ 169], 20.00th=[ 188],
| 30.00th=[ 201], 40.00th=[ 213], 50.00th=[ 226], 60.00th=[ 239],
| 70.00th=[ 251], 80.00th=[ 268], 90.00th=[ 296], 95.00th=[ 326],
| 99.00th=[ 667], 99.50th=[ 2769], 99.90th=[ 3205], 99.95th=[ 3440],
| 99.99th=[ 3641]
bw ( KiB/s): min= 7, max= 223, per=26.61%, avg=100.59, stdev=34.36, samples=1693
iops : min= 1, max= 55, avg=24.47, stdev= 8.59, samples=1693
lat (usec) : 20=0.01%
lat (msec) : 100=0.09%, 250=67.94%, 500=30.62%, 750=0.40%, 1000=0.05%
lat (msec) : 2000=0.16%, >=2000=0.73%
cpu : usr=0.09%, sys=0.89%, ctx=67816, majf=0, minf=19
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=31730,21487,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=565KiB/s (579kB/s), 141KiB/s-142KiB/s (144kB/s-145kB/s), io=498MiB (522MB), run=900753-901172msec
WRITE: bw=378KiB/s (387kB/s), 94.3KiB/s-95.4KiB/s (96.5kB/s-97.7kB/s), io=333MiB (349MB), run=900753-901172msec
Disk stats (read/write):
dm-3: ios=123568/85178, merge=0/0, ticks=1814445/107073064, in_queue=108887509, util=20.35%, aggrios=123568/85178, aggrmerge=0/0, aggrticks=1815844/107585903, aggrin_queue=109401747, aggrutil=20.41%
dm-0: ios=123568/85178, merge=0/0, ticks=1815844/107585903, in_queue=109401747, util=20.41%, aggrios=123574/84872, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
md127: ios=123574/84872, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=61787/42433, aggrmerge=0/0, aggrticks=902035/1902456, aggrin_queue=2752314, aggrutil=14.40%
sdb: ios=61589/42295, merge=0/0, ticks=896372/1921815, in_queue=2766186, util=14.25%
sda: ios=61985/42572, merge=0/0, ticks=907698/1883097, in_queue=2738442, util=14.40%
machine 4:
NOTE: tests on this machine done under some light io load which cannot be avoided, so they are not very accurate
storage:
SanDisk SDSSDH3 500G (ssd)
storage layout:
ssd: cryptsetup luks with aes-xts-plain64 + lvm
raw linear read io performance with hdparm -Tt (to understand bus limitations):
ssd:
Timing cached reads: 4814 MB in 2.00 seconds = 2408.90 MB/sec
Timing buffered disk reads: 1354 MB in 3.00 seconds = 451.18 MB/sec
crypto performance:
[h1]Tests are approximate using memory only (no storage IO).[/h1]
[h1]Algorithm | Key | Encryption | Decryption[/h1]
twofish-xts 512b 125.8 MiB/s 129.4 MiB/s
[h1]Tests are approximate using memory only (no storage IO).[/h1]
[h1]Algorithm | Key | Encryption | Decryption[/h1]
serpent-xts 512b 168.2 MiB/s 171.9 MiB/s
[h1]Tests are approximate using memory only (no storage IO).[/h1]
[h1]Algorithm | Key | Encryption | Decryption[/h1]
aes-xts 512b 738.9 MiB/s 721.8 MiB/s
[h1]Tests are approximate using memory only (no storage IO).[/h1]
[h1]Algorithm | Key | Encryption | Decryption[/h1]
camellia-xts 512b 183.5 MiB/s 184.7 MiB/s
[h1]Tests are approximate using memory only (no storage IO).[/h1]
PBKDF2-ripemd160 70167 iterations per second for 512-bit key
[h1]Tests are approximate using memory only (no storage IO).[/h1]
PBKDF2-sha256 110329 iterations per second for 512-bit key
[h1]Tests are approximate using memory only (no storage IO).[/h1]
PBKDF2-sha512 188321 iterations per second for 512-bit key
[h1]Tests are approximate using memory only (no storage IO).[/h1]
PBKDF2-whirlpool 107789 iterations per second for 512-bit key
xfs filesystem tests:
fio-rand-RW.fio:
ssd array:
file1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.18
Starting 4 processes
file1: Laying out IO file (1 file / 10240MiB)
Jobs: 4 (f=4): [m(4)][100.0%][r=104MiB/s,w=69.6MiB/s][r=26.7k,w=17.8k IOPS][eta 00m:00s]
file1: (groupid=0, jobs=1): err= 0: pid=864500: Wed Apr 15 02:04:11 2020
read: IOPS=4388, BW=17.1MiB/s (17.0MB/s)(15.1GiB/900001msec)
slat (usec): min=3, max=2554.5k, avg=119.35, stdev=1869.84
clat (usec): min=4, max=16401k, avg=2058.41, stdev=31678.60
lat (usec): min=14, max=16401k, avg=2178.06, stdev=31763.13
clat percentiles (usec):
| 1.00th=[ 186], 5.00th=[ 221], 10.00th=[ 249], 20.00th=[ 363],
| 30.00th=[ 644], 40.00th=[ 988], 50.00th=[ 1352], 60.00th=[ 1713],
| 70.00th=[ 2114], 80.00th=[ 2671], 90.00th=[ 3621], 95.00th=[ 5080],
| 99.00th=[ 12649], 99.50th=[ 20579], 99.90th=[ 54264], 99.95th=[ 83362],
| 99.99th=[154141]
bw ( KiB/s): min= 7, max=148095, per=26.09%, avg=18286.59, stdev=19853.55, samples=1723
iops : min= 1, max=37023, avg=4571.29, stdev=4963.39, samples=1723
write: IOPS=2926, BW=11.4MiB/s (11.0MB/s)(10.0GiB/900001msec); 0 zone resets
slat (usec): min=4, max=1758.8k, avg=141.70, stdev=1462.20
clat (usec): min=18, max=16401k, avg=2042.26, stdev=39682.13
lat (usec): min=49, max=16401k, avg=2184.29, stdev=39771.47
clat percentiles (usec):
| 1.00th=[ 186], 5.00th=[ 221], 10.00th=[ 247], 20.00th=[ 351],
| 30.00th=[ 619], 40.00th=[ 947], 50.00th=[ 1287], 60.00th=[ 1663],
| 70.00th=[ 2073], 80.00th=[ 2606], 90.00th=[ 3556], 95.00th=[ 4883],
| 99.00th=[ 12125], 99.50th=[ 20055], 99.90th=[ 52691], 99.95th=[ 80217],
| 99.99th=[149947]
bw ( KiB/s): min= 7, max=98794, per=26.11%, avg=12200.11, stdev=13247.13, samples=1722
iops : min= 1, max=24698, avg=3049.65, stdev=3311.77, samples=1722
lat (usec) : 10=0.01%, 20=0.01%, 100=0.01%, 250=10.22%, 500=15.13%
lat (usec) : 750=8.36%, 1000=7.15%
lat (msec) : 2=26.74%, 4=24.52%, 10=6.51%, 20=0.85%, 50=0.40%
lat (msec) : 100=0.08%, 250=0.03%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%, >=2000=0.01%
cpu : usr=2.96%, sys=19.35%, ctx=4143437, majf=0, minf=24
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3949966,2633725,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=864501: Wed Apr 15 02:04:11 2020
read: IOPS=4371, BW=17.1MiB/s (17.9MB/s)(15.0GiB/900001msec)
slat (usec): min=3, max=447445, avg=117.01, stdev=754.98
clat (usec): min=78, max=15782k, avg=2093.16, stdev=36392.78
lat (usec): min=112, max=15782k, avg=2210.48, stdev=36425.14
clat percentiles (usec):
| 1.00th=[ 159], 5.00th=[ 212], 10.00th=[ 241], 20.00th=[ 338],
| 30.00th=[ 619], 40.00th=[ 971], 50.00th=[ 1352], 60.00th=[ 1729],
| 70.00th=[ 2147], 80.00th=[ 2704], 90.00th=[ 3687], 95.00th=[ 5145],
| 99.00th=[ 12780], 99.50th=[ 20841], 99.90th=[ 55313], 99.95th=[ 82314],
| 99.99th=[160433]
bw ( KiB/s): min= 7, max=187361, per=25.97%, avg=18202.14, stdev=20885.08, samples=1724
iops : min= 1, max=46840, avg=4550.17, stdev=5221.28, samples=1724
write: IOPS=2914, BW=11.4MiB/s (11.9MB/s)(10.0GiB/900001msec); 0 zone resets
slat (usec): min=4, max=2554.6k, avg=146.89, stdev=2574.23
clat (usec): min=3, max=15791k, avg=2010.83, stdev=29391.42
lat (usec): min=18, max=15791k, avg=2158.06, stdev=29591.43
clat percentiles (usec):
| 1.00th=[ 159], 5.00th=[ 212], 10.00th=[ 241], 20.00th=[ 330],
| 30.00th=[ 594], 40.00th=[ 930], 50.00th=[ 1303], 60.00th=[ 1680],
| 70.00th=[ 2089], 80.00th=[ 2638], 90.00th=[ 3589], 95.00th=[ 5014],
| 99.00th=[ 12256], 99.50th=[ 20055], 99.90th=[ 52691], 99.95th=[ 80217],
| 99.99th=[154141]
bw ( KiB/s): min= 7, max=123968, per=25.98%, avg=12137.94, stdev=13935.60, samples=1724
iops : min= 1, max=30992, avg=3034.11, stdev=3483.90, samples=1724
lat (usec) : 4=0.01%, 50=0.01%, 100=0.01%, 250=11.59%, 500=14.77%
lat (usec) : 750=8.02%, 1000=6.88%
lat (msec) : 2=25.91%, 4=24.69%, 10=6.75%, 20=0.88%, 50=0.41%
lat (msec) : 100=0.08%, 250=0.03%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%, >=2000=0.01%
cpu : usr=2.99%, sys=19.20%, ctx=4148391, majf=0, minf=24
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3933948,2623346,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=864502: Wed Apr 15 02:04:11 2020
read: IOPS=4379, BW=17.1MiB/s (17.9MB/s)(15.0GiB/900001msec)
slat (usec): min=3, max=2206.4k, avg=118.59, stdev=1655.52
clat (usec): min=5, max=16019k, avg=2076.84, stdev=34200.33
lat (usec): min=111, max=16019k, avg=2195.74, stdev=34296.39
clat percentiles (usec):
| 1.00th=[ 182], 5.00th=[ 219], 10.00th=[ 247], 20.00th=[ 351],
| 30.00th=[ 635], 40.00th=[ 979], 50.00th=[ 1352], 60.00th=[ 1729],
| 70.00th=[ 2147], 80.00th=[ 2671], 90.00th=[ 3654], 95.00th=[ 5080],
| 99.00th=[ 12649], 99.50th=[ 20841], 99.90th=[ 55313], 99.95th=[ 83362],
| 99.99th=[162530]
bw ( KiB/s): min= 15, max=172047, per=26.02%, avg=18238.54, stdev=20148.48, samples=1724
iops : min= 3, max=43011, avg=4559.27, stdev=5037.11, samples=1724
write: IOPS=2918, BW=11.4MiB/s (11.0MB/s)(10.0GiB/900001msec); 0 zone resets
slat (usec): min=4, max=2554.6k, avg=143.84, stdev=1820.61
clat (usec): min=105, max=16010k, avg=2026.25, stdev=34670.41
lat (usec): min=111, max=16010k, avg=2170.42, stdev=34749.00
clat percentiles (usec):
| 1.00th=[ 182], 5.00th=[ 217], 10.00th=[ 245], 20.00th=[ 343],
| 30.00th=[ 603], 40.00th=[ 938], 50.00th=[ 1303], 60.00th=[ 1663],
| 70.00th=[ 2089], 80.00th=[ 2606], 90.00th=[ 3556], 95.00th=[ 4948],
| 99.00th=[ 12256], 99.50th=[ 20055], 99.90th=[ 52691], 99.95th=[ 80217],
| 99.99th=[160433]
bw ( KiB/s): min= 7, max=116630, per=26.00%, avg=12147.34, stdev=13451.00, samples=1725
iops : min= 1, max=29157, avg=3036.47, stdev=3362.75, samples=1725
lat (usec) : 10=0.01%, 250=10.65%, 500=15.18%, 750=8.20%, 1000=7.01%
lat (msec) : 2=26.33%, 4=24.64%, 10=6.61%, 20=0.87%, 50=0.40%
lat (msec) : 100=0.08%, 250=0.03%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%, >=2000=0.01%
cpu : usr=2.89%, sys=19.29%, ctx=4154263, majf=0, minf=24
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3941858,2626995,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
file1: (groupid=0, jobs=1): err= 0: pid=864503: Wed Apr 15 02:04:11 2020
read: IOPS=4385, BW=17.1MiB/s (17.0MB/s)(15.1GiB/900001msec)
slat (usec): min=3, max=700760, avg=118.74, stdev=889.52
clat (usec): min=115, max=16070k, avg=2068.03, stdev=32558.52
lat (usec): min=119, max=16070k, avg=2187.07, stdev=32600.79
clat percentiles (usec):
| 1.00th=[ 186], 5.00th=[ 219], 10.00th=[ 247], 20.00th=[ 359],
| 30.00th=[ 635], 40.00th=[ 979], 50.00th=[ 1352], 60.00th=[ 1713],
| 70.00th=[ 2147], 80.00th=[ 2671], 90.00th=[ 3654], 95.00th=[ 5080],
| 99.00th=[ 12780], 99.50th=[ 20579], 99.90th=[ 54789], 99.95th=[ 81265],
| 99.99th=[156238]
bw ( KiB/s): min= 7, max=152247, per=26.07%, avg=18271.75, stdev=20053.83, samples=1723
iops : min= 1, max=38061, avg=4567.57, stdev=5013.43, samples=1723
write: IOPS=2920, BW=11.4MiB/s (11.0MB/s)(10.0GiB/900001msec); 0 zone resets
slat (usec): min=4, max=2554.6k, avg=143.21, stdev=2515.70
clat (usec): min=3, max=16029k, avg=2034.65, stdev=36777.34
lat (usec): min=121, max=16029k, avg=2178.19, stdev=36931.23
clat percentiles (usec):
| 1.00th=[ 184], 5.00th=[ 219], 10.00th=[ 245], 20.00th=[ 347],
| 30.00th=[ 611], 40.00th=[ 938], 50.00th=[ 1287], 60.00th=[ 1663],
| 70.00th=[ 2073], 80.00th=[ 2606], 90.00th=[ 3556], 95.00th=[ 4948],
| 99.00th=[ 12125], 99.50th=[ 20055], 99.90th=[ 53216], 99.95th=[ 80217],
| 99.99th=[156238]
bw ( KiB/s): min= 7, max=101357, per=26.06%, avg=12174.06, stdev=13378.53, samples=1722
iops : min= 1, max=25339, avg=3043.14, stdev=3344.63, samples=1722
lat (usec) : 4=0.01%, 250=10.55%, 500=15.05%, 750=8.30%, 1000=7.10%
lat (msec) : 2=26.45%, 4=24.60%, 10=6.55%, 20=0.88%, 50=0.40%
lat (msec) : 100=0.08%, 250=0.03%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2000=0.01%, >=2000=0.01%
cpu : usr=2.98%, sys=19.22%, ctx=4133616, majf=0, minf=22
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=3946881,2628140,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=16
Run status group 0 (all jobs):
READ: bw=68.5MiB/s (71.8MB/s), 17.1MiB/s-17.1MiB/s (17.9MB/s-17.0MB/s), io=60.2GiB (64.6GB), run=900001-900001msec
WRITE: bw=45.6MiB/s (47.8MB/s), 11.4MiB/s-11.4MiB/s (11.9MB/s-11.0MB/s), io=40.1GiB (43.1GB), run=900001-900001msec
Disk stats (read/write):
dm-5: ios=4095694/8382399, merge=0/0, ticks=944686/590705838, in_queue=591655521, util=98.08%, aggrios=4097011/8385705, aggrmerge=0/0, aggrticks=933147/590913671, aggrin_queue=591851814, aggrutil=98.14%
dm-0: ios=4097011/8385705, merge=0/0, ticks=933147/590913671, in_queue=591851814, util=98.14%, aggrios=4097011/7853564, aggrmerge=0/532163, aggrticks=787654/937480, aggrin_queue=148997, aggrutil=98.17%
sdc: ios=4097011/7853564, merge=0/532163, ticks=787654/937480, in_queue=148997, util=98.17%
fs_mark on same machines:
btrfs filesystem:
ssd arryy:
[h1]fs_mark -D 1024 -N 1024 -L 100 -n 1024 -p 128 -r 32 -s 4096 -t 64 -d .[/h1]
[h1]Version 3.3, 64 thread(s) starting at Thu Apr 16 23:14:11 2020[/h1]
[h1]Sync method: INBAND FSYNC: fsync() per file in write loop.[/h1]
[h1]Directories: Round Robin between directories across 1024 subdirectories with 1024 files per subdirectory.[/h1]
[h1]File names: 128 bytes long, (96 initial bytes of time stamp with 32 random bytes at end of name)[/h1]
[h1]Files info: size 4096 bytes, written with an IO size of 16384 bytes per write[/h1]
[h1]App overhead is time in microseconds spent in the test not doing file writing related system calls.[/h1]
#
FSUse% Count Size Files/sec App Overhead
1 65536 4096 10310.7 724416
1 131072 4096 10780.5 727404
1 196608 4096 10841.3 727947
1 262144 4096 9799.0 747258
1 327680 4096 7313.5 824792
1 393216 4096 4994.7 902028
1 458752 4096 4586.5 968847
1 524288 4096 4771.6 875701
1 589824 4096 4902.5 964081
1 655360 4096 4981.5 922469
1 720896 4096 4953.7 953278
1 786432 4096 4940.7 970283
1 851968 4096 4905.0 960853
1 917504 4096 4866.4 975005
1 983040 4096 4904.3 961047
1 1048576 4096 4470.7 971468
1 1114112 4096 4881.0 928539
1 1179648 4096 4898.9 985251
1 1245184 4096 4426.8 998318
1 1310720 4096 4717.5 1024192
1 1376256 4096 4795.5 1010882
1 1441792 4096 4768.4 948396
1 1507328 4096 4881.9 930210
1 1572864 4096 4706.8 926804
1 1638400 4096 4362.5 1014378
1 1703936 4096 4165.5 1017512
1 1769472 4096 4454.5 979459
1 1835008 4096 4414.1 910763
1 1900544 4096 4584.9 935274
1 1966080 4096 4245.9 974326
1 2031616 4096 4192.8 940469
1 2097152 4096 4390.3 940586
1 2162688 4096 4618.7 929184
1 2228224 4096 4399.2 926809
1 2293760 4096 4120.1 1197906
1 2359296 4096 4321.8 975645
1 2424832 4096 4282.7 946189
1 2490368 4096 4406.6 998897
1 2555904 4096 4323.5 970225
1 2621440 4096 4433.8 929924
1 2686976 4096 3859.8 972714
1 2752512 4096 3277.5 959557
1 2818048 4096 3333.6 961802
1 2883584 4096 3387.0 929832
1 2949120 4096 3329.5 947360
1 3014656 4096 3342.3 952607
1 3080192 4096 3352.9 951114
1 3145728 4096 3364.2 1045857
1 3211264 4096 3262.2 993562
1 3276800 4096 3347.9 984814
1 3342336 4096 3311.6 1007202
1 3407872 4096 3302.7 1014171
1 3473408 4096 3339.7 986689
1 3538944 4096 3327.9 1179439
1 3604480 4096 3297.4 1012885
1 3670016 4096 3315.7 955640
1 3735552 4096 3273.4 991597
1 3801088 4096 3271.2 949219
1 3866624 4096 3187.1 1085862
1 3932160 4096 3306.2 1034104
1 3997696 4096 3243.0 1086646
1 4063232 4096 3118.9 1064811
1 4128768 4096 3153.9 985299
1 4194304 4096 3169.8 1043782
1 4259840 4096 3151.4 1005640
1 4325376 4096 2933.7 1152361
1 4390912 4096 3182.4 985131
1 4456448 4096 3104.6 1024866
1 4521984 4096 3049.8 975983
1 4587520 4096 3011.5 1061210
1 4653056 4096 2894.0 1071062
1 4718592 4096 3019.0 986460
1 4784128 4096 3084.8 1041762
1 4849664 4096 3151.8 1062671
1 4915200 4096 3152.9 994083
1 4980736 4096 2962.9 1082241
1 5046272 4096 3207.6 974949
1 5111808 4096 2941.4 1251538
1 5177344 4096 3095.8 1104318
1 5242880 4096 3130.1 1037710
1 5308416 4096 3056.7 1010320
1 5373952 4096 2874.9 1013132
1 5439488 4096 3155.1 1036078
1 5505024 4096 3194.9 932772
1 5570560 4096 3162.1 1013478
1 5636096 4096 3027.3 1000038
1 5701632 4096 3064.4 976337
1 5767168 4096 3114.2 998867
1 5832704 4096 3038.4 1022219
1 5898240 4096 3135.7 1063486
1 5963776 4096 3113.2 974104
1 6029312 4096 2926.5 1044568
1 6094848 4096 3022.6 938138
1 6160384 4096 3080.7 946747
1 6225920 4096 3035.1 1041103
1 6291456 4096 2757.0 1070905
1 6356992 4096 3095.7 1002116
1 6422528 4096 2960.9 996828
1 6488064 4096 2834.2 1067899
1 6553600 4096 3015.3 1025490
xfs filesystem:
ssd array:
[h1]fs_mark -D 1024 -N 1024 -L 100 -n 1024 -p 128 -r 32 -s 4096 -t 64 -d test[/h1]
[h1]Version 3.3, 64 thread(s) starting at Thu Apr 16 23:48:42 2020[/h1]
[h1]Sync method: INBAND FSYNC: fsync() per file in write loop.[/h1]
[h1]Directories: Round Robin between directories across 1024 subdirectories with 1024 files per subdirectory.[/h1]
[h1]File names: 128 bytes long, (96 initial bytes of time stamp with 32 random bytes at end of name)[/h1]
[h1]Files info: size 4096 bytes, written with an IO size of 16384 bytes per write[/h1]
[h1]App overhead is time in microseconds spent in the test not doing file writing related system calls.[/h1]
#
FSUse% Count Size Files/sec App Overhead
54 65536 4096 7999.0 648553
54 131072 4096 7017.5 669987
54 196608 4096 4130.0 800848
54 262144 4096 3797.9 843377
54 327680 4096 3129.0 847898
54 393216 4096 3077.2 873509
54 458752 4096 2738.3 927161
54 524288 4096 2994.4 861687
55 589824 4096 3123.8 822056
55 655360 4096 3166.8 851053
55 720896 4096 3080.1 814271
55 786432 4096 2995.3 874025
55 851968 4096 3134.7 825648
55 917504 4096 3108.9 838706
55 983040 4096 2922.5 882262
55 1048576 4096 3082.5 844851
55 1114112 4096 2960.5 939395
56 1179648 4096 2942.2 873126
56 1245184 4096 2891.3 932275
56 1310720 4096 2938.7 868676
56 1376256 4096 3028.6 823515
56 1441792 4096 2754.5 865692
56 1507328 4096 2971.4 846978
56 1572864 4096 2919.7 862855
56 1638400 4096 2937.6 884359
56 1703936 4096 2809.8 920798
57 1769472 4096 2917.3 851571
57 1835008 4096 2804.6 887153
57 1900544 4096 2707.8 889684
57 1966080 4096 2894.6 910081
57 2031616 4096 2930.0 973586
57 2097152 4096 2736.2 873193
57 2162688 4096 2596.3 1074422
58 2228224 4096 2586.6 1037621
58 2293760 4096 2646.0 907799
58 2359296 4096 2746.2 909818
58 2424832 4096 2897.5 874643
58 2490368 4096 2884.1 889218
58 2555904 4096 2889.7 829827
58 2621440 4096 2723.1 980979
59 2686976 4096 2884.7 849886
59 2752512 4096 2804.5 892372
59 2818048 4096 2821.7 824137
59 2883584 4096 2837.9 855938
59 2949120 4096 2874.3 925673
59 3014656 4096 2882.0 867375
59 3080192 4096 2662.0 904706
59 3145728 4096 2909.3 852527
59 3211264 4096 2821.0 866250
60 3276800 4096 2979.5 853112
60 3342336 4096 2882.4 957123
59 3407872 4096 2735.5 870543
60 3473408 4096 2784.8 859071
60 3538944 4096 2957.5 911676
60 3604480 4096 2775.3 885856
60 3670016 4096 2778.3 940815
60 3735552 4096 2989.9 867906
60 3801088 4096 2880.0 910774
61 3866624 4096 2833.5 861072
61 3932160 4096 2783.7 910543
61 3997696 4096 2740.0 916634
61 4063232 4096 2692.5 928860
61 4128768 4096 2688.5 935526
61 4194304 4096 2691.0 907043
61 4259840 4096 2741.5 916414
61 4325376 4096 2787.0 951843
61 4390912 4096 2798.5 898106
62 4456448 4096 2878.6 870864
62 4521984 4096 2927.3 912141
62 4587520 4096 2868.6 976210
62 4653056 4096 2923.0 877059
62 4718592 4096 2739.4 899465
62 4784128 4096 2693.2 1005320
62 4849664 4096 2614.0 965127
62 4915200 4096 2826.6 895421
62 4980736 4096 2867.8 865318
63 5046272 4096 2833.4 896990
63 5111808 4096 2823.4 896109
63 5177344 4096 2763.7 880317
63 5242880 4096 2802.1 865805
63 5308416 4096 2828.5 874843
63 5373952 4096 2808.4 855479
63 5439488 4096 2890.9 871179
63 5505024 4096 2820.4 958657
63 5570560 4096 2878.4 842219
64 5636096 4096 2793.5 832474
64 5701632 4096 2783.6 872805
64 5767168 4096 2653.2 874041
64 5832704 4096 2661.8 905106
64 5898240 4096 2859.8 864755
64 5963776 4096 2863.5 879432
64 6029312 4096 2899.7 862990
64 6094848 4096 2641.9 929778
64 6160384 4096 2675.6 851607
65 6225920 4096 2671.1 935284
65 6291456 4096 2759.3 865565
65 6356992 4096 2637.7 904031
65 6422528 4096 2460.4 941102
65 6488064 4096 2738.5 895132
65 6553600 4096 2543.7 969335
ext4 filesystem:
ssd array:
[h1]fs_mark -D 1024 -N 1024 -L 100 -n 1024 -p 128 -r 32 -s 4096 -t 64 -d test[/h1]
[h1]Version 3.3, 64 thread(s) starting at Fri Apr 17 01:09:51 2020[/h1]
[h1]Sync method: INBAND FSYNC: fsync() per file in write loop.[/h1]
[h1]Directories: Round Robin between directories across 1024 subdirectories with 1024 files per subdirectory.[/h1]
[h1]File names: 128 bytes long, (96 initial bytes of time stamp with 32 random bytes at end of name)[/h1]
[h1]Files info: size 4096 bytes, written with an IO size of 16384 bytes per write[/h1]
[h1]App overhead is time in microseconds spent in the test not doing file writing related system calls.[/h1]
#
FSUse% Count Size Files/sec App Overhead
5 65536 4096 7864.5 552086
5 131072 4096 6016.0 659293
5 196608 4096 3132.8 830952
5 262144 4096 3111.3 868417
5 327680 4096 3024.1 856514
5 393216 4096 3054.6 855310
5 458752 4096 2965.3 855742
5 524288 4096 2939.1 848230
5 589824 4096 2954.9 996805
5 655360 4096 2777.9 902412
5 720896 4096 2888.6 862273
5 786432 4096 2896.1 824348
5 851968 4096 2913.1 845055
5 917504 4096 2890.0 880318
5 983040 4096 2894.7 857438
5 1048576 4096 2801.2 816060
5 1114112 4096 2911.2 841817
5 1179648 4096 2713.7 844881
5 1245184 4096 2906.6 838485
5 1310720 4096 2680.9 887251
5 1376256 4096 2699.2 878854
5 1441792 4096 2853.2 833365
5 1507328 4096 2607.5 920782
5 1572864 4096 2696.1 851208
5 1638400 4096 2855.7 843803
5 1703936 4096 2481.7 992642
5 1769472 4096 2775.0 826382
5 1835008 4096 2882.5 839825
5 1900544 4096 2781.9 833644
5 1966080 4096 2847.1 837430
5 2031616 4096 2425.9 949966
5 2097152 4096 2765.9 861270
5 2162688 4096 2645.1 861435
5 2228224 4096 2643.1 872495
5 2293760 4096 2696.8 865095
5 2359296 4096 2788.9 821588
5 2424832 4096 2701.5 854663
5 2490368 4096 2720.7 845920
5 2555904 4096 2715.9 887622
5 2621440 4096 2677.9 911478
5 2686976 4096 2769.2 849543
5 2752512 4096 2724.9 904973
5 2818048 4096 2750.3 844592
5 2883584 4096 2754.1 825710
5 2949120 4096 2773.4 834391
5 3014656 4096 2494.8 926544
5 3080192 4096 2646.0 856238
5 3145728 4096 2618.4 833934
5 3211264 4096 2773.9 839651
5 3276800 4096 2589.9 850756
5 3342336 4096 2657.2 831601
5 3407872 4096 2370.0 960856
5 3473408 4096 2706.7 817512
5 3538944 4096 2797.4 797210
5 3604480 4096 2529.4 899022
5 3670016 4096 2596.7 815678
5 3735552 4096 2711.9 819621
5 3801088 4096 2631.6 851691
5 3866624 4096 2751.4 842720
5 3932160 4096 2565.7 855631
5 3997696 4096 2778.3 795482
5 4063232 4096 2723.5 889595
5 4128768 4096 2658.2 861042
5 4194304 4096 2699.9 819071
5 4259840 4096 2644.2 852887
5 4325376 4096 2754.3 807878
5 4390912 4096 2692.8 840796
5 4456448 4096 2819.3 807838
5 4521984 4096 2690.4 835833
5 4587520 4096 2778.1 846772
5 4653056 4096 2629.5 930968
5 4718592 4096 2708.7 831528
5 4784128 4096 2558.6 846661
5 4849664 4096 2663.4 898813
5 4915200 4096 2591.5 895358
5 4980736 4096 2765.3 853465
5 5046272 4096 2539.0 875678
5 5111808 4096 2720.3 864972
5 5177344 4096 2757.4 834596
5 5242880 4096 2640.7 842025
5 5308416 4096 2686.0 837971
5 5373952 4096 2642.4 869985
5 5439488 4096 2744.4 800594
5 5505024 4096 2736.5 941328
5 5570560 4096 2649.0 850955
5 5636096 4096 2729.2 821149
5 5701632 4096 2648.9 861357
5 5767168 4096 2548.3 876694
5 5832704 4096 2525.8 893559
5 5898240 4096 2703.9 848639
5 5963776 4096 2553.9 864792
5 6029312 4096 2548.8 874867
5 6094848 4096 2592.4 884055
5 6160384 4096 2548.6 868029
5 6225920 4096 2667.0 828056
5 6291456 4096 2559.1 881193
5 6356992 4096 2661.5 847813
5 6422528 4096 2501.0 838245
5 6488064 4096 2674.3 829347
5 6553600 4096 2566.9 860537