Proxmox VE 6.1 + Cephで速度計測 各ノードHDD4台の記録
目次
最初に
前記事のCephを拡張して各ノードHDD4台のパターンを計測します
3台までは同一の7200rpmHDDでしたが4台目は5400rpmHDDを使用してます
そしてOSDが増えるのでプールのPG値を変更したパターンも計測してみました
計測環境
HDD1台で検証した前記事に詳細を掲載してますのでそちらをご覧ください
Proxmox VE 6.1 + Cephで速度計測：HDDx1 | fefcc.net
https://fefcc.net/archives/1040
今回のCeph構成
各ノードに登録しているOSDは
HDD ST1000DM010 1TB 7200rpmを3台とHDD WD40EZRZ 4TB 5400rpmを1台
DB/WALともにOSDと同じHDDを指定
プールは各ノードHDD1台の構成を拡張して使用
Pool01という名でCeph/RBD形式のプール作成
Pool01のPG値は128のまま
9OSDから12OSDへ拡張
前記事と同様に既存環境へOSDを追加
各ノードHDD4台で合計12OSDへ拡張した結果がこちら
使用状況のTotal Sizeは6.64TB（6.04TiB）になり
12OSDなのでクラスタ全体のPG値は250×12=3000になります
各HDD4台：計12OSD（PG:128）での計測結果
Proxmox起動直後のメモリ使用状況
pve01は1.96GiB、pve01は1.92GiB、pve03は1.76GiB
9OSD構成に比べて更に少し増加しました
CentOS7のVMから計測
[root@centos7-bench ~]# ./fio_cdm.sh /root Testing in /root Configuration: Size:1024m Loops:5 Write Only Zeroes:0 Running Benchmark, please wait... Results: Sequential Read: 148MB/s IOPS=0 Sequential Write: 28MB/s IOPS=0 512KB Read: 73MB/s IOPS=147 512KB Write: 6MB/s IOPS=12 Sequential Q32T1 Read: 134MB/s IOPS=4 Sequential Q32T1 Write: 27MB/s IOPS=0 4KB Read: 4MB/s IOPS=1250 4KB Write: 0MB/s IOPS=42 4KB Q32T1 Read: 4MB/s IOPS=1250 4KB Q32T1 Write: 0MB/s IOPS=64 4KB Q8T8 Read: 31MB/s IOPS=8000 4KB Q8T8 Write: 0MB/s IOPS=114 [root@centos7-bench ~]#
Windows10のVMから計測
------------------------------------------------------------------------------
CrystalDiskMark 7.0.0 x64 (C) 2007-2019 hiyohiyo
Crystal Dew World: https://crystalmark.info/
------------------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes
[Read]
Sequential 1MiB (Q= 8, T= 1): 168.201 MB/s [ 160.4 IOPS] < 49445.50 us>
Sequential 1MiB (Q= 1, T= 1): 143.258 MB/s [ 136.6 IOPS] < 7317.25 us>
Random 4KiB (Q= 32, T=16): 124.382 MB/s [ 30366.7 IOPS] < 16706.34 us>
Random 4KiB (Q= 1, T= 1): 7.295 MB/s [ 1781.0 IOPS] < 560.60 us>
[Write]
Sequential 1MiB (Q= 8, T= 1): 24.749 MB/s [ 23.6 IOPS] <321442.40 us>
Sequential 1MiB (Q= 1, T= 1): 7.760 MB/s [ 7.4 IOPS] <134540.40 us>
Random 4KiB (Q= 32, T=16): 1.261 MB/s [ 307.9 IOPS] <490518.05 us>
Random 4KiB (Q= 1, T= 1): 0.181 MB/s [ 44.2 IOPS] < 22559.02 us>
[Mix] Read 70%/Write 30%
Sequential 1MiB (Q= 8, T= 1): 46.559 MB/s [ 44.4 IOPS] <176739.63 us>
Sequential 1MiB (Q= 1, T= 1): 21.600 MB/s [ 20.6 IOPS] < 48440.56 us>
Random 4KiB (Q= 32, T=16): 3.660 MB/s [ 893.6 IOPS] <245186.48 us>
Random 4KiB (Q= 1, T= 1): 0.571 MB/s [ 139.4 IOPS] < 7143.01 us>
Profile: Default
Test: 1 GiB (x5) [Interval: 5 sec] <DefaultAffinity=DISABLED>
Date: 2020/02/10 1:31:15
OS: Windows 10 Professional [10.0 Build 18363] (x64)
rados benchで計測
ブロックサイズ1M＆16スレッドで60秒間書き込み
結果は78.2615MB/s
root@pve01:~# rados bench -p Pool01 -t 16 -b 1M 60 write --no-cleanup
hints = 1
Maintaining 16 concurrent writes of 1048576 bytes to objects of size 1048576 for up to 60 seconds or 0 objects
Object prefix: benchmark_data_pve01_17801
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 16 72 56 55.9972 56 0.165496 0.250112
2 16 154 138 68.9957 82 0.159491 0.221346
3 16 238 222 73.9951 84 0.153577 0.212933
4 16 311 295 73.745 73 0.191293 0.212819
5 16 397 381 76.1947 86 0.171945 0.205416
6 16 474 458 76.328 77 0.101957 0.206043
7 16 554 538 76.8518 80 0.221669 0.205987
8 16 630 614 76.7445 76 0.132657 0.205616
9 16 704 688 76.4387 74 0.128847 0.20595
10 16 783 767 76.6943 79 0.12302 0.206275
11 16 858 842 76.5397 75 0.246698 0.20653
12 16 936 920 76.6609 78 0.117597 0.206276
13 16 1016 1000 76.9173 80 0.231439 0.206097
14 16 1096 1080 77.1371 80 0.15169 0.205232
15 16 1177 1161 77.3942 81 0.174891 0.205383
16 16 1255 1239 77.4318 78 0.169834 0.205156
17 16 1329 1313 77.2295 74 0.359513 0.205391
18 16 1414 1398 77.6608 85 0.183955 0.204992
19 16 1488 1472 77.4676 74 0.110496 0.204643
2020-02-10 01:36:20.618643 min lat: 0.0588456 max lat: 0.507087 avg lat: 0.205063
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 16 1572 1556 77.7937 84 0.198934 0.205063
21 16 1646 1630 77.6127 74 0.128768 0.204992
22 16 1732 1716 77.9936 86 0.139595 0.204756
23 16 1805 1789 77.7763 73 0.123642 0.204774
24 16 1888 1872 77.9937 83 0.0906235 0.20458
25 16 1968 1952 78.0737 80 0.172659 0.204392
26 16 2040 2024 77.8398 72 0.197347 0.204638
27 16 2117 2101 77.8084 77 0.124578 0.204366
28 16 2197 2181 77.8865 80 0.225285 0.204436
29 16 2274 2258 77.8557 77 0.183817 0.204546
30 16 2355 2339 77.9603 81 0.1724 0.204751
31 16 2426 2410 77.7356 71 0.122725 0.204995
32 16 2505 2489 77.7749 79 0.194299 0.205124
33 16 2580 2564 77.6907 75 0.261757 0.205421
34 16 2663 2647 77.8467 83 0.0875907 0.204936
35 16 2743 2727 77.908 80 0.169021 0.204809
36 16 2821 2805 77.9104 78 0.190399 0.204631
37 16 2897 2881 77.8586 76 0.209928 0.204845
38 16 2977 2961 77.9148 80 0.274185 0.204599
39 16 3056 3040 77.9424 79 0.223076 0.204754
2020-02-10 01:36:40.620237 min lat: 0.0588456 max lat: 0.507087 avg lat: 0.204877
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
40 16 3129 3113 77.8187 73 0.143219 0.204877
41 16 3208 3192 77.8474 79 0.204169 0.20515
42 16 3276 3260 77.6127 68 0.300644 0.205497
43 16 3352 3336 77.5749 76 0.178236 0.205643
44 16 3436 3420 77.7207 84 0.193754 0.205442
45 16 3511 3495 77.6601 75 0.203588 0.205416
46 16 3594 3578 77.776 83 0.155098 0.205125
47 16 3676 3660 77.8656 82 0.231091 0.205073
48 16 3753 3737 77.8475 77 0.148205 0.205011
49 16 3831 3815 77.8504 78 0.25077 0.205146
50 16 3912 3896 77.9132 81 0.187128 0.20498
51 16 3992 3976 77.954 80 0.230374 0.204821
52 16 4070 4054 77.9548 78 0.177336 0.204677
53 16 4150 4134 77.9932 80 0.226882 0.204675
54 16 4227 4211 77.9747 77 0.182765 0.204709
55 16 4313 4297 78.1204 86 0.182894 0.20449
56 16 4394 4378 78.1717 81 0.263955 0.20436
57 16 4470 4454 78.1335 76 0.211644 0.204446
58 16 4547 4531 78.1138 77 0.184792 0.204335
59 16 4623 4607 78.0778 76 0.133372 0.204439
2020-02-10 01:37:00.622338 min lat: 0.0588456 max lat: 0.507087 avg lat: 0.204404
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
60 16 4706 4690 78.1597 83 0.128414 0.204404
Total time run: 60.1317
Total writes made: 4706
Write size: 1048576
Object size: 1048576
Bandwidth (MB/sec): 78.2615
Stddev Bandwidth: 4.89956
Max bandwidth (MB/sec): 86
Min bandwidth (MB/sec): 56
Average IOPS: 78
Stddev IOPS: 4.89956
Max IOPS: 86
Min IOPS: 56
Average Latency(s): 0.204426
Stddev Latency(s): 0.0579736
Max latency(s): 0.507087
Min latency(s): 0.0588456
root@pve01:~#
先ほど書き込んだデータを16スレッドでシーケンシャル読み込み
結果は151.829MB/s
root@pve01:~# rados bench -p Pool01 -t 16 60 seq
hints = 1
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 16 179 163 162.979 163 0.146066 0.0887821
2 16 334 318 158.977 155 0.157372 0.0956821
3 16 492 476 158.645 158 0.0338668 0.0973808
4 16 632 616 153.981 140 0.107825 0.102012
5 16 782 766 153.182 150 0.0137733 0.10315
6 16 919 903 150.483 137 0.00345593 0.104152
7 16 1075 1059 151.269 156 0.0843525 0.104431
8 16 1233 1217 152.106 158 0.210866 0.103928
9 16 1393 1377 152.982 160 0.0450929 0.10341
10 16 1545 1529 152.882 152 0.179122 0.103876
11 16 1705 1689 153.528 160 0.0693339 0.103302
12 16 1851 1835 152.899 146 0.00299264 0.103563
13 16 1996 1980 152.291 145 0.00916029 0.104225
14 16 2162 2146 153.269 166 0.0903366 0.103499
15 16 2319 2303 153.517 157 0.013828 0.103621
16 16 2472 2456 153.483 153 0.0799746 0.103672
17 16 2613 2597 152.748 141 0.0604253 0.104167
18 16 2769 2753 152.928 156 0.00954085 0.104141
19 16 2924 2908 153.037 155 0.00837231 0.103778
2020-02-10 01:37:42.778832 min lat: 0.00279779 max lat: 0.463324 avg lat: 0.104138
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 16 3072 3056 152.784 148 0.273736 0.104138
21 16 3221 3205 152.604 149 0.018459 0.104435
22 16 3373 3357 152.576 152 0.0430277 0.104178
23 16 3520 3504 152.333 147 0.242463 0.104565
24 16 3681 3665 152.693 161 0.0175145 0.104325
25 16 3831 3815 152.585 150 0.0103998 0.104412
26 16 3983 3967 152.562 152 0.262054 0.104396
27 16 4121 4105 152.022 138 0.345315 0.104779
28 16 4265 4249 151.735 144 0.201918 0.105102
29 16 4421 4405 151.881 156 0.0443349 0.10486
30 16 4575 4559 151.951 154 0.00309545 0.104876
Total time run: 30.9954
Total reads made: 4706
Read size: 1048576
Object size: 1048576
Bandwidth (MB/sec): 151.829
Average IOPS: 151
Stddev IOPS: 7.3975
Max IOPS: 166
Min IOPS: 137
Average Latency(s): 0.105261
Max latency(s): 0.463324
Min latency(s): 0.00279779
root@pve01:~#
ブロックサイズ4K＆16スレッドで60秒間書き込み
結果は1.37528MB/s 352IOPS
root@pve01:~# rados bench -p Pool01 -t 16 -b 4K 60 write --no-cleanup
hints = 1
Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 60 seconds or 0 objects
Object prefix: benchmark_data_pve01_19047
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 16 412 396 1.54674 1.54688 0.0219786 0.0392206
2 16 780 764 1.49201 1.4375 0.0700195 0.0417246
3 16 1117 1101 1.43339 1.31641 0.00792616 0.0431849
4 16 1466 1450 1.41582 1.36328 0.035507 0.0438561
5 16 1786 1770 1.38263 1.25 0.0302969 0.0450807
6 16 2157 2141 1.39371 1.44922 0.101288 0.0446493
7 16 2503 2487 1.38767 1.35156 0.0948331 0.0449212
8 16 2831 2815 1.37435 1.28125 0.0226422 0.0452966
9 16 3180 3164 1.37309 1.36328 0.018731 0.0453217
10 16 3503 3487 1.36194 1.26172 0.0222619 0.0457924
11 16 3869 3853 1.36808 1.42969 0.0187413 0.0456161
12 16 4204 4188 1.36312 1.30859 0.0948958 0.0457482
13 16 4572 4556 1.36883 1.4375 0.0218892 0.0456125
14 16 4929 4913 1.37066 1.39453 0.0168531 0.0455475
15 16 5307 5291 1.37771 1.47656 0.0180912 0.0453181
16 16 5637 5621 1.37216 1.28906 0.0297729 0.0455184
17 16 5962 5946 1.36611 1.26953 0.0514815 0.045652
18 16 6287 6271 1.36073 1.26953 0.0324946 0.0458158
19 16 6676 6660 1.36909 1.51953 0.0415566 0.0456135
2020-02-10 01:38:47.411570 min lat: 0.00559087 max lat: 0.227312 avg lat: 0.0453695
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 16 7060 7044 1.37562 1.5 0.0341274 0.0453695
21 16 7454 7438 1.38339 1.53906 0.058937 0.045118
22 16 7844 7828 1.38974 1.52344 0.0226823 0.0449424
23 16 8196 8180 1.38909 1.375 0.123987 0.0449226
24 16 8519 8503 1.38378 1.26172 0.0100244 0.0450275
25 16 8864 8848 1.38233 1.34766 0.0197402 0.0450687
26 16 9255 9239 1.38789 1.52734 0.021905 0.0450016
27 16 9627 9611 1.3903 1.45312 0.010088 0.0448937
28 16 9966 9950 1.38794 1.32422 0.0246682 0.0449382
29 16 10271 10255 1.38115 1.19141 0.0720025 0.04513
30 16 10642 10626 1.38341 1.44922 0.0169297 0.0451129
31 16 10977 10961 1.38099 1.30859 0.102894 0.0451629
32 16 11326 11310 1.38044 1.36328 0.0222394 0.0452338
33 16 11708 11692 1.38382 1.49219 0.0626263 0.0451206
34 16 12042 12026 1.38149 1.30469 0.0405936 0.0452114
35 16 12386 12370 1.3804 1.34375 0.0210983 0.0452566
36 16 12737 12721 1.38014 1.37109 0.0936307 0.0452485
37 16 13087 13071 1.37978 1.36719 0.0412265 0.0452646
38 16 13420 13404 1.3777 1.30078 0.0227441 0.0453229
39 16 13762 13746 1.37663 1.33594 0.088562 0.0453544
2020-02-10 01:39:07.414329 min lat: 0.00559087 max lat: 0.227312 avg lat: 0.0452414
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
40 16 14158 14142 1.38088 1.54688 0.0225702 0.0452414
41 16 14521 14505 1.38178 1.41797 0.0226132 0.045193
42 16 14882 14866 1.38245 1.41016 0.0222648 0.0451686
43 16 15196 15180 1.37882 1.22656 0.0168413 0.0453136
44 16 15547 15531 1.37864 1.37109 0.0318043 0.0453123
45 16 15913 15897 1.37977 1.42969 0.0566436 0.0452802
46 16 16270 16254 1.38009 1.39453 0.0371996 0.0452358
47 16 16626 16610 1.38031 1.39062 0.0193016 0.0452585
48 16 16984 16968 1.38068 1.39844 0.0941691 0.045236
49 16 17309 17293 1.37841 1.26953 0.0146899 0.0452944
50 16 17644 17628 1.37701 1.30859 0.0366679 0.0453443
51 16 17975 17959 1.37536 1.29297 0.0226249 0.045377
52 16 18334 18318 1.37588 1.40234 0.118096 0.0454173
53 16 18679 18663 1.37534 1.34766 0.0138271 0.0453933
54 16 19080 19064 1.37888 1.56641 0.03536 0.0453034
55 16 19456 19440 1.38051 1.46875 0.0302081 0.0452346
56 16 19753 19737 1.37657 1.16016 0.0206638 0.045377
57 16 20102 20086 1.37633 1.36328 0.0113361 0.0453688
58 16 20450 20434 1.37604 1.35938 0.0727052 0.0454109
59 16 20789 20773 1.37516 1.32422 0.0174198 0.0454218
2020-02-10 01:39:27.416769 min lat: 0.00559087 max lat: 0.227312 avg lat: 0.0453787
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
60 16 21157 21141 1.37619 1.4375 0.0645447 0.0453787
Total time run: 60.093
Total writes made: 21157
Write size: 4096
Object size: 4096
Bandwidth (MB/sec): 1.37528
Stddev Bandwidth: 0.0933819
Max bandwidth (MB/sec): 1.56641
Min bandwidth (MB/sec): 1.16016
Average IOPS: 352
Stddev IOPS: 23.9058
Max IOPS: 401
Min IOPS: 297
Average Latency(s): 0.0454356
Stddev Latency(s): 0.0324911
Max latency(s): 0.227312
Min latency(s): 0.00559087
root@pve01:~#
クリーンアップ
root@pve01:~# rados -p Pool01 cleanup Removed 21157 objects root@pve01:~#
rbd benchで計測
プール内に20GBでイメージを作成
root@pve01:~# rbd create --size=20G --pool=Pool01 Test01 root@pve01:~#
ブロックサイズ1M＆16スレッドで1024MBをシーケンシャル書き込み＆読み込みを計測
結果はWrite:17秒 約63MB/s・Read:6秒 約158MB/s
root@pve01:~# rbd bench Test01 --io-type write --io-size 1M --io-threads 16 --io-total 1024M --io-pattern seq --pool=Pool01
bench type write io_size 1048576 io_threads 16 bytes 1073741824 pattern sequential
SEC OPS OPS/SEC BYTES/SEC
1 80 124.35 130387700.05
2 144 84.56 88670986.56
3 192 76.92 80656493.37
4 256 72.03 75530021.11
5 304 67.97 71268269.01
6 368 57.28 60059249.75
7 432 58.30 61129130.76
8 496 59.61 62500873.46
9 560 58.96 61822042.79
10 624 61.54 64525192.48
11 688 62.69 65738832.08
12 752 62.64 65687352.59
13 800 61.49 64472986.55
14 864 61.84 64840204.13
15 928 59.93 62845907.26
16 976 59.60 62495424.37
elapsed: 17 ops: 1024 ops/sec: 60.06 bytes/sec: 62980947.77
root@pve01:~# rbd bench Test01 --io-type read --io-size 1M --io-threads 16 --io-total 1024M --io-pattern seq --pool=Pool01
bench type read io_size 1048576 io_threads 16 bytes 1073741824 pattern sequential
SEC OPS OPS/SEC BYTES/SEC
1 160 178.13 186783427.20
2 288 156.37 163968316.84
3 448 159.12 166844953.46
4 624 163.26 171189262.14
5 800 163.98 171946127.33
6 928 155.59 163143087.99
elapsed: 6 ops: 1024 ops/sec: 151.12 bytes/sec: 158456175.35
root@pve01:~#
ブロックサイズ4K＆16スレッドで512MBをランダム書き込み＆読み込みを計測
結果はWrite:240秒 約2.23MB/s・Read:8秒 約60MB/s
root@pve01:~# rbd bench Test01 --io-type write --io-size 4K --io-threads 16 --io-total 512M --io-pattern rand --pool=Pool01
bench type write io_size 4096 io_threads 16 bytes 536870912 pattern random
SEC OPS OPS/SEC BYTES/SEC
1 5360 4199.83 17202513.73
2 5424 2277.97 9330549.02
3 5472 1714.93 7024359.74
4 5536 1329.45 5445422.57
5 5616 1083.87 4439521.07
6 5680 65.25 267265.03
7 5744 67.39 276046.27
8 5808 68.01 278583.21
9 5872 67.63 277013.09
10 5936 65.36 267701.73
11 6000 62.89 257599.78
12 6064 64.72 265102.68
13 6112 60.90 249425.93
14 6160 58.58 239951.36
15 6240 58.73 240559.17
16 6304 63.28 259186.98
58 6368 6.54 26786.29
60 6384 5.77 23645.20
61 6400 5.08 20824.52
62 6416 3.75 15367.68
63 10752 94.51 387128.72
64 10944 830.76 3402785.51
65 11072 916.30 3753181.39
66 11136 996.59 4082044.62
67 11296 995.07 4075790.42
68 11424 138.27 566337.72
69 11584 128.72 527219.34
70 11728 140.77 576581.10
71 11920 157.42 644806.19
72 12080 158.44 648976.31
73 12224 158.85 650648.96
74 12384 160.25 656383.84
75 12576 164.85 675207.66
76 12704 152.64 625220.94
77 12864 155.06 635113.67
78 13072 170.14 696883.66
79 13280 179.48 735149.88
80 13488 186.72 764824.23
81 13648 192.96 790365.55
82 13824 195.27 799837.59
83 14032 190.62 780778.71
84 14224 187.00 765940.61
85 14384 180.93 741088.07
86 14672 205.12 840171.27
87 14944 222.21 910185.54
88 15184 231.50 948231.87
89 15440 246.34 1009022.53
90 15664 256.40 1050214.07
91 15968 255.51 1046569.79
92 16256 263.44 1079063.31
93 16576 279.06 1143023.67
94 16896 291.42 1193662.02
95 17264 317.20 1299233.87
96 17632 335.47 1374086.50
97 18016 348.64 1428024.79
98 18400 366.25 1500161.15
99 18784 373.11 1528248.20
100 19216 389.76 1596459.52
107 19456 155.04 635056.30
108 19488 134.40 550499.32
109 23248 439.91 1801869.87
112 23264 324.44 1328891.85
113 23296 314.80 1289429.38
114 26688 1045.65 4282977.82
116 26704 806.77 3304510.41
117 26720 432.68 1772275.30
118 29776 1242.70 5090090.33
119 29824 1056.27 4326473.56
120 29840 578.11 2367951.17
121 32592 1461.71 5987157.98
122 32752 1147.59 4700546.50
123 32784 540.21 2212692.07
124 35360 1132.52 4638817.19
126 35568 892.18 3654351.78
127 38000 898.30 3679447.08
129 38176 765.64 3136053.72
130 40672 1248.84 5115253.30
132 40880 678.11 2777518.97
133 43504 1236.86 5066168.88
134 43616 709.06 2904318.72
135 43712 985.01 4034615.20
136 46544 978.63 4008456.26
137 46608 1179.52 4831329.76
138 46672 631.05 2584781.31
139 49776 1508.26 6177849.51
140 50480 1141.65 4676215.31
141 50512 672.97 2756490.70
142 50848 861.75 3529743.08
143 54000 1430.07 5857586.07
145 54656 748.44 3065593.47
146 54672 793.31 3249382.22
147 58048 1464.95 6000428.46
148 58304 1093.85 4480419.19
150 58320 603.66 2472610.70
151 62000 1343.03 5501061.35
153 62912 1078.49 4417496.11
155 62928 583.15 2388583.74
156 67152 1231.58 5044534.65
158 68128 1179.37 4830680.25
161 68144 606.61 2484676.35
162 72368 1159.34 4748676.19
163 73312 1365.54 5593258.33
164 74208 881.96 3612525.25
165 75184 1102.46 4515656.53
166 76192 1653.18 6771419.16
167 77216 966.47 3958652.51
168 78160 963.39 3946065.37
169 79056 956.55 3918034.90
170 79808 928.48 3803038.97
171 80384 829.08 3395912.57
172 81120 778.90 3190369.88
173 82288 832.22 3408790.42
174 83360 867.01 3551262.51
175 84288 898.84 3681648.01
176 85248 979.03 4010091.32
177 86224 1007.06 4124927.70
178 87664 973.17 3986097.06
179 88112 955.71 3914601.79
180 89008 941.70 3857209.37
181 89728 883.94 3620626.14
182 90416 845.81 3464433.66
183 91216 784.42 3212985.83
184 91920 757.33 3102015.34
185 92704 709.65 2906720.38
186 93312 720.81 2952426.86
187 93984 720.49 2951117.32
188 94752 694.94 2846475.28
189 95264 632.35 2590103.29
190 95760 631.38 2586129.38
191 96448 630.20 2581296.79
192 97376 669.27 2741331.11
193 98384 709.35 2905481.63
194 99008 792.52 3246147.17
195 99872 829.67 3398318.35
196 100864 886.71 3631967.66
197 101632 858.72 3517325.58
198 102560 862.06 3531007.20
199 103280 856.42 3507898.71
200 104112 846.61 3467718.06
201 104944 821.22 3363727.60
202 105760 807.48 3307433.22
203 106560 818.97 3354490.57
204 107472 835.03 3420265.31
205 108224 823.02 3371109.92
206 109072 818.36 3352021.89
207 109936 855.70 3504958.67
208 110848 827.77 3390527.32
209 111536 820.65 3361362.06
210 112448 846.12 3465704.75
211 113184 811.33 3323217.20
212 114000 816.03 3342462.80
213 114800 819.20 3355458.42
214 115584 796.82 3263766.00
215 116320 770.67 3156660.96
216 117216 793.04 3248307.99
217 118144 801.82 3284274.59
218 118688 773.85 3169704.98
219 119568 809.07 3313931.30
220 120480 836.65 3426923.61
224 121808 536.43 2197203.71
225 121840 442.09 1810789.19
226 121856 388.79 1592488.70
227 126320 842.70 3451705.21
228 127200 839.55 3438780.04
229 128032 1458.23 5972929.28
230 128704 1541.72 6314868.59
231 129552 1555.31 6370559.33
232 130224 782.65 3205719.80
elapsed: 240 ops: 131072 ops/sec: 545.08 bytes/sec: 2232629.67
root@pve01:~# rbd bench Test01 --io-type read --io-size 4K --io-threads 16 --io-total 512M --io-pattern rand --pool=Pool01
bench type read io_size 4096 io_threads 16 bytes 536870912 pattern random
SEC OPS OPS/SEC BYTES/SEC
1 15344 15421.05 63164628.70
2 28896 14484.37 59327994.50
3 43648 14534.69 59534081.70
4 58608 14670.07 60088593.22
5 74128 14840.06 60784892.12
6 89168 14764.19 60474130.83
7 104336 15087.38 61797903.35
8 119088 15111.56 61896938.28
elapsed: 8 ops: 131072 ops/sec: 14833.25 bytes/sec: 60756994.84
root@pve01:~#
イメージ削除
root@pve01:~# rbd rm --pool=Pool01 Test01 Removing image: 100% complete...done. root@pve01:~#
計測後のメモリ使用状況
ピーク値で5.6GiBと速度が変わらないせいか9OSDを同レベルの結果
プールのpg_numを128から512へ変更
ここまでの計測は3OSDをベースにpg_num:128で行ってきました
ここからはPG値を増やすと計測結果にどのような変化があるかを確認していきます
12OSDまで大きくなると1024を試したいところですが全体のPG値が3000なので
PG不足のため今回は512に変更することにします
pg_numを変更する際でもプールは利用可能なのでVMを起動したままでも処理が可能です
しかし現実にはIOPSが極端に低下、もしくは一時的に0になるケースもあるようなので
安全の為にVMを停止してから実行することが推奨されています
また既存プールのpg_numを変更する注意点としては
一度設定すると減らすことはできません
pg_numを変更する場合はProxmoxのシェルからコマンドで対応します
（Pool01の部分に変更したいプール名を指定）
root@pve01:~# ceph osd pool set Pool01 pg_num 512 set pool 4 pg_num to 512 root@pve01:~#
コマンドを実行するとすぐにWEBGUIに反映されます
そして裏ではリマップ処理が行われていました
PGsの数値が128から増えているのがわかります
あとはひたすら待つだけです
完了するといつも通り全て緑になります（PGも512）
各HDD4台：計12OSD（PG:512）での計測結果
Proxmox起動直後のメモリ使用状況
pve01は2.00GiB、pve01は1.91GiB、pve03は1.88GiB
PG:128に比べると減っているノードもあるので誤差の範囲内だと思います
CentOS7のVMから計測
[root@centos7-bench ~]# ./fio_cdm.sh /root Testing in /root Configuration: Size:1024m Loops:5 Write Only Zeroes:0 Running Benchmark, please wait... Results: Sequential Read: 167MB/s IOPS=0 Sequential Write: 26MB/s IOPS=0 512KB Read: 89MB/s IOPS=178 512KB Write: 6MB/s IOPS=13 Sequential Q32T1 Read: 135MB/s IOPS=4 Sequential Q32T1 Write: 26MB/s IOPS=0 4KB Read: 3MB/s IOPS=1000 4KB Write: 0MB/s IOPS=56 4KB Q32T1 Read: 3MB/s IOPS=1000 4KB Q32T1 Write: 0MB/s IOPS=56 4KB Q8T8 Read: 31MB/s IOPS=8000 4KB Q8T8 Write: 0MB/s IOPS=96 [root@centos7-bench ~]#
Windows10のVMから計測
------------------------------------------------------------------------------
CrystalDiskMark 7.0.0 x64 (C) 2007-2019 hiyohiyo
Crystal Dew World: https://crystalmark.info/
------------------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes
[Read]
Sequential 1MiB (Q= 8, T= 1): 180.573 MB/s [ 172.2 IOPS] < 45960.04 us>
Sequential 1MiB (Q= 1, T= 1): 147.644 MB/s [ 140.8 IOPS] < 7087.28 us>
Random 4KiB (Q= 32, T=16): 143.350 MB/s [ 34997.6 IOPS] < 14520.04 us>
Random 4KiB (Q= 1, T= 1): 7.221 MB/s [ 1762.9 IOPS] < 566.34 us>
[Write]
Sequential 1MiB (Q= 8, T= 1): 23.280 MB/s [ 22.2 IOPS] <353682.79 us>
Sequential 1MiB (Q= 1, T= 1): 7.130 MB/s [ 6.8 IOPS] <141298.37 us>
Random 4KiB (Q= 32, T=16): 1.191 MB/s [ 290.8 IOPS] <493584.28 us>
Random 4KiB (Q= 1, T= 1): 0.180 MB/s [ 43.9 IOPS] < 22189.37 us>
[Mix] Read 70%/Write 30%
Sequential 1MiB (Q= 8, T= 1): 48.450 MB/s [ 46.2 IOPS] <170087.97 us>
Sequential 1MiB (Q= 1, T= 1): 20.974 MB/s [ 20.0 IOPS] < 48525.71 us>
Random 4KiB (Q= 32, T=16): 4.117 MB/s [ 1005.1 IOPS] <246383.30 us>
Random 4KiB (Q= 1, T= 1): 0.556 MB/s [ 135.7 IOPS] < 7308.08 us>
Profile: Default
Test: 1 GiB (x5) [Interval: 5 sec] <DefaultAffinity=DISABLED>
Date: 2020/02/10 3:00:00
OS: Windows 10 Professional [10.0 Build 18363] (x64)
rados benchで計測
ブロックサイズ1M＆16スレッドで60秒間書き込み
結果は75.797MB/s
root@pve01:~# rados bench -p Pool01 -t 16 -b 1M 60 write --no-cleanup
hints = 1
Maintaining 16 concurrent writes of 1048576 bytes to objects of size 1048576 for up to 60 seconds or 0 objects
Object prefix: benchmark_data_pve01_16994
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 16 61 45 45.0003 45 0.179009 0.313645
2 16 134 118 58.9981 73 0.146351 0.252613
3 16 209 193 64.3304 75 0.0970457 0.24142
4 16 270 254 63.4966 61 0.208428 0.243244
5 16 344 328 65.5962 74 0.313737 0.23769
6 16 412 396 65.996 68 0.259696 0.238033
7 16 478 462 65.9957 66 0.134319 0.236217
8 16 555 539 67.3704 77 0.217593 0.234341
9 16 628 612 67.9956 73 0.266892 0.231898
10 16 703 687 68.6954 75 0.212791 0.229813
11 16 782 766 69.6316 79 0.123396 0.22836
12 16 850 834 69.4953 68 0.364031 0.229183
13 16 920 904 69.5337 70 0.233671 0.228082
14 16 998 982 70.138 78 0.200291 0.226404
15 16 1071 1055 70.3284 73 0.216116 0.225258
16 16 1142 1126 70.3697 71 0.248266 0.225412
17 16 1218 1202 70.7006 76 0.186702 0.224394
18 16 1296 1280 71.1058 78 0.310638 0.223829
19 16 1371 1355 71.3105 75 0.078892 0.223243
2020-02-10 03:09:47.222286 min lat: 0.0667525 max lat: 0.560139 avg lat: 0.222428
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 16 1448 1432 71.5947 77 0.151228 0.222428
21 16 1524 1508 71.8041 76 0.133552 0.222046
22 16 1598 1582 71.9036 74 0.200115 0.221884
23 16 1672 1656 71.9946 74 0.167725 0.221046
24 16 1751 1735 72.2862 79 0.279627 0.220434
25 16 1823 1807 72.2745 72 0.17569 0.220422
26 16 1898 1882 72.3792 75 0.315959 0.21977
27 16 1978 1962 72.6612 80 0.194949 0.21922
28 16 2054 2038 72.7802 76 0.177378 0.219383
29 16 2130 2114 72.8911 76 0.151689 0.21887
30 16 2207 2191 73.0279 77 0.115531 0.218245
31 16 2287 2271 73.2526 80 0.138093 0.217774
32 16 2367 2351 73.4633 80 0.255374 0.217
33 16 2446 2430 73.6309 79 0.300343 0.216503
34 16 2518 2502 73.5827 72 0.179602 0.216418
35 16 2591 2575 73.5659 73 0.100665 0.216636
36 16 2672 2656 73.7723 81 0.157614 0.216181
37 16 2751 2735 73.9134 79 0.0901166 0.215817
38 16 2825 2809 73.9155 74 0.292583 0.215703
39 16 2906 2890 74.097 81 0.220832 0.215078
2020-02-10 03:10:07.223844 min lat: 0.0667525 max lat: 0.560139 avg lat: 0.214557
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
40 16 2992 2976 74.3943 86 0.322516 0.214557
41 16 3068 3052 74.4334 76 0.361068 0.214233
42 16 3152 3136 74.661 84 0.235363 0.214126
43 16 3226 3210 74.6455 74 0.225949 0.213862
44 16 3302 3286 74.6761 76 0.166667 0.213423
45 16 3384 3368 74.8387 82 0.182444 0.21335
46 16 3462 3446 74.9073 78 0.146709 0.213126
47 16 3539 3523 74.9517 77 0.165489 0.212974
48 16 3617 3601 75.015 78 0.129591 0.212937
49 16 3696 3680 75.0962 79 0.11229 0.212658
50 16 3776 3760 75.1942 80 0.241682 0.212631
51 16 3845 3829 75.0727 69 0.155895 0.212475
52 16 3929 3913 75.2442 84 0.0754374 0.21227
53 16 4007 3991 75.2961 78 0.146067 0.212184
54 16 4086 4070 75.3646 79 0.328053 0.211903
55 16 4164 4148 75.4124 78 0.0978352 0.211787
56 16 4245 4229 75.5121 81 0.191431 0.211554
57 16 4324 4308 75.5732 79 0.155393 0.211252
58 16 4405 4389 75.6666 81 0.0990196 0.211114
59 16 4482 4466 75.6891 77 0.109239 0.211034
2020-02-10 03:10:27.225386 min lat: 0.0667525 max lat: 0.560139 avg lat: 0.210973
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
60 16 4557 4541 75.6775 75 0.166502 0.210973
Total time run: 60.1343
Total writes made: 4558
Write size: 1048576
Object size: 1048576
Bandwidth (MB/sec): 75.797
Stddev Bandwidth: 6.00986
Max bandwidth (MB/sec): 86
Min bandwidth (MB/sec): 45
Average IOPS: 75
Stddev IOPS: 6.00986
Max IOPS: 86
Min IOPS: 45
Average Latency(s): 0.211022
Stddev Latency(s): 0.0626341
Max latency(s): 0.560139
Min latency(s): 0.0667525
root@pve01:~#
先ほど書き込んだデータを16スレッドでシーケンシャル読み込み
結果は160.485MB/s
root@pve01:~# rados bench -p Pool01 -t 16 60 seq
hints = 1
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 16 173 157 156.973 157 0.0574327 0.0939676
2 16 306 290 144.974 133 0.0604245 0.10393
3 16 463 447 148.977 157 0.0566775 0.104088
4 16 649 633 158.228 186 0.292006 0.098729
5 16 836 820 163.978 187 0.217264 0.0957541
6 16 996 980 163.313 160 0.0416325 0.0960492
7 16 1175 1159 165.55 179 0.28379 0.0950192
8 16 1326 1310 163.73 151 0.0836956 0.0967349
9 16 1481 1465 162.758 155 0.265245 0.0975261
10 16 1641 1625 162.481 160 0.0899419 0.097037
11 16 1788 1772 161.072 147 0.0630237 0.0985439
12 16 1950 1934 161.148 162 0.252735 0.0984893
13 16 2094 2078 159.827 144 0.00313608 0.0991515
14 16 2239 2223 158.767 145 0.353117 0.0998608
15 16 2402 2386 159.048 163 0.280573 0.0999204
16 16 2573 2557 159.794 171 0.104912 0.0994631
17 16 2741 2725 160.276 168 0.0891733 0.099362
18 16 2904 2888 160.426 163 0.146705 0.0991488
19 16 3067 3051 160.561 163 0.176214 0.0989968
2020-02-10 03:11:26.189370 min lat: 0.00282085 max lat: 0.441131 avg lat: 0.0994188
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 16 3220 3204 160.182 153 0.0122941 0.0994188
21 16 3389 3373 160.601 169 0.0552028 0.0990401
22 16 3548 3532 160.528 159 0.0534175 0.0991134
23 16 3717 3701 160.896 169 0.117185 0.0989803
24 16 3878 3862 160.899 161 0.144505 0.0990599
25 16 4048 4032 161.263 170 0.0781308 0.0988304
26 16 4203 4187 161.022 155 0.154898 0.0990267
27 16 4358 4342 160.798 155 0.112947 0.0990486
28 16 4516 4500 160.698 158 0.0431214 0.0991056
Total time run: 28.4014
Total reads made: 4558
Read size: 1048576
Object size: 1048576
Bandwidth (MB/sec): 160.485
Average IOPS: 160
Stddev IOPS: 11.9221
Max IOPS: 187
Min IOPS: 133
Average Latency(s): 0.0995795
Max latency(s): 0.441131
Min latency(s): 0.00282085
root@pve01:~#
ブロックサイズ4K＆16スレッドで60秒間書き込み
結果は1.36973MB/s 350IOPS
root@pve01:~# rados bench -p Pool01 -t 16 -b 4K 60 write --no-cleanup
hints = 1
Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 60 seconds or 0 objects
Object prefix: benchmark_data_pve01_18518
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
0 0 0 0 0 0 - 0
1 16 469 453 1.76934 1.76953 0.0285695 0.0341653
2 16 855 839 1.6385 1.50781 0.0886942 0.0380841
3 16 1243 1227 1.59751 1.51562 0.0225074 0.0389533
4 16 1563 1547 1.5106 1.25 0.0192213 0.0407375
5 16 1900 1884 1.47174 1.31641 0.018719 0.0423084
6 16 2266 2250 1.4647 1.42969 0.0383777 0.0424462
7 16 2617 2601 1.4513 1.37109 0.0199866 0.0427303
8 16 2994 2978 1.45395 1.47266 0.0167087 0.0429101
9 16 3350 3334 1.4469 1.39062 0.0226296 0.0430536
10 16 3707 3691 1.44165 1.39453 0.0387178 0.0432296
11 16 3998 3982 1.41391 1.13672 0.117072 0.044058
12 16 4348 4332 1.41 1.36719 0.015178 0.0442681
13 16 4727 4711 1.41541 1.48047 0.030594 0.0441222
14 16 5062 5046 1.40778 1.30859 0.018518 0.0443593
15 16 5403 5387 1.40272 1.33203 0.02676 0.0444248
16 16 5762 5746 1.40269 1.40234 0.0297216 0.0444263
17 16 6105 6089 1.39897 1.33984 0.0110362 0.044478
18 16 6461 6445 1.39851 1.39062 0.0190955 0.0446487
19 16 6827 6811 1.40014 1.42969 0.0473343 0.0444944
2020-02-10 03:12:40.498270 min lat: 0.0059555 max lat: 0.339018 avg lat: 0.0446279
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
20 16 7177 7161 1.39848 1.36719 0.0718719 0.0446279
21 16 7523 7507 1.39624 1.35156 0.102084 0.0446394
22 16 7867 7851 1.39385 1.34375 0.0422839 0.0447869
23 16 8192 8176 1.38843 1.26953 0.0371815 0.0449834
24 16 8509 8493 1.38217 1.23828 0.042311 0.0451412
25 16 8837 8821 1.37813 1.28125 0.0889875 0.0453092
26 16 9172 9156 1.37545 1.30859 0.0274299 0.0454203
27 16 9534 9518 1.37687 1.41406 0.0723538 0.0452903
28 16 9896 9880 1.3782 1.41406 0.106638 0.0453199
29 16 10270 10254 1.38104 1.46094 0.0149348 0.0452156
30 16 10666 10650 1.38656 1.54688 0.0541185 0.0450589
31 16 11049 11033 1.39009 1.49609 0.0383536 0.0449283
32 16 11405 11389 1.3901 1.39062 0.0455414 0.0449231
33 16 11761 11745 1.39011 1.39062 0.0313031 0.044905
34 16 12103 12087 1.38851 1.33594 0.0588919 0.044975
35 16 12468 12452 1.38957 1.42578 0.0220484 0.044962
36 16 12798 12782 1.38677 1.28906 0.12836 0.0450133
37 16 13147 13131 1.38613 1.36328 0.0262662 0.0450799
38 16 13498 13482 1.38573 1.37109 0.022454 0.0450754
39 16 13852 13836 1.38565 1.38281 0.03527 0.0450479
2020-02-10 03:13:00.500887 min lat: 0.0059555 max lat: 0.339018 avg lat: 0.045066
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
40 16 14208 14192 1.38577 1.39062 0.0833164 0.045066
41 16 14553 14537 1.38484 1.34766 0.082726 0.0450829
42 16 14905 14889 1.3846 1.375 0.0169701 0.0451043
43 16 15240 15224 1.38283 1.30859 0.0148114 0.0451582
44 16 15579 15563 1.38149 1.32422 0.0389806 0.0452232
45 16 15951 15935 1.38308 1.45312 0.0694952 0.0451314
46 16 16297 16281 1.38239 1.35156 0.0328218 0.0451822
47 16 16652 16636 1.38248 1.38672 0.0203276 0.0451807
48 16 16979 16963 1.38029 1.27734 0.134524 0.0452518
49 16 17307 17291 1.37827 1.28125 0.0752695 0.0453283
50 16 17678 17662 1.37968 1.44922 0.00985989 0.0452673
51 16 18021 18005 1.3789 1.33984 0.0225295 0.0453002
52 16 18359 18343 1.37777 1.32031 0.0133397 0.045347
53 16 18691 18675 1.37623 1.29688 0.0238378 0.0453851
54 16 19063 19047 1.37765 1.45312 0.0170903 0.0453498
55 16 19442 19426 1.37952 1.48047 0.0219286 0.0452925
56 16 19764 19748 1.37735 1.25781 0.0865308 0.0453386
57 16 20075 20059 1.37449 1.21484 0.126323 0.0454297
58 16 20419 20403 1.37396 1.34375 0.0221119 0.0454755
59 16 20736 20720 1.37165 1.23828 0.0220878 0.0455134
2020-02-10 03:13:20.503382 min lat: 0.00556043 max lat: 0.339018 avg lat: 0.0455463
sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)
60 16 21068 21052 1.37041 1.29688 0.0190516 0.0455463
Total time run: 60.0853
Total writes made: 21069
Write size: 4096
Object size: 4096
Bandwidth (MB/sec): 1.36973
Stddev Bandwidth: 0.0962262
Max bandwidth (MB/sec): 1.76953
Min bandwidth (MB/sec): 1.13672
Average IOPS: 350
Stddev IOPS: 24.6339
Max IOPS: 453
Min IOPS: 291
Average Latency(s): 0.0456012
Stddev Latency(s): 0.0322419
Max latency(s): 0.339018
Min latency(s): 0.00556043
root@pve01:~#
クリーンアップ
root@pve01:~# rados -p Pool01 cleanup Removed 21069 objects root@pve01:~#
rbd benchで計測
プール内に20GBでイメージを作成
root@pve01:~# rbd create --size=20G --pool=Pool01 Test01 root@pve01:~#
ブロックサイズ1M＆16スレッドで1024MBをシーケンシャル書き込み＆読み込みを計測
結果はWrite:19秒 約54MB/s・Read:6秒 約174MB/s
root@pve01:~# rbd bench Test01 --io-type write --io-size 1M --io-threads 16 --io-total 1024M --io-pattern seq --pool=Pool01
bench type write io_size 1048576 io_threads 16 bytes 1073741824 pattern sequential
SEC OPS OPS/SEC BYTES/SEC
1 80 78.69 82509278.47
2 112 63.62 66707302.71
3 160 58.20 61027034.77
4 224 57.75 60551804.62
5 272 56.65 59398890.68
6 320 49.87 52297028.40
7 384 51.17 53650691.23
8 432 52.39 54932025.24
9 480 50.55 53007541.22
10 528 50.04 52468762.55
11 576 50.71 53175552.61
12 624 50.85 53316377.80
13 672 49.92 52340537.36
14 736 51.82 54338095.34
15 784 53.11 55690911.07
16 832 51.82 54338095.49
17 896 53.12 55704507.66
18 944 53.75 56365035.50
19 1008 52.79 55358517.11
elapsed: 19 ops: 1024 ops/sec: 51.99 bytes/sec: 54514660.87
root@pve01:~# rbd bench Test01 --io-type read --io-size 1M --io-threads 16 --io-total 1024M --io-pattern seq --pool=Pool01
bench type read io_size 1048576 io_threads 16 bytes 1073741824 pattern sequential
SEC OPS OPS/SEC BYTES/SEC
1 160 191.30 200593221.87
2 320 173.91 182357474.58
3 496 172.50 180882885.92
4 656 171.95 180304320.03
5 832 173.48 181909788.83
6 1008 170.28 178549206.57
elapsed: 6 ops: 1024 ops/sec: 166.34 bytes/sec: 174418585.64
root@pve01:~#
ブロックサイズ4K＆16スレッドで512MBをランダム書き込み＆読み込みを計測
結果はWrite:263秒 約2.03MB/s・Read:8秒 約62MB/s
root@pve01:~# rbd bench Test01 --io-type write --io-size 4K --io-threads 16 --io-total 512M --io-pattern rand --pool=Pool01
bench type write io_size 4096 io_threads 16 bytes 536870912 pattern random
SEC OPS OPS/SEC BYTES/SEC
1 5344 5296.34 21693805.32
2 5408 2690.42 11019975.14
3 5472 1802.86 7384496.97
4 5552 1386.43 5678805.38
5 5632 1108.30 4539591.05
6 5712 72.96 298830.01
7 5792 75.29 308398.68
8 5872 77.94 319245.52
9 5920 72.84 298356.80
10 6016 75.77 310345.95
11 6112 77.16 316043.21
12 6192 78.93 323277.03
13 6256 79.01 323628.25
14 6336 83.00 339964.63
15 6432 80.81 330983.44
54 6480 8.53 34952.66
55 6496 7.01 28706.19
56 6512 5.92 24238.45
57 10832 104.48 427943.32
58 10976 106.13 434694.07
59 11104 964.92 3952323.75
60 11248 1056.92 4329139.95
61 11376 1024.41 4195989.11
62 11552 144.34 591231.48
63 11712 148.86 609748.65
64 11840 151.44 620287.52
65 11984 146.96 601956.34
66 12128 150.64 617013.64
67 12272 144.81 593134.08
68 12416 138.14 565841.43
69 12544 140.46 575324.80
70 12736 149.20 611137.32
71 12880 151.49 620493.98
72 13024 150.04 614551.50
73 13216 163.13 668176.24
74 13360 162.03 663675.74
75 13552 164.91 675479.19
76 13728 167.98 688062.69
77 13888 175.89 720455.05
78 14032 165.05 676025.69
79 14208 170.96 700270.26
80 14416 173.77 711760.91
81 14624 178.91 732816.43
82 14864 193.49 792549.23
83 15120 214.85 880008.18
84 15344 227.56 932084.45
85 15536 224.71 920431.56
86 15760 229.77 941133.82
87 16032 232.85 953755.00
88 16272 233.38 955936.05
89 16528 237.17 971468.31
90 16800 252.59 1034620.99
91 17040 256.20 1049395.15
92 17360 263.49 1079242.53
93 17696 285.94 1171202.96
94 18064 307.19 1258266.77
95 18368 313.59 1284480.66
96 18656 322.42 1320631.98
97 18960 317.71 1301325.02
98 19296 319.74 1309646.84
107 19392 97.27 398431.18
108 19440 82.41 337548.19
109 23184 348.41 1427079.73
111 23200 295.18 1209043.66
113 23216 257.75 1055760.69
114 26432 1097.92 4497080.60
117 26496 709.56 2906357.16
118 26512 369.77 1514580.41
119 29296 811.92 3325616.93
121 29328 721.76 2956337.59
122 31696 650.83 2665792.04
124 31792 781.11 3199409.38
125 33888 1035.93 4243189.68
127 34000 538.70 2206506.03
128 35888 1037.95 4251461.92
129 36000 568.40 2328161.97
130 36016 756.97 3100567.89
131 37728 649.51 2660408.54
132 37776 773.12 3166706.88
133 39312 669.26 2741288.57
134 39584 835.03 3420265.54
135 39968 845.87 3464700.02
136 41248 703.42 2881223.18
137 41520 704.28 2884711.63
138 41536 453.13 1856016.22
139 42944 668.24 2737130.77
140 42976 598.24 2450383.74
141 44320 616.86 2526640.24
142 45392 948.07 3883301.90
143 45600 722.60 2959783.39
144 46720 758.22 3105661.97
145 46848 727.81 2981091.17
146 48000 732.47 3000196.71
147 48064 536.54 2197650.64
148 49232 727.55 2980044.89
149 50192 698.30 2860224.62
150 50432 729.63 2988554.55
151 51520 691.27 2831429.90
152 52352 852.81 3493100.27
153 52768 799.26 3273773.91
154 53872 734.22 3007379.97
155 54784 915.81 3751146.29
156 55568 826.11 3383732.10
157 56384 799.98 3276736.62
158 57232 881.50 3610622.11
159 57952 806.95 3305252.53
160 58640 768.73 3148698.27
161 59408 766.76 3140642.14
162 60112 754.64 3091010.67
163 60864 745.47 3053445.06
164 61616 739.89 3030584.37
165 62336 739.78 3030128.76
166 63072 739.29 3028138.39
167 63856 745.21 3052374.11
168 64656 740.61 3033541.35
169 65312 738.59 3025284.44
170 65984 727.26 2978851.50
171 66608 683.67 2800304.72
172 67312 694.52 2844755.30
173 68208 707.56 2898149.56
174 68912 719.99 2949063.01
175 69648 724.67 2968246.44
176 70416 781.59 3201409.41
177 71136 766.01 3137580.40
178 71952 758.49 3106792.49
179 72736 763.56 3127556.19
180 73408 766.08 3137869.29
181 74192 751.58 3078462.83
182 75008 770.69 3156728.83
183 75760 767.73 3144610.23
184 76384 729.00 2985995.08
185 77280 774.39 3171881.14
186 78128 798.69 3271417.34
187 78864 774.28 3171460.04
188 79776 799.98 3276736.73
189 80496 820.42 3360420.35
190 80752 672.33 2753880.21
191 81760 702.77 2878557.43
192 82592 742.02 3039329.86
193 83488 750.79 3075254.55
194 84336 764.32 3130640.20
195 85232 921.79 3775663.74
196 85968 867.25 3552274.39
197 86896 866.33 3548479.02
198 87680 838.38 3434020.12
199 88592 841.76 3447832.19
200 89328 823.80 3374274.38
201 90160 831.73 3406766.01
202 91008 809.43 3315438.39
203 91728 806.36 3302846.23
204 92560 815.10 3338663.61
205 93424 809.47 3315591.37
206 94176 807.71 3308370.61
207 94976 791.68 3242740.32
208 95840 811.99 3325913.73
209 96592 798.08 3268953.57
210 97424 802.55 3287256.02
211 98320 826.14 3383870.94
212 99088 831.03 3403885.85
213 99904 822.66 3369599.80
214 100704 824.36 3376589.26
215 101488 819.34 3356012.71
216 102288 796.77 3263577.23
217 103152 820.00 3358721.36
218 103984 806.31 3302639.88
219 104816 830.36 3401136.37
220 105696 839.57 3438873.85
221 106528 841.25 3445774.85
222 107392 847.98 3473341.05
223 108144 844.83 3460404.46
224 108992 829.87 3399158.65
225 109824 823.61 3373496.04
226 110528 811.01 3321914.89
227 111392 789.87 3235324.70
228 112240 814.62 3336693.06
229 112752 755.01 3092502.66
230 113632 764.03 3129467.79
231 114512 796.15 3261021.10
232 115408 784.97 3215250.02
233 116240 805.79 3300500.48
234 116976 842.09 3449196.71
235 117904 855.07 3502356.87
236 118688 836.52 3426395.54
237 119616 844.96 3460971.09
238 120304 808.90 3313261.02
239 121056 820.58 3361093.75
240 122000 821.15 3363450.86
241 122720 781.99 3203016.67
242 123360 765.32 3134736.04
243 124096 758.39 3106346.58
244 125088 801.89 3284557.28
245 125920 780.24 3195862.03
246 126624 805.27 3298365.54
247 127520 840.73 3443620.04
248 128336 845.28 3462261.90
249 128928 728.92 2985637.29
250 129360 685.25 2806766.69
251 130176 709.82 2907416.46
252 131040 691.27 2831430.18
elapsed: 263 ops: 131072 ops/sec: 497.24 bytes/sec: 2036710.18
root@pve01:~# rbd bench Test01 --io-type read --io-size 4K --io-threads 16 --io-total 512M --io-pattern rand --pool=Pool01
bench type read io_size 4096 io_threads 16 bytes 536870912 pattern random
SEC OPS OPS/SEC BYTES/SEC
1 14720 14794.90 60599900.69
2 30272 15174.06 62152943.04
3 41664 13911.62 56981980.51
4 57760 14458.18 59220712.78
5 74512 14905.31 61052169.50
6 91168 15289.31 62625003.46
7 106512 15247.71 62454613.16
8 121744 16015.69 65600281.00
elapsed: 8 ops: 131072 ops/sec: 15247.73 bytes/sec: 62454704.64
root@pve01:~#
イメージ削除
root@pve01:~# rbd rm --pool=Pool01 Test01 Removing image: 100% complete...done. root@pve01:~#
計測後のメモリ使用状況
5.6GiBぐらいでPG:128に比べて特に変化なし
まとめ
まず5400rpmのHDDを追加したパターンについてです
そもそもCephは同期書き込みな仕様上、一番低速なデバイスに引っ張られやすく
HDDとSSDの混在は推奨されていません
ただ今回使っているST1000DM010とWD40EZRZは性能だけで見れば
全ての項目でST1000DM010の方が高性能というわけではなく（特に4K周り）
単純にOSD増加の速度アップ分も期待して案外大丈夫かなと思いきや
特に書き込み速度がかなり落ちてしまいました
理由は書き込みベンチマーク中のOSD画面を見れば
一番右の「Apply/Commit Latency(ms)」の項目で3.64TB表記のWD40EZRZだけ
桁違いな数値が出ているのでレイテンシ問題だと思われます
ST1000DM010だけで構成すればレイテンシは150msぐらいは記録することから
今回の構成ではWD40EZRZの反応が遅くボトルネックになってしまっていて
ST1000DM010に負荷がかかりきっていないことが予想されます
Cephは10GbEケーブルすら光ファイバー推奨なぐらいレイテンシが重要な要素なので
単純性能に加えて回転数の影響から発生するレイテンシを気にする必要がある様子
ただこの程度の落ち具合で大容量化が可能という見方もできると思います
次にpg_numを512に変更したパターンについてです
書き込み速度が若干低下してしまってますが
それよりも明らかな読み込み速度アップが確認できました
特にCentOS7での計測時がわかりやすく
pg_num:128の時は3～12OSD全てのケースでSequential Read: 148MB/sになっていて
明らかに頭打ちしている様子でしたが
pg_num:512にするだけでSequential Read: 167MB/sに上がっています
Ceph公式によると基本的にはOSD数を参考に設定すればいいものの
細かくいうとストレージ使用率によって最適なPG値が変化するらしく
使用率が少ない場合はOSDが多くてもPGは少なくする方がいいと言われているので
ベンチマークVMしかないような使用率1%程度の環境では512は多すぎたかもしれません
この結果だけ見るとPG値が不足時は読み込み速度に、多すぎると書き込み速度に
反映されやすい様子なのでどちらを優先するかにもよると思います
コメント