Proxmox VE 6.1 + Cephで速度計測 各ノードHDD2~3台の記録
最初に
前記事のCephを拡張して各ノードHDD2~3台のパターンを計測します
計測環境
HDD1台で検証した前記事に詳細を掲載してますのでそちらをご覧ください
Proxmox VE 6.1 + Cephで速度計測:HDDx1 | fefcc.net
https://fefcc.net/archives/1040
今回のCeph構成
各ノードHDD2台の構成はHDD ST1000DM010 1TB 7200rpmを2台をOSD登録
各ノードHDD3台の構成はHDD ST1000DM010 1TB 7200rpmを3台をOSD登録
DB/WALともにOSDと同じHDDを指定
プールは各ノードHDD1台の構成を拡張して使用
Pool01という名でCeph/RBD形式のプール作成
Pool01のPG値は128のまま
既存のCephにOSD追加して6OSDへ
Cephは既存のプールを登録したままOSD追加が可能なので簡単に容量追加ができます
OSD追加の処理中でも(速度は低下しますが)Cephは利用可能なので
プール内のVMデータが維持されるのはもちろんのことVMを停止する必要もありません
OSD追加前の様子がこちら
各ノードHDD1台で合計3OSD
追加したいノードでCeph -> OSD画面へ移動して「作成: OSD」を選択
今回はpve01から追加していきます
Ceph状態を確認するとすぐにリマップ処理が行われているのがわかります
1THDDを1台追加した場合の使用状況推移がこちら
3OSD:948.35GB→4OSD:1.25TBで約300GB増加
このように必ずしも全ノードが同数のOSDでなくても動作はしますが
一部のデータが保存されていないノードが出てくるのであまり推奨はされていません
使用状況のTotal Sizeは1.89TB(1.72TiB)になり
6OSDなのでクラスタ全体のPG値は250×6=1500になります
各HDD2台:計6OSDでの計測結果
Proxmox起動直後のメモリ使用状況
pve01は1.74GiB、pve01は1.73GiB、pve03は1.59GiB
3OSD構成と比較しても誤差レベルでした
CentOS7のVMから計測
[root@centos7-bench ~]# ./fio_cdm.sh /root Testing in /root Configuration: Size:1024m Loops:5 Write Only Zeroes:0 Running Benchmark, please wait... Results: Sequential Read: 148MB/s IOPS=0 Sequential Write: 25MB/s IOPS=0 512KB Read: 73MB/s IOPS=147 512KB Write: 5MB/s IOPS=11 Sequential Q32T1 Read: 142MB/s IOPS=4 Sequential Q32T1 Write: 23MB/s IOPS=0 4KB Read: 4MB/s IOPS=1250 4KB Write: 0MB/s IOPS=74 4KB Q32T1 Read: 4MB/s IOPS=1250 4KB Q32T1 Write: 0MB/s IOPS=80 4KB Q8T8 Read: 37MB/s IOPS=9500 4KB Q8T8 Write: 1MB/s IOPS=264 [root@centos7-bench ~]#
Windows10のVMから計測
------------------------------------------------------------------------------ CrystalDiskMark 7.0.0 x64 (C) 2007-2019 hiyohiyo Crystal Dew World: https://crystalmark.info/ ------------------------------------------------------------------------------ * MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s] * KB = 1000 bytes, KiB = 1024 bytes [Read] Sequential 1MiB (Q= 8, T= 1): 159.601 MB/s [ 152.2 IOPS] < 51580.54 us> Sequential 1MiB (Q= 1, T= 1): 135.920 MB/s [ 129.6 IOPS] < 7709.83 us> Random 4KiB (Q= 32, T=16): 125.471 MB/s [ 30632.6 IOPS] < 16587.69 us> Random 4KiB (Q= 1, T= 1): 7.120 MB/s [ 1738.3 IOPS] < 574.43 us> [Write] Sequential 1MiB (Q= 8, T= 1): 25.794 MB/s [ 24.6 IOPS] <311662.26 us> Sequential 1MiB (Q= 1, T= 1): 7.760 MB/s [ 7.4 IOPS] <133764.91 us> Random 4KiB (Q= 32, T=16): 1.066 MB/s [ 260.3 IOPS] <545741.70 us> Random 4KiB (Q= 1, T= 1): 0.158 MB/s [ 38.6 IOPS] < 25655.59 us> [Mix] Read 70%/Write 30% Sequential 1MiB (Q= 8, T= 1): 50.753 MB/s [ 48.4 IOPS] <163766.61 us> Sequential 1MiB (Q= 1, T= 1): 21.810 MB/s [ 20.8 IOPS] < 48025.08 us> Random 4KiB (Q= 32, T=16): 2.782 MB/s [ 679.2 IOPS] <272531.02 us> Random 4KiB (Q= 1, T= 1): 0.429 MB/s [ 104.7 IOPS] < 9196.32 us> Profile: Default Test: 1 GiB (x5) [Interval: 5 sec] <DefaultAffinity=DISABLED> Date: 2020/02/09 21:21:02 OS: Windows 10 Professional [10.0 Build 18363] (x64)
rados benchで計測
ブロックサイズ1M&16スレッドで60秒間書き込み
結果は71.73MB/s
root@pve01:~# rados bench -p Pool01 -t 16 -b 1M 60 write --no-cleanup hints = 1 Maintaining 16 concurrent writes of 1048576 bytes to objects of size 1048576 for up to 60 seconds or 0 objects Object prefix: benchmark_data_pve01_16205 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 70 54 53.9973 54 0.166681 0.259034 2 16 142 126 62.9954 72 0.238393 0.238169 3 16 223 207 68.9943 81 0.200345 0.224065 4 16 294 278 69.4939 71 0.253345 0.223467 5 16 365 349 69.7937 71 0.216933 0.223317 6 16 447 431 71.8271 82 0.234301 0.220181 7 16 515 499 71.2795 68 0.198107 0.220523 8 16 580 564 70.494 65 0.133414 0.222649 9 16 656 640 71.1051 76 0.201653 0.223094 10 16 717 701 70.0936 61 0.273147 0.223451 11 16 796 780 70.9026 79 0.20506 0.223114 12 16 872 856 71.3267 76 0.150906 0.221818 13 16 950 934 71.8395 78 0.127223 0.22168 14 16 1021 1005 71.7789 71 0.299701 0.220322 15 16 1091 1075 71.66 70 0.19942 0.221493 16 16 1159 1143 71.4308 68 0.188427 0.221625 17 16 1235 1219 71.6991 76 0.165404 0.221815 18 16 1309 1293 71.8266 74 0.275926 0.221312 19 16 1380 1364 71.7827 71 0.181099 0.22155 2020-02-09 21:28:38.358066 min lat: 0.0665327 max lat: 0.42292 avg lat: 0.22127 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 20 16 1449 1433 71.6433 69 0.234812 0.22127 21 16 1525 1509 71.8504 76 0.219319 0.221361 22 16 1594 1578 71.7206 69 0.15018 0.22164 23 16 1663 1647 71.602 69 0.158402 0.221779 24 16 1731 1715 71.4514 68 0.226752 0.222457 25 16 1798 1782 71.2731 67 0.298069 0.223112 26 16 1874 1858 71.4544 76 0.212011 0.222688 27 16 1954 1938 71.7706 80 0.138644 0.222001 28 16 2019 2003 71.5285 65 0.17797 0.221776 29 16 2088 2072 71.4411 69 0.244765 0.222723 30 16 2169 2153 71.7594 81 0.215854 0.222631 31 16 2233 2217 71.5089 64 0.165947 0.22262 32 16 2294 2278 71.1803 61 0.127047 0.223701 33 16 2369 2353 71.2956 75 0.242882 0.223626 34 16 2442 2426 71.3455 73 0.210134 0.223278 35 16 2515 2499 71.3925 73 0.243776 0.223325 36 16 2591 2575 71.5203 76 0.186243 0.223044 37 16 2660 2644 71.4518 69 0.165274 0.22293 38 16 2734 2718 71.5185 74 0.255108 0.222551 39 16 2795 2779 71.2486 61 0.260296 0.223336 2020-02-09 21:28:58.360671 min lat: 0.0620772 max lat: 0.481738 avg lat: 0.224004 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 40 16 2864 2848 71.192 69 0.21892 0.224004 41 16 2941 2925 71.3335 77 0.179598 0.223554 42 16 3012 2996 71.3254 71 0.221595 0.223573 43 16 3094 3078 71.5734 82 0.165702 0.223045 44 16 3166 3150 71.583 72 0.130623 0.223021 45 16 3238 3222 71.5921 72 0.166878 0.223004 46 16 3313 3297 71.666 75 0.242924 0.222809 47 16 3384 3368 71.6516 71 0.121162 0.222541 48 16 3457 3441 71.6795 73 0.0991777 0.222378 49 16 3536 3520 71.8288 79 0.206459 0.22237 50 16 3605 3589 71.7721 69 0.231927 0.222256 51 16 3679 3663 71.8157 74 0.22896 0.22226 52 16 3751 3735 71.8191 72 0.153848 0.222213 53 16 3823 3807 71.8224 72 0.24974 0.222416 54 16 3891 3875 71.7514 68 0.156326 0.222408 55 16 3965 3949 71.7922 74 0.319579 0.222451 56 16 4033 4017 71.7243 68 0.147869 0.222437 57 16 4104 4088 71.7116 71 0.302004 0.222709 58 16 4172 4156 71.6474 68 0.138367 0.222872 59 16 4245 4229 71.6701 73 0.179308 0.222916 2020-02-09 21:29:18.362748 min lat: 0.0620772 max lat: 0.485794 avg lat: 0.222904 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 60 16 4315 4299 71.6422 70 0.226698 0.222904 Total time run: 60.1701 Total writes made: 4316 Write size: 1048576 Object size: 1048576 Bandwidth (MB/sec): 71.73 Stddev Bandwidth: 5.46119 Max bandwidth (MB/sec): 82 Min bandwidth (MB/sec): 54 Average IOPS: 71 Stddev IOPS: 5.46119 Max IOPS: 82 Min IOPS: 54 Average Latency(s): 0.223058 Stddev Latency(s): 0.0603301 Max latency(s): 0.485794 Min latency(s): 0.0620772 root@pve01:~#
先ほど書き込んだデータを16スレッドでシーケンシャル読み込み
結果は152.708MB/s
root@pve01:~# rados bench -p Pool01 -t 16 60 seq hints = 1 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 163 147 146.978 147 0.00664183 0.093832 2 16 326 310 154.98 163 0.0751217 0.0971496 3 16 492 476 158.648 166 0.0144741 0.0982731 4 16 635 619 154.732 143 0.0758463 0.100723 5 16 794 778 155.583 159 0.0856299 0.101027 6 16 949 933 155.483 155 0.0483979 0.100902 7 16 1100 1084 154.841 151 0.0159092 0.101981 8 16 1260 1244 155.484 160 0.0147741 0.101862 9 16 1404 1388 154.207 144 0.0591117 0.10216 10 16 1561 1545 154.485 157 0.148539 0.102394 11 16 1714 1698 154.349 153 0.00348469 0.102949 12 16 1872 1856 154.652 158 0.115412 0.102751 13 16 2025 2009 154.524 153 0.10636 0.10263 14 16 2172 2156 153.985 147 0.0199766 0.103172 15 16 2333 2317 154.452 161 0.286223 0.102805 16 16 2484 2468 154.236 151 0.16923 0.103173 17 16 2629 2613 153.691 145 0.104399 0.103422 18 16 2796 2780 154.43 167 0.0513866 0.102821 19 16 2940 2924 153.88 144 0.145655 0.103415 2020-02-09 21:30:25.276064 min lat: 0.00274203 max lat: 0.427698 avg lat: 0.103573 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 20 16 3093 3077 153.835 153 0.170269 0.103573 21 16 3230 3214 153.033 137 0.114728 0.103968 22 16 3384 3368 153.076 154 0.0599532 0.103977 23 16 3528 3512 152.681 144 0.0873498 0.104282 24 16 3680 3664 152.652 152 0.0119661 0.104372 25 16 3816 3800 151.986 136 0.0446538 0.104877 26 16 3978 3962 152.37 162 0.329855 0.104561 27 16 4144 4128 152.874 166 0.0559258 0.104169 28 16 4299 4283 152.95 155 0.128285 0.10428 Total time run: 28.263 Total reads made: 4316 Read size: 1048576 Object size: 1048576 Bandwidth (MB/sec): 152.708 Average IOPS: 152 Stddev IOPS: 8.49175 Max IOPS: 167 Min IOPS: 136 Average Latency(s): 0.104669 Max latency(s): 0.427698 Min latency(s): 0.00272696 root@pve01:~#
ブロックサイズ4K&16スレッドで60秒間書き込み
結果は1.1907MB/s 304IOPS
root@pve01:~# rados bench -p Pool01 -t 16 -b 4K 60 write --no-cleanup hints = 1 Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 60 seconds or 0 objects Object prefix: benchmark_data_pve01_17174 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 445 429 1.67545 1.67578 0.0500281 0.0356037 2 16 773 757 1.4783 1.28125 0.0172264 0.0415868 3 16 1049 1033 1.34484 1.07812 0.0148049 0.0432157 4 16 1371 1355 1.32305 1.25781 0.0159175 0.0470511 5 16 1733 1717 1.34123 1.41406 0.051864 0.0464846 6 16 1995 1979 1.28822 1.02344 0.0101456 0.0465766 7 16 2310 2294 1.27994 1.23047 0.0157812 0.0486082 8 16 2584 2568 1.25371 1.07031 0.0164025 0.0496671 9 16 2914 2898 1.25761 1.28906 0.0165869 0.0496375 10 16 3198 3182 1.24278 1.10938 0.122408 0.0500057 11 16 3462 3446 1.22354 1.03125 0.0477646 0.0509933 12 16 3678 3662 1.19188 0.84375 0.0167316 0.0515104 13 16 4008 3992 1.19934 1.28906 0.0737472 0.0520085 14 16 4292 4276 1.19291 1.10938 0.0420287 0.0521601 15 16 4638 4622 1.20346 1.35156 0.0470655 0.0515758 16 16 4956 4940 1.20587 1.24219 0.0160688 0.0515776 17 16 5267 5251 1.20639 1.21484 0.0327591 0.0517119 18 16 5606 5590 1.21292 1.32422 0.0395547 0.051423 19 16 5940 5924 1.21774 1.30469 0.0676541 0.0511385 2020-02-09 21:32:20.363004 min lat: 0.00771718 max lat: 0.551061 avg lat: 0.0510562 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 20 16 6281 6265 1.22345 1.33203 0.0166202 0.0510562 21 16 6634 6618 1.23085 1.37891 0.0587667 0.0506805 22 16 6943 6927 1.22976 1.20703 0.0161021 0.0507724 23 16 7249 7233 1.22826 1.19531 0.0184894 0.050805 24 16 7539 7523 1.22427 1.13281 0.0198849 0.0509466 25 16 7840 7824 1.22232 1.17578 0.0415983 0.0510539 26 16 8130 8114 1.21888 1.13281 0.148898 0.0512602 27 16 8453 8437 1.22046 1.26172 0.0927031 0.0511313 28 16 8713 8697 1.21314 1.01562 0.266455 0.0512155 29 16 9054 9038 1.21723 1.33203 0.0568609 0.051261 30 16 9395 9379 1.22105 1.33203 0.0697839 0.0510779 31 16 9736 9720 1.22463 1.33203 0.0162915 0.0509812 32 16 9958 9942 1.21346 0.867188 0.0270542 0.0514915 33 16 10245 10229 1.21065 1.12109 0.033327 0.0515395 34 16 10519 10503 1.20652 1.07031 0.176737 0.0516079 35 16 10806 10790 1.20407 1.12109 0.0171315 0.0518185 36 16 11092 11076 1.20165 1.11719 0.0166156 0.0519688 37 16 11359 11343 1.19736 1.04297 0.0789557 0.0521708 38 16 11654 11638 1.19617 1.15234 0.0699342 0.0521939 39 16 11947 11931 1.19484 1.14453 0.0357696 0.0522795 2020-02-09 21:32:40.365527 min lat: 0.00610495 max lat: 0.551061 avg lat: 0.0525676 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 40 16 12182 12166 1.18792 0.917969 0.0169527 0.0525676 41 16 12499 12483 1.18914 1.23828 0.00873827 0.0524147 42 16 12823 12807 1.19096 1.26562 0.0712804 0.0523685 43 16 13111 13095 1.18942 1.125 0.191711 0.0525092 44 16 13388 13372 1.18698 1.08203 0.0166987 0.0526447 45 16 13725 13709 1.18985 1.31641 0.0163342 0.0524994 46 16 14051 14035 1.19166 1.27344 0.0557663 0.052392 47 16 14267 14251 1.18426 0.84375 0.0795275 0.0527226 48 16 14576 14560 1.18473 1.20703 0.0201785 0.0527235 49 16 14882 14866 1.18494 1.19531 0.0422899 0.052728 50 16 15182 15166 1.18468 1.17188 0.0279977 0.05267 51 16 15520 15504 1.18733 1.32031 0.0687027 0.0525769 52 16 15859 15843 1.18996 1.32422 0.0155929 0.0525014 53 16 16150 16134 1.18895 1.13672 0.161741 0.0525362 54 16 16405 16389 1.18538 0.996094 0.0723689 0.0526843 55 16 16692 16676 1.18421 1.12109 0.0265968 0.0527625 56 16 17029 17013 1.18657 1.31641 0.120635 0.0526498 57 16 17321 17305 1.18576 1.14062 0.0171969 0.0526781 58 16 17586 17570 1.18316 1.03516 0.017574 0.0527803 59 16 17947 17931 1.187 1.41016 0.0264296 0.0526476 2020-02-09 21:33:00.368241 min lat: 0.00610495 max lat: 0.551061 avg lat: 0.052446 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 60 16 18316 18300 1.19124 1.44141 0.0312203 0.052446 Total time run: 60.0879 Total writes made: 18316 Write size: 4096 Object size: 4096 Bandwidth (MB/sec): 1.1907 Stddev Bandwidth: 0.153304 Max bandwidth (MB/sec): 1.67578 Min bandwidth (MB/sec): 0.84375 Average IOPS: 304 Stddev IOPS: 39.2459 Max IOPS: 429 Min IOPS: 216 Average Latency(s): 0.0524896 Stddev Latency(s): 0.047584 Max latency(s): 0.551061 Min latency(s): 0.00610495 root@pve01:~#
クリーンアップ
root@pve01:~# rados -p Pool01 cleanup Removed 18316 objects root@pve01:~#
rbd benchで計測
プール内に20GBでイメージを作成
root@pve01:~# rbd create --size=20G --pool=Pool01 Test01 root@pve01:~#
ブロックサイズ1M&16スレッドで1024MBをシーケンシャル書き込み&読み込みを計測
結果はWrite:19秒 約55MB/s・Read:6秒 約157MB/s
root@pve01:~# rbd bench Test01 --io-type write --io-size 1M --io-threads 16 --io-total 1024M --io-pattern seq --pool=Pool01 bench type write io_size 1048576 io_threads 16 bytes 1073741824 pattern sequential SEC OPS OPS/SEC BYTES/SEC 1 80 80.53 84446717.75 2 128 65.57 68757163.62 3 176 62.66 65705279.06 4 240 60.43 63368264.08 5 288 60.08 62995693.74 6 336 52.24 54781217.64 7 384 52.29 54825973.49 8 432 51.53 54031394.13 9 496 52.20 54736534.64 10 544 51.45 53944526.87 11 608 53.42 56010352.18 12 656 55.28 57968437.65 13 704 51.32 53812209.99 14 736 48.08 50410900.88 15 784 48.23 50572993.79 16 848 47.69 50010178.23 17 896 46.73 48997511.04 18 944 51.41 53909857.88 19 1008 54.22 56858993.61 elapsed: 19 ops: 1024 ops/sec: 52.68 bytes/sec: 55243458.60 root@pve01:~# rbd bench Test01 --io-type read --io-size 1M --io-threads 16 --io-total 1024M --io-pattern seq --pool=Pool01 bench type read io_size 1048576 io_threads 16 bytes 1073741824 pattern sequential SEC OPS OPS/SEC BYTES/SEC 1 160 181.81 190644857.94 2 320 168.33 176508866.43 3 496 172.97 181370243.16 4 656 169.01 177219163.60 5 784 159.74 167499475.68 6 912 150.52 157827688.40 elapsed: 6 ops: 1024 ops/sec: 150.41 bytes/sec: 157713254.77 root@pve01:~#
ブロックサイズ4K&16スレッドで512MBをランダム書き込み&読み込みを計測
結果はWrite:280秒 約1.9MB/s・Read:8秒 約60MB/s
root@pve01:~# rbd bench Test01 --io-type write --io-size 4K --io-threads 16 --io-total 512M --io-pattern rand --pool=Pool01 bench type write io_size 4096 io_threads 16 bytes 536870912 pattern random SEC OPS OPS/SEC BYTES/SEC 7 5920 811.35 3323288.05 8 5968 746.49 3057607.02 9 6080 649.04 2658487.76 10 6112 611.07 2502950.27 11 6176 552.84 2264439.46 50 6240 7.46 30540.67 51 6256 6.65 27245.36 52 6272 4.49 18382.65 53 6304 4.40 18040.24 54 6336 3.74 15308.86 55 10752 921.54 3774639.35 56 10864 977.90 4005483.71 57 10976 971.87 3980794.26 58 11072 1077.73 4414371.80 59 11248 973.80 3988697.13 60 11376 125.80 515288.79 61 11520 128.32 525606.56 62 11632 126.15 516711.68 63 11744 135.26 554032.79 64 11920 132.59 543100.80 65 12064 139.27 570439.08 66 12176 134.53 551046.09 67 12352 147.54 604310.94 68 12528 153.96 630631.19 69 12672 152.97 626547.13 70 12848 156.67 641721.43 71 12976 159.49 653251.23 72 13184 168.28 689275.18 73 13360 164.81 675074.58 74 13536 169.01 692262.29 75 13664 164.38 673296.20 76 13840 167.83 687421.30 77 13984 158.35 648596.23 78 14128 157.63 645656.78 79 14304 153.60 629127.97 80 14480 162.67 666316.25 81 14704 177.77 728157.37 82 14960 198.37 812517.07 83 15184 202.60 829864.69 84 15328 208.63 854561.22 85 15520 208.33 853309.42 86 15744 207.83 851263.11 87 15904 189.25 775163.52 88 16128 197.65 809571.95 89 16416 219.35 898452.24 90 16688 233.03 954507.96 91 16944 240.19 983799.49 92 17264 270.69 1108758.73 93 17584 285.71 1170252.91 94 17856 287.53 1177730.57 95 18160 293.92 1203902.35 96 18432 291.76 1195034.74 97 18608 269.87 1105395.52 98 18880 262.13 1073678.64 99 19232 276.08 1130806.18 110 19440 84.68 346833.36 111 23152 315.92 1294014.59 116 23248 241.71 990045.17 117 26464 400.24 1639391.65 120 26544 336.21 1377097.31 121 29232 901.63 3693085.40 123 29472 490.82 2010406.50 124 29488 747.46 3061610.36 125 31904 673.92 2760358.67 127 32000 808.51 3311670.02 128 34160 687.86 2817493.29 130 34288 714.52 2926674.01 131 36240 1039.38 4257299.33 133 36416 532.06 2179320.01 134 38272 964.30 3949783.45 136 38416 531.72 2177921.93 137 40256 945.47 3872661.98 138 40416 596.55 2443488.04 139 41824 990.45 4056876.69 141 42224 532.89 2182704.69 142 43840 923.68 3783392.52 143 43936 533.01 2183206.43 144 45248 803.17 3289780.06 146 45552 522.99 2142180.09 147 47056 847.10 3469725.95 148 47072 515.95 2113330.25 149 48496 896.91 3673728.58 150 48624 536.88 2199063.00 151 50096 920.56 3770604.11 152 50224 595.02 2437220.89 153 51664 972.85 3984810.25 154 51856 596.57 2443567.81 155 53120 966.44 3958533.83 156 53488 608.74 2493403.13 157 54848 1001.71 4102983.68 158 55152 608.07 2490663.74 159 56416 1018.74 4172751.57 160 56752 630.98 2584477.88 161 57824 984.53 4032642.54 163 58352 588.69 2411287.19 164 59904 905.46 3708773.99 165 59920 577.82 2366751.54 166 61408 881.79 3611825.94 167 61440 590.06 2416893.02 168 62880 887.82 3636503.46 169 62912 579.34 2372962.70 170 64320 900.14 3686966.93 171 64368 559.32 2290971.24 172 65776 849.51 3479576.35 173 65792 568.29 2327716.11 174 67120 871.56 3569901.49 175 67200 552.56 2263267.27 176 68528 876.87 3591669.87 177 68848 637.33 2610489.40 178 70176 923.31 3781878.71 179 70896 757.60 3103142.46 180 71760 947.60 3881387.34 181 72528 801.90 3284591.07 182 73344 885.01 3625019.53 183 74064 776.96 3182414.29 184 74864 792.31 3245301.90 185 75600 779.20 3191598.75 186 76272 753.00 3084270.64 187 77072 762.04 3121312.24 188 77792 725.27 2970711.96 189 77952 594.29 2434198.04 190 78944 669.85 2743717.87 191 79664 678.92 2780873.12 192 80288 649.42 2660009.35 193 81040 667.74 2735081.13 194 81616 767.15 3142236.77 195 82368 684.23 2802619.98 196 83104 686.88 2813467.36 197 83712 686.43 2811609.94 198 84432 675.68 2767578.10 199 85056 687.43 2815716.33 200 85808 680.90 2788963.68 201 86608 689.20 2822970.33 202 87248 698.79 2862262.75 203 88016 700.53 2869361.21 204 88544 695.91 2850448.32 205 89232 691.42 2832049.75 206 89872 662.86 2715062.70 207 90592 660.33 2704707.65 208 91120 636.05 2605251.46 209 91840 641.73 2628511.89 210 92480 645.96 2645870.04 211 93168 644.24 2638787.54 212 93792 657.34 2692447.02 213 94224 615.37 2520544.63 214 94880 626.53 2566259.37 215 95600 624.98 2559928.14 216 96224 629.31 2577640.98 217 96816 606.24 2483150.82 218 97456 646.38 2647580.07 219 98144 652.78 2673793.74 220 98864 655.40 2684531.86 221 99680 673.93 2760409.24 222 100288 692.17 2835110.20 223 101040 722.56 2959607.23 224 101792 718.09 2941296.95 225 102352 695.36 2848175.22 226 103040 674.14 2761270.79 227 103616 662.93 2715359.62 228 104288 645.45 2643766.79 229 105008 636.56 2607356.72 230 105648 660.77 2706502.99 231 106336 671.54 2750618.77 232 106928 659.74 2702304.99 233 107520 647.42 2651822.97 234 107968 608.54 2492561.59 235 108672 598.08 2449754.00 236 109248 586.13 2400808.59 237 109856 592.70 2427682.44 238 110560 611.90 2506338.64 239 111216 636.35 2606472.26 240 111888 642.67 2632367.33 241 112496 650.62 2664950.81 242 113184 658.73 2698160.18 243 113872 663.97 2719641.35 244 114496 669.92 2743975.23 245 115152 659.64 2701893.41 246 115728 645.87 2645463.68 247 116448 647.60 2652573.13 248 117088 638.08 2613564.70 249 117760 650.70 2665264.86 250 118304 620.94 2543386.41 251 118864 625.68 2562788.36 252 119536 624.58 2558271.20 253 120240 614.65 2517595.44 254 120784 605.75 2481161.09 255 121552 634.36 2598327.03 256 122080 640.11 2621888.15 257 122800 645.55 2644178.90 258 123424 653.51 2676785.25 259 124128 663.47 2717587.17 260 124816 682.26 2794516.83 261 125536 691.73 2833342.36 262 126192 686.06 2810121.72 263 126816 684.40 2803317.55 264 127568 671.33 2749774.44 265 128240 683.69 2800381.41 266 128912 669.82 2743592.78 267 129552 654.70 2681639.41 268 130016 637.94 2613004.76 269 130768 656.80 2690236.42 elapsed: 280 ops: 131072 ops/sec: 467.18 bytes/sec: 1913569.87 root@pve01:~# rbd bench Test01 --io-type read --io-size 4K --io-threads 16 --io-total 512M --io-pattern rand --pool=Pool01 bench type read io_size 4096 io_threads 16 bytes 536870912 pattern random SEC OPS OPS/SEC BYTES/SEC 1 13840 13911.26 56980503.30 2 28576 14324.25 58672114.86 3 43312 14461.54 59234479.72 4 57296 14341.94 58744583.52 5 72304 14475.17 59290311.54 6 86832 14597.99 59793366.21 7 101328 14549.99 59596763.72 8 116432 14623.59 59898220.84 elapsed: 8 ops: 131072 ops/sec: 14647.78 bytes/sec: 59997298.42 root@pve01:~#
イメージ削除
root@pve01:~# rbd rm --pool=Pool01 Test01 Removing image: 100% complete...done. root@pve01:~#
計測後のメモリ使用状況
3OSD構成と同様にピーク値で5GiB弱
6OSDから9OSDへ
OSD追加手順は以前と同じなので省略して
各ノードHDD3台で合計9OSDへ拡張した様子がこちら
OSD追加直後の画面を記録してしまったので
一番上のosd.8の使用中と合計値が0になってますが異常ではありません
pve01のosd.6とpve02のosd.7もリマップ中なので右側にレイテンシが表示されてます
使用状況のTotal Sizeは2.84TB(2.58TiB)になり
9OSDなのでクラスタ全体のPG値は250×9=2250になります
このラインからプールに512を割り当て可能(512×3=1536)になるので
プールのpg_numを512に増やしても問題ないが
今回はとりあえずプールのPG値は128のまま計測した
各HDD3台:計9OSDでの計測結果
Proxmox起動直後のメモリ使用状況
pve01は1.80GiB、pve01は1.82GiB、pve03は1.64GiB
6OSD構成に比べて全ノードで少し増加しました
CentOS7のVMから計測
[root@centos7-bench ~]# ./fio_cdm.sh /root Testing in /root Configuration: Size:1024m Loops:5 Write Only Zeroes:0 Running Benchmark, please wait... Results: Sequential Read: 148MB/s IOPS=0 Sequential Write: 37MB/s IOPS=0 512KB Read: 73MB/s IOPS=147 512KB Write: 6MB/s IOPS=12 Sequential Q32T1 Read: 146MB/s IOPS=4 Sequential Q32T1 Write: 36MB/s IOPS=1 4KB Read: 4MB/s IOPS=1250 4KB Write: 0MB/s IOPS=86 4KB Q32T1 Read: 3MB/s IOPS=1000 4KB Q32T1 Write: 0MB/s IOPS=78 4KB Q8T8 Read: 33MB/s IOPS=8500 4KB Q8T8 Write: 0MB/s IOPS=126 [root@centos7-bench ~]#
Windows10のVMから計測
------------------------------------------------------------------------------ CrystalDiskMark 7.0.0 x64 (C) 2007-2019 hiyohiyo Crystal Dew World: https://crystalmark.info/ ------------------------------------------------------------------------------ * MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s] * KB = 1000 bytes, KiB = 1024 bytes [Read] Sequential 1MiB (Q= 8, T= 1): 160.408 MB/s [ 153.0 IOPS] < 51924.64 us> Sequential 1MiB (Q= 1, T= 1): 131.922 MB/s [ 125.8 IOPS] < 7927.97 us> Random 4KiB (Q= 32, T=16): 142.109 MB/s [ 34694.6 IOPS] < 14659.91 us> Random 4KiB (Q= 1, T= 1): 7.093 MB/s [ 1731.7 IOPS] < 576.59 us> [Write] Sequential 1MiB (Q= 8, T= 1): 33.138 MB/s [ 31.6 IOPS] <247359.28 us> Sequential 1MiB (Q= 1, T= 1): 9.227 MB/s [ 8.8 IOPS] <110682.31 us> Random 4KiB (Q= 32, T=16): 1.522 MB/s [ 371.6 IOPS] <477505.40 us> Random 4KiB (Q= 1, T= 1): 0.158 MB/s [ 38.6 IOPS] < 24816.74 us> [Mix] Read 70%/Write 30% Sequential 1MiB (Q= 8, T= 1): 54.948 MB/s [ 52.4 IOPS] <147167.45 us> Sequential 1MiB (Q= 1, T= 1): 24.749 MB/s [ 23.6 IOPS] < 41885.18 us> Random 4KiB (Q= 32, T=16): 3.515 MB/s [ 858.2 IOPS] <245029.82 us> Random 4KiB (Q= 1, T= 1): 0.469 MB/s [ 114.5 IOPS] < 8700.89 us> Profile: Default Test: 1 GiB (x5) [Interval: 5 sec] <DefaultAffinity=DISABLED> Date: 2020/02/09 23:46:34 OS: Windows 10 Professional [10.0 Build 18363] (x64)
rados benchで計測
ブロックサイズ1M&16スレッドで60秒間書き込み
結果は79.0386MB/s
root@pve01:~# rados bench -p Pool01 -t 16 -b 1M 60 write --no-cleanup hints = 1 Maintaining 16 concurrent writes of 1048576 bytes to objects of size 1048576 for up to 60 seconds or 0 objects Object prefix: benchmark_data_pve01_23130 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 88 72 71.9902 72 0.16564 0.215228 2 16 169 153 76.492 81 0.104767 0.201348 3 16 259 243 80.9917 90 0.146392 0.193345 4 16 339 323 80.7417 80 0.329188 0.194219 5 16 419 403 80.5917 80 0.185352 0.193601 6 16 505 489 81.4916 86 0.16277 0.194242 7 16 579 563 80.4203 74 0.206485 0.194572 8 16 656 640 79.9918 77 0.219053 0.197334 9 16 733 717 79.6588 77 0.236515 0.198844 10 16 804 788 78.7922 71 0.185672 0.199997 11 16 889 873 79.3558 85 0.197623 0.199314 12 16 967 951 79.2421 78 0.111071 0.199652 13 16 1049 1033 79.4536 82 0.285896 0.199345 14 16 1128 1112 79.4206 79 0.191494 0.199678 15 16 1212 1196 79.7253 84 0.183798 0.1994 16 16 1291 1275 79.6795 79 0.157768 0.199617 17 16 1363 1347 79.2274 72 0.195446 0.200265 18 16 1439 1423 79.0477 76 0.33341 0.201031 19 16 1511 1495 78.6765 72 0.233796 0.201837 2020-02-09 23:53:07.334454 min lat: 0.0553582 max lat: 0.472811 avg lat: 0.202945 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 20 16 1581 1565 78.242 70 0.128632 0.202945 21 16 1656 1640 78.0868 75 0.221697 0.20391 22 16 1735 1719 78.128 79 0.17049 0.203842 23 16 1809 1793 77.9482 74 0.190351 0.204148 24 16 1886 1870 77.9082 77 0.246939 0.204175 25 16 1965 1949 77.9514 79 0.317125 0.20425 26 16 2036 2020 77.6837 71 0.162932 0.204374 27 16 2111 2095 77.5839 75 0.163913 0.205195 28 16 2186 2170 77.4914 75 0.17129 0.20606 29 16 2264 2248 77.5086 78 0.196872 0.205964 30 16 2337 2321 77.3581 73 0.236624 0.205939 31 16 2416 2400 77.4109 79 0.238466 0.205884 32 16 2497 2481 77.5229 81 0.163186 0.205636 33 16 2576 2560 77.5674 79 0.228701 0.205775 34 16 2645 2629 77.3152 69 0.234228 0.206282 35 16 2715 2699 77.106 70 0.175623 0.20695 36 16 2800 2784 77.3251 85 0.297798 0.206251 37 16 2886 2870 77.5593 86 0.170507 0.205774 38 16 2970 2954 77.7287 84 0.17443 0.205348 39 16 3047 3031 77.7096 77 0.107504 0.205023 2020-02-09 23:53:27.336681 min lat: 0.0553582 max lat: 0.540013 avg lat: 0.204824 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 40 16 3130 3114 77.8417 83 0.293891 0.204824 41 16 3217 3201 78.0649 87 0.17514 0.20445 42 16 3300 3284 78.1822 83 0.183305 0.204133 43 16 3385 3369 78.3406 85 0.162006 0.203623 44 16 3472 3456 78.5372 87 0.210051 0.203274 45 16 3557 3541 78.6807 85 0.13023 0.203015 46 16 3634 3618 78.6441 77 0.194956 0.202844 47 16 3716 3700 78.7153 82 0.392102 0.202842 48 16 3797 3781 78.7628 81 0.161462 0.202733 49 16 3886 3870 78.9715 89 0.22085 0.202208 50 16 3971 3955 79.0919 85 0.218483 0.201975 51 16 4052 4036 79.1292 81 0.213389 0.201685 52 16 4132 4116 79.1458 80 0.100467 0.2018 53 16 4215 4199 79.2183 83 0.336192 0.201748 54 16 4293 4277 79.1956 78 0.292019 0.201574 55 16 4370 4354 79.1555 77 0.129269 0.201884 56 16 4440 4424 78.9918 70 0.300838 0.20196 57 16 4524 4508 79.0795 84 0.129737 0.201957 58 16 4602 4586 79.0608 78 0.099031 0.201846 59 16 4680 4664 79.0427 78 0.196407 0.202116 2020-02-09 23:53:47.338588 min lat: 0.0553582 max lat: 0.540013 avg lat: 0.202367 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 60 16 4755 4739 78.9752 75 0.154389 0.202367 Total time run: 60.1732 Total writes made: 4756 Write size: 1048576 Object size: 1048576 Bandwidth (MB/sec): 79.0386 Stddev Bandwidth: 5.19613 Max bandwidth (MB/sec): 90 Min bandwidth (MB/sec): 69 Average IOPS: 79 Stddev IOPS: 5.19613 Max IOPS: 90 Min IOPS: 69 Average Latency(s): 0.202416 Stddev Latency(s): 0.0661593 Max latency(s): 0.540013 Min latency(s): 0.0553582 root@pve01:~#
先ほど書き込んだデータを16スレッドでシーケンシャル読み込み
結果は152.694MB/s
root@pve01:~# rados bench -p Pool01 -t 16 60 seq hints = 1 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 177 161 160.976 161 0.0227583 0.0913365 2 16 334 318 158.982 157 0.094527 0.0961563 3 16 474 458 152.652 140 0.139593 0.101263 4 16 615 599 149.735 141 0.0713489 0.104206 5 16 779 763 152.585 164 0.0125062 0.102944 6 16 931 915 152.486 152 0.105617 0.103223 7 16 1084 1068 152.555 153 0.123844 0.103764 8 16 1222 1206 150.734 138 0.217225 0.104581 9 16 1369 1353 150.318 147 0.0783718 0.105103 10 16 1521 1505 150.483 152 0.0115414 0.10538 11 16 1679 1663 151.165 158 0.183321 0.105096 12 16 1822 1806 150.484 143 0.0968277 0.105568 13 16 1974 1958 150.599 152 0.0133907 0.105537 14 16 2135 2119 151.34 161 0.0106423 0.104835 15 16 2289 2273 151.517 154 0.0715924 0.104747 16 16 2446 2430 151.858 157 0.0540856 0.104543 17 16 2604 2588 152.218 158 0.0462262 0.104236 18 16 2764 2748 152.649 160 0.0356474 0.104002 19 16 2919 2903 152.772 155 0.0176023 0.103778 2020-02-09 23:54:30.363160 min lat: 0.00275774 max lat: 0.520861 avg lat: 0.103969 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 20 16 3073 3057 152.833 154 0.343562 0.103969 21 16 3243 3227 153.649 170 0.106917 0.103613 22 16 3395 3379 153.574 152 0.280434 0.103806 23 16 3539 3523 153.157 144 0.00316049 0.104076 24 16 3693 3677 153.191 154 0.0448456 0.103918 25 16 3834 3818 152.703 141 0.16493 0.104326 26 16 3986 3970 152.675 152 0.171044 0.104359 27 16 4158 4142 153.391 172 0.00495006 0.103897 28 16 4311 4295 153.376 153 0.132305 0.103947 29 16 4465 4449 153.397 154 0.344788 0.103945 30 16 4607 4591 153.017 142 0.0582578 0.104127 31 16 4751 4735 152.726 144 0.0614718 0.104419 Total time run: 31.1472 Total reads made: 4756 Read size: 1048576 Object size: 1048576 Bandwidth (MB/sec): 152.694 Average IOPS: 152 Stddev IOPS: 8.44972 Max IOPS: 172 Min IOPS: 138 Average Latency(s): 0.104689 Max latency(s): 0.520861 Min latency(s): 0.00274626 root@pve01:~#
ブロックサイズ4K&16スレッドで60秒間書き込み
結果は1.32421MB/s 338IOPS
root@pve01:~# rados bench -p Pool01 -t 16 -b 4K 60 write --no-cleanup hints = 1 Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 60 seconds or 0 objects Object prefix: benchmark_data_pve01_23791 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 502 486 1.89822 1.89844 0.0103219 0.0313584 2 16 902 886 1.73031 1.5625 0.0315711 0.0359186 3 15 1248 1233 1.60532 1.35547 0.130497 0.0386013 4 16 1600 1584 1.54673 1.37109 0.0435803 0.040001 5 16 1947 1931 1.50842 1.35547 0.0624262 0.0408456 6 16 2280 2264 1.47378 1.30078 0.0119706 0.042123 7 16 2633 2617 1.46019 1.37891 0.01155 0.0424989 8 16 2968 2952 1.44123 1.30859 0.0331773 0.0429989 9 16 3282 3266 1.41736 1.22656 0.0630744 0.0440284 10 16 3617 3601 1.40646 1.30859 0.0147043 0.0442333 11 16 3916 3900 1.38478 1.16797 0.0126619 0.0450905 12 16 4287 4271 1.39013 1.44922 0.0172625 0.0449271 13 16 4639 4623 1.38896 1.375 0.0241574 0.0449506 14 16 4985 4969 1.38627 1.35156 0.0157414 0.044883 15 16 5372 5356 1.39463 1.51172 0.0360859 0.0446777 16 16 5718 5702 1.39192 1.35156 0.0137916 0.0448113 17 16 6021 6005 1.37965 1.18359 0.118752 0.0450827 18 16 6379 6363 1.38069 1.39844 0.0123304 0.0451727 19 16 6725 6709 1.37915 1.35156 0.0500335 0.0451378 2020-02-09 23:55:35.959600 min lat: 0.00675178 max lat: 0.235167 avg lat: 0.0451564 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 20 16 7094 7078 1.38225 1.44141 0.0174515 0.0451564 21 16 7436 7420 1.38004 1.33594 0.041436 0.0452083 22 16 7779 7763 1.37821 1.33984 0.0170461 0.045289 23 16 8109 8093 1.37433 1.28906 0.0358381 0.0454448 24 16 8446 8430 1.37191 1.31641 0.0182906 0.0455392 25 16 8763 8747 1.36656 1.23828 0.0121734 0.0456746 26 16 9111 9095 1.36627 1.35938 0.0525774 0.0457022 27 16 9426 9410 1.36124 1.23047 0.0144335 0.0458341 28 16 9748 9732 1.35754 1.25781 0.00849643 0.0459322 29 16 10060 10044 1.35275 1.21875 0.0117517 0.0461174 30 16 10386 10370 1.3501 1.27344 0.0302381 0.0461997 31 16 10688 10672 1.34459 1.17969 0.010182 0.0462968 32 16 11009 10993 1.34176 1.25391 0.0349307 0.0465635 33 16 11372 11356 1.34406 1.41797 0.0169311 0.046477 34 16 11688 11672 1.34083 1.23438 0.00974312 0.0465929 35 16 12043 12027 1.34214 1.38672 0.016703 0.0465258 36 16 12345 12329 1.33762 1.17969 0.0193157 0.0466212 37 16 12699 12683 1.33884 1.38281 0.015773 0.0466508 38 16 13004 12988 1.33495 1.19141 0.108326 0.0467399 39 16 13363 13347 1.33668 1.40234 0.0279975 0.0466892 2020-02-09 23:55:55.961930 min lat: 0.00641742 max lat: 0.389619 avg lat: 0.047048 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 40 16 13614 13598 1.32777 0.980469 0.0394219 0.047048 41 16 13953 13937 1.32768 1.32422 0.0335617 0.046999 42 16 14300 14284 1.32834 1.35547 0.0977868 0.0470156 43 16 14613 14597 1.32588 1.22266 0.0873738 0.0470861 44 16 14939 14923 1.32469 1.27344 0.0177279 0.0471329 45 16 15242 15226 1.32155 1.18359 0.0534714 0.0472605 46 16 15606 15590 1.32372 1.42188 0.0345469 0.0471906 47 16 15980 15964 1.32664 1.46094 0.0167525 0.04707 48 16 16309 16293 1.32577 1.28516 0.106103 0.0470467 49 16 16635 16619 1.3247 1.27344 0.0133591 0.0471593 50 16 16999 16983 1.32664 1.42188 0.016411 0.0470897 51 16 17354 17338 1.32781 1.38672 0.0167391 0.0470441 52 16 17702 17686 1.32841 1.35938 0.0168721 0.0470385 53 16 18034 18018 1.32782 1.29688 0.0217958 0.0470402 54 16 18380 18364 1.32825 1.35156 0.0544518 0.0470161 55 16 18744 18728 1.32995 1.42188 0.0357476 0.0469673 56 16 19108 19092 1.33159 1.42188 0.0233487 0.0469232 57 16 19428 19412 1.33015 1.25 0.0180554 0.0469441 58 16 19766 19750 1.32998 1.32031 0.0355035 0.0469846 59 16 20109 20093 1.33015 1.33984 0.0288256 0.0469803 2020-02-09 23:56:15.964641 min lat: 0.00615572 max lat: 0.389619 avg lat: 0.0469429 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 60 16 20456 20440 1.33056 1.35547 0.0334281 0.0469429 Total time run: 60.3457 Total writes made: 20457 Write size: 4096 Object size: 4096 Bandwidth (MB/sec): 1.32421 Stddev Bandwidth: 0.122039 Max bandwidth (MB/sec): 1.89844 Min bandwidth (MB/sec): 0.980469 Average IOPS: 338 Stddev IOPS: 31.2419 Max IOPS: 486 Min IOPS: 251 Average Latency(s): 0.0470925 Stddev Latency(s): 0.0406414 Max latency(s): 0.390156 Min latency(s): 0.00615572 root@pve01:~#
クリーンアップ
root@pve01:~# rados -p Pool01 cleanup Removed 20457 objects root@pve01:~#
rbd benchで計測
プール内に20GBでイメージを作成
root@pve01:~# rbd create --size=20G --pool=Pool01 Test01 root@pve01:~#
ブロックサイズ1M&16スレッドで1024MBをシーケンシャル書き込み&読み込みを計測
結果はWrite:17秒 約62MB/s・Read:6秒 約156MB/s
root@pve01:~# rbd bench Test01 --io-type write --io-size 1M --io-threads 16 --io-total 1024M --io-pattern seq --pool=Pool01 bench type write io_size 1048576 io_threads 16 bytes 1073741824 pattern sequential SEC OPS OPS/SEC BYTES/SEC 1 80 114.83 120407192.69 2 144 83.86 87928383.94 3 192 76.24 79947787.80 4 256 72.80 76339537.06 5 320 69.65 73033054.96 6 384 60.17 63095405.77 7 432 59.26 62136061.40 8 496 59.94 62846608.74 9 560 60.32 63245635.85 10 624 60.22 63145406.16 11 688 59.89 62797087.41 12 736 60.75 63700643.50 13 800 60.70 63649765.46 14 864 58.01 60831685.30 15 912 57.60 60396259.42 16 976 57.69 60493049.36 elapsed: 17 ops: 1024 ops/sec: 59.33 bytes/sec: 62208065.43 root@pve01:~# rbd bench Test01 --io-type read --io-size 1M --io-threads 16 --io-total 1024M --io-pattern seq --pool=Pool01 bench type read io_size 1048576 io_threads 16 bytes 1073741824 pattern sequential SEC OPS OPS/SEC BYTES/SEC 1 160 194.68 204141904.92 2 320 175.36 183878817.11 3 480 168.47 176657387.13 4 624 163.26 171191359.90 5 768 159.60 167357678.35 6 912 148.73 155954790.41 elapsed: 6 ops: 1024 ops/sec: 149.18 bytes/sec: 156426606.95 root@pve01:~#
ブロックサイズ4K&16スレッドで512MBをランダム書き込み&読み込みを計測
結果はWrite:227秒 約2.35MB/s・Read:7秒 約70MB/s
root@pve01:~# rbd bench Test01 --io-type write --io-size 4K --io-threads 16 --io-total 512M --io-pattern rand --pool=Pool01 bench type write io_size 4096 io_threads 16 bytes 536870912 pattern random SEC OPS OPS/SEC BYTES/SEC 1 5392 4916.23 20136889.34 2 5488 2352.07 9634095.72 3 5584 1719.86 7044529.98 4 5664 1323.36 5420470.89 5 5744 1132.05 4636857.79 6 5824 85.37 349688.73 7 5904 86.02 352334.72 8 5984 84.03 344192.54 9 6032 76.22 312197.18 10 6096 70.79 289974.61 11 6176 71.02 290910.77 12 6240 68.40 280174.98 13 6336 68.59 280934.10 14 6432 79.74 326626.12 51 6496 9.63 39436.67 52 6512 8.05 32987.05 53 6528 7.01 28709.53 54 10912 111.81 457990.53 55 11024 112.33 460086.57 56 11200 1056.58 4327736.53 57 11376 1148.22 4703122.34 58 11520 994.40 4073047.96 59 11664 150.76 617504.41 60 11856 164.16 672411.90 61 12000 161.42 661161.21 62 12112 149.47 612221.33 63 12272 155.37 636386.81 64 12464 158.85 650658.28 65 12688 163.13 668192.89 66 12864 170.07 696624.48 67 13056 185.38 759333.06 68 13248 196.45 804668.39 69 13504 211.72 867208.84 70 13728 215.05 880837.44 71 13936 212.02 868433.26 72 14096 210.18 860899.35 73 14240 197.60 809387.87 74 14576 210.19 860940.92 75 14864 228.29 935075.55 76 15136 244.89 1003076.17 77 15344 249.00 1019887.52 78 15616 274.32 1123594.65 79 15840 255.86 1048018.35 80 16128 248.42 1017533.55 81 16336 241.73 990143.74 82 16608 255.04 1044634.95 83 17008 277.06 1134849.80 84 17312 296.29 1213598.44 85 17568 293.15 1200750.91 86 18048 343.49 1406936.58 87 18400 356.11 1458633.46 88 18848 370.66 1518220.51 89 19184 375.89 1539661.71 98 19472 144.81 593138.45 99 23136 390.05 1597662.89 104 23360 284.14 1163819.94 105 26592 450.01 1843248.12 107 26720 414.15 1696343.59 108 26736 696.30 2852050.03 109 29568 641.13 2626076.65 110 29744 988.82 4050216.68 112 29760 468.63 1919496.76 113 32480 991.03 4059249.91 114 32512 986.99 4042702.52 115 34784 877.50 3594245.21 116 35024 1000.73 4099001.36 117 35056 1037.59 4249973.69 118 37312 949.66 3889813.31 119 37440 922.13 3777053.98 120 39408 927.00 3796997.61 121 39680 844.68 3459813.69 122 39696 911.21 3732317.13 123 41760 903.31 3699948.60 124 41808 942.17 3859118.95 125 43712 856.67 3508904.89 126 43808 862.13 3531299.22 127 45632 1233.55 5052627.10 128 45824 721.06 2953464.08 129 47520 1243.87 5094895.75 130 47744 785.03 3215470.24 131 49472 1232.34 5047684.66 132 50208 884.74 3623902.41 133 51792 1353.87 5545442.78 134 52096 912.99 3739589.71 135 53696 1210.71 4959067.90 136 54416 991.15 4059766.24 137 56064 1209.89 4955696.73 138 57184 1091.47 4470663.20 139 58384 1256.56 5146882.51 140 59472 1169.20 4789048.80 141 60656 1260.07 5161227.29 142 61456 1068.12 4375015.26 143 61584 877.87 3595759.64 144 62960 915.91 3751566.33 145 64112 932.45 3819325.36 146 65120 890.64 3648061.70 147 66112 944.02 3866686.47 148 67200 1103.75 4520943.26 149 68256 1061.72 4348811.62 150 69216 1015.09 4157808.32 151 69952 979.70 4012850.03 152 70992 975.20 3994400.40 153 71984 972.33 3982677.57 154 73104 943.90 3866221.04 155 73920 938.52 3844194.37 156 74736 954.49 3909571.86 157 75680 932.36 3818927.64 158 76608 929.24 3806155.72 159 77536 910.41 3729050.68 160 78480 914.17 3744445.42 161 79328 922.07 3776779.65 162 80304 914.53 3745931.88 163 81200 917.64 3758665.67 164 82064 907.03 3715189.44 165 82928 893.87 3661284.98 166 83856 903.41 3700364.33 167 84752 881.82 3611925.93 168 85536 868.57 3557655.04 169 86352 849.42 3479241.48 170 87184 845.09 3461505.64 171 87888 805.74 3300292.04 172 88880 846.57 3467569.27 173 89824 856.21 3507031.07 174 90480 820.98 3362742.25 175 91536 873.17 3576514.73 176 92464 918.12 3760600.08 177 93520 909.78 3726464.41 178 94416 916.91 3755664.04 179 95328 974.25 3990537.72 180 96368 956.43 3917533.94 181 97200 945.66 3873437.58 182 98288 966.72 3959700.80 183 99200 952.21 3900234.65 184 100000 936.62 3836415.07 185 100736 886.34 3630455.63 186 101696 899.90 3685980.89 187 102528 850.02 3481678.15 188 103440 837.92 3432136.52 189 104256 857.35 3511713.40 190 105088 865.53 3545217.13 191 105952 847.79 3472538.96 192 106928 884.93 3624689.30 193 107936 916.03 3752070.42 194 108912 886.50 3631093.99 195 109312 846.13 3465760.70 196 110544 914.72 3746686.94 197 111520 913.99 3743703.94 198 112400 884.29 3622047.08 199 113312 916.64 3754574.22 200 114080 956.64 3918388.30 201 115040 902.06 3694856.12 202 115984 889.93 3645154.36 203 116736 849.51 3479588.42 204 117632 863.29 3536028.19 205 118560 896.70 3672864.11 206 119376 852.85 3493276.90 207 119472 663.61 2718131.13 208 121280 919.07 3764516.01 209 121536 729.43 2987741.34 210 123168 923.05 3780830.33 212 123760 717.26 2937896.56 213 125968 1098.01 4497458.40 214 126032 709.66 2906767.50 215 126048 801.69 3283706.35 217 128416 681.54 2791595.04 218 130256 1122.29 4596918.61 220 130912 681.73 2792350.91 elapsed: 227 ops: 131072 ops/sec: 575.49 bytes/sec: 2357201.11 root@pve01:~# rbd bench Test01 --io-type read --io-size 4K --io-threads 16 --io-total 512M --io-pattern rand --pool=Pool01 bench type read io_size 4096 io_threads 16 bytes 536870912 pattern random SEC OPS OPS/SEC BYTES/SEC 1 15328 15405.25 63099908.29 2 32944 16512.63 67635724.23 3 49216 16432.18 67306212.16 4 66400 16620.22 68076421.46 5 84080 16832.26 68944939.19 6 103184 17570.78 71969901.57 7 120592 17529.18 71799512.60 elapsed: 7 ops: 131072 ops/sec: 17066.26 bytes/sec: 69903382.98 root@pve01:~#
イメージ削除
root@pve01:~# rbd rm --pool=Pool01 Test01 Removing image: 100% complete...done. root@pve01:~#
計測後のメモリ使用状況
ピーク値で5.6GiBを記録しているので6OSD構成より増加
まとめ
CephはOSDの数が多ければ多いほど高速になると言われている通り
3OSD→6OSD→9OSDと順にパフォーマンスが上がっているのがわかります
ただその速度増加幅がOSD3台追加した結果にしては何とも辛い数値なので
このアプローチは正直コストパフォーマンスは悪いです
コメント