Proxmox VE 6.1 + Cephで速度計測 各ノードSSD+HDD3台の記録
最初に
前記事の続きでDB/WALにSSDを使ったパターンを測定していきます
計測環境
以前の記事にスペックや計測方法を掲載してますのでそちらをご覧ください
Proxmox VE 6.1 + Cephで速度計測:HDDx1 | fefcc.net
https://fefcc.net/archives/1040
今回のCeph構成
各ノードに登録しているOSDはHDD ST1000DM010 1TB 7200rpmを3台
各OSDのDB/WALはSSD SATA Crucial MX500 500GBを共有で登録
Pool01という名でCeph/RBD形式のプール作成
Pool01のPG値は128のまま
6OSDから9OSDへ拡張
拡張手順は前記事と同じなので省略します
9OSDになった状態がこちら
各HDD3台+SSD:計9OSDでの計測結果
Proxmox起動直後のメモリ使用状況
pve01は1.83GiB、pve01は1.82GiB、pve03は1.70GiB
CentOS7のVMから計測
[root@centos7-bench ~]# ./fio_cdm.sh /root Testing in /root Configuration: Size:1024m Loops:5 Write Only Zeroes:0 Running Benchmark, please wait... Results: Sequential Read: 167MB/s IOPS=0 Sequential Write: 31MB/s IOPS=0 512KB Read: 89MB/s IOPS=178 512KB Write: 9MB/s IOPS=19 Sequential Q32T1 Read: 159MB/s IOPS=4 Sequential Q32T1 Write: 31MB/s IOPS=0 4KB Read: 3MB/s IOPS=1000 4KB Write: 1MB/s IOPS=454 4KB Q32T1 Read: 3MB/s IOPS=1000 4KB Q32T1 Write: 1MB/s IOPS=454 4KB Q8T8 Read: 33MB/s IOPS=8500 4KB Q8T8 Write: 1MB/s IOPS=443 [root@centos7-bench ~]#
Windows10のVMから計測
------------------------------------------------------------------------------ CrystalDiskMark 7.0.0 x64 (C) 2007-2019 hiyohiyo Crystal Dew World: https://crystalmark.info/ ------------------------------------------------------------------------------ * MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s] * KB = 1000 bytes, KiB = 1024 bytes [Read] Sequential 1MiB (Q= 8, T= 1): 175.559 MB/s [ 167.4 IOPS] < 47565.99 us> Sequential 1MiB (Q= 1, T= 1): 149.546 MB/s [ 142.6 IOPS] < 7009.63 us> Random 4KiB (Q= 32, T=16): 171.832 MB/s [ 41951.2 IOPS] < 12167.36 us> Random 4KiB (Q= 1, T= 1): 7.871 MB/s [ 1921.6 IOPS] < 519.55 us> [Write] Sequential 1MiB (Q= 8, T= 1): 41.949 MB/s [ 40.0 IOPS] <194496.06 us> Sequential 1MiB (Q= 1, T= 1): 13.423 MB/s [ 12.8 IOPS] < 75841.40 us> Random 4KiB (Q= 32, T=16): 1.993 MB/s [ 486.6 IOPS] <409282.24 us> Random 4KiB (Q= 1, T= 1): 0.395 MB/s [ 96.4 IOPS] < 10267.84 us> [Mix] Read 70%/Write 30% Sequential 1MiB (Q= 8, T= 1): 83.253 MB/s [ 79.4 IOPS] <100251.29 us> Sequential 1MiB (Q= 1, T= 1): 34.602 MB/s [ 33.0 IOPS] < 30072.45 us> Random 4KiB (Q= 32, T=16): 5.622 MB/s [ 1372.6 IOPS] <220665.57 us> Random 4KiB (Q= 1, T= 1): 1.012 MB/s [ 247.1 IOPS] < 4003.04 us> Profile: Default Test: 1 GiB (x5) [Interval: 5 sec] <DefaultAffinity=DISABLED> Date: 2020/02/12 4:02:39 OS: Windows 10 Professional [10.0 Build 18363] (x64)
rados benchで計測
ブロックサイズ1M&16スレッドで60秒間書き込み
結果は103.14MB/s
root@pve01:~# rados bench -p Pool01 -t 16 -b 1M 60 write --no-cleanup hints = 1 Maintaining 16 concurrent writes of 1048576 bytes to objects of size 1048576 for up to 60 seconds or 0 objects Object prefix: benchmark_data_pve01_12417 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 95 79 78.9943 79 0.151648 0.186919 2 16 202 186 92.9919 107 0.146749 0.165073 3 16 305 289 96.3254 103 0.19436 0.160054 4 16 407 391 97.7422 102 0.173736 0.159574 5 16 512 496 99.1917 105 0.0796191 0.157412 6 16 614 598 99.658 102 0.189238 0.158253 7 16 717 701 100.134 103 0.170663 0.157495 8 16 827 811 101.366 110 0.220368 0.156527 9 16 933 917 101.88 106 0.0990909 0.155447 10 16 1044 1028 102.791 111 0.103954 0.154413 11 16 1143 1127 102.446 99 0.291773 0.154736 12 16 1250 1234 102.824 107 0.123457 0.154235 13 16 1356 1340 103.068 106 0.0947886 0.153908 14 16 1462 1446 103.277 106 0.195154 0.154174 15 16 1564 1548 103.191 102 0.17496 0.15424 16 16 1665 1649 103.054 101 0.415889 0.154329 17 16 1765 1749 102.873 100 0.0910751 0.154555 18 16 1866 1850 102.769 101 0.166857 0.154938 19 16 1973 1957 102.991 107 0.131006 0.154545 2020-02-12 04:05:22.078779 min lat: 0.0419693 max lat: 0.437319 avg lat: 0.154428 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 20 16 2079 2063 103.141 106 0.0902121 0.154428 21 16 2183 2167 103.181 104 0.166472 0.154477 22 16 2286 2270 103.173 103 0.0953462 0.154507 23 16 2387 2371 103.078 101 0.103126 0.154517 24 16 2488 2472 102.991 101 0.116258 0.154877 25 16 2592 2576 103.031 104 0.137099 0.154807 26 16 2692 2676 102.914 100 0.150684 0.155049 27 16 2794 2778 102.88 102 0.0801669 0.154879 28 16 2902 2886 103.062 108 0.107229 0.154967 29 16 3005 2989 103.06 103 0.300001 0.154834 30 16 3103 3087 102.891 98 0.140845 0.154911 31 16 3208 3192 102.959 105 0.118837 0.154922 32 16 3307 3291 102.835 99 0.0533002 0.154924 33 16 3407 3391 102.749 100 0.128777 0.155309 34 16 3516 3500 102.932 109 0.285367 0.155085 35 16 3621 3605 102.991 105 0.0580663 0.154891 36 16 3727 3711 103.074 106 0.289533 0.154987 37 16 3822 3806 102.856 95 0.188138 0.154945 38 16 3930 3914 102.991 108 0.220447 0.154921 39 16 4037 4021 103.094 107 0.108037 0.154935 2020-02-12 04:05:42.080542 min lat: 0.0362313 max lat: 0.475229 avg lat: 0.154997 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 40 16 4140 4124 103.091 103 0.162846 0.154997 41 16 4241 4225 103.04 101 0.247305 0.155041 42 16 4342 4326 102.991 101 0.334396 0.155022 43 16 4440 4424 102.875 98 0.223068 0.155165 44 16 4549 4533 103.014 109 0.19159 0.155047 45 16 4645 4629 102.858 96 0.131663 0.155224 46 16 4747 4731 102.839 102 0.167792 0.155293 47 16 4852 4836 102.885 105 0.0679347 0.155206 48 16 4952 4936 102.824 100 0.186389 0.155217 49 16 5057 5041 102.869 105 0.163051 0.15523 50 16 5163 5147 102.931 106 0.110075 0.155175 51 16 5266 5250 102.932 103 0.191388 0.155178 52 16 5368 5352 102.914 102 0.142332 0.155127 53 16 5474 5458 102.972 106 0.11335 0.155106 54 16 5584 5568 103.102 110 0.0522058 0.155 55 16 5686 5670 103.082 102 0.212901 0.154987 56 16 5787 5771 103.045 101 0.14213 0.155005 57 16 5888 5872 103.009 101 0.167801 0.154976 58 16 5994 5978 103.06 106 0.170768 0.155027 59 16 6101 6085 103.127 107 0.184237 0.154861 2020-02-12 04:06:02.082082 min lat: 0.0362313 max lat: 0.501009 avg lat: 0.155036 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 60 16 6198 6182 103.025 97 0.0932122 0.155036 Total time run: 60.1027 Total writes made: 6199 Write size: 1048576 Object size: 1048576 Bandwidth (MB/sec): 103.14 Stddev Bandwidth: 4.76558 Max bandwidth (MB/sec): 111 Min bandwidth (MB/sec): 79 Average IOPS: 103 Stddev IOPS: 4.76558 Max IOPS: 111 Min IOPS: 79 Average Latency(s): 0.155081 Stddev Latency(s): 0.0675867 Max latency(s): 0.501009 Min latency(s): 0.0362313 root@pve01:~#
先ほど書き込んだデータを16スレッドでシーケンシャル読み込み
結果は166.588MB/s
root@pve01:~# rados bench -p Pool01 -t 16 60 seq hints = 1 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 179 163 162.98 163 0.106703 0.0907979 2 16 337 321 160.482 158 0.00305732 0.0950809 3 16 504 488 162.649 167 0.0693441 0.0949025 4 16 679 663 165.732 175 0.0112448 0.0940464 5 16 849 833 166.58 170 0.00308126 0.0942842 6 16 1017 1001 166.814 168 0.0850132 0.0946196 7 16 1178 1162 165.981 161 0.125311 0.0949395 8 16 1336 1320 164.982 158 0.247367 0.0957647 9 16 1520 1504 167.093 184 0.209451 0.0947676 10 16 1696 1680 167.982 176 0.0031072 0.0944238 11 16 1863 1847 167.892 167 0.00771158 0.0943504 12 16 2050 2034 169.483 187 0.154345 0.0937686 13 16 2217 2201 169.291 167 0.276421 0.093776 14 16 2374 2358 168.412 157 0.00453055 0.0944787 15 16 2540 2524 168.251 166 0.0546916 0.0942288 16 16 2713 2697 168.547 173 0.144331 0.0941479 17 16 2876 2860 168.22 163 0.285592 0.0945412 18 16 3049 3033 168.484 173 0.0751339 0.0943583 19 16 3218 3202 168.511 169 0.171937 0.0944731 2020-02-12 04:06:54.846803 min lat: 0.00275273 max lat: 0.479925 avg lat: 0.0942669 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 20 16 3389 3373 168.635 171 0.0033018 0.0942669 21 16 3566 3550 169.032 177 0.258975 0.0941919 22 16 3723 3707 168.485 157 0.0678435 0.0944179 23 16 3882 3866 168.072 159 0.00304002 0.0946061 24 16 4051 4035 168.11 169 0.00303442 0.0947492 25 16 4207 4191 167.625 156 0.0102796 0.0951243 26 16 4359 4343 167.024 152 0.109681 0.0954078 27 16 4520 4504 166.8 161 0.104861 0.0955588 28 16 4680 4664 166.557 160 0.239691 0.0956712 29 16 4855 4839 166.848 175 0.331598 0.0955015 30 16 5020 5004 166.786 165 0.24207 0.0955813 31 16 5184 5168 166.696 164 0.0793535 0.0956012 32 16 5352 5336 166.736 168 0.00715408 0.0956675 33 16 5520 5504 166.774 168 0.00939843 0.0956116 34 16 5690 5674 166.868 170 0.00313725 0.0955374 35 16 5861 5845 166.986 171 0.114454 0.0955115 36 16 6022 6006 166.82 161 0.00324846 0.0956329 37 16 6184 6168 166.689 162 0.250121 0.0956618 Total time run: 37.2116 Total reads made: 6199 Read size: 1048576 Object size: 1048576 Bandwidth (MB/sec): 166.588 Average IOPS: 166 Stddev IOPS: 7.68789 Max IOPS: 187 Min IOPS: 152 Average Latency(s): 0.0959081 Max latency(s): 0.479925 Min latency(s): 0.00275273 root@pve01:~#
ブロックサイズ4K&16スレッドで60秒間書き込み
結果は3.88144MB/s 993IOPS
root@pve01:~# rados bench -p Pool01 -t 16 -b 4K 60 write --no-cleanup hints = 1 Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 60 seconds or 0 objects Object prefix: benchmark_data_pve01_13098 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 1054 1038 4.05428 4.05469 0.00286729 0.0154049 2 16 2096 2080 4.06192 4.07031 0.026003 0.0151394 3 16 3037 3021 3.93311 3.67578 0.00166826 0.0158184 4 16 3919 3903 3.81096 3.44531 0.00131153 0.0163322 5 16 4816 4800 3.74942 3.50391 0.00186369 0.0164931 6 16 5856 5840 3.80155 4.0625 0.00356462 0.0164175 7 16 6977 6961 3.88396 4.37891 0.00245743 0.0160592 8 16 7985 7969 3.89057 3.9375 0.00169477 0.0160347 9 16 8955 8939 3.87921 3.78906 0.0324712 0.016041 10 16 9958 9942 3.88304 3.91797 0.00180067 0.0160666 11 16 10994 10978 3.89789 4.04688 0.00331544 0.0160296 12 16 12000 11984 3.90052 3.92969 0.059654 0.0159171 13 16 12995 12979 3.89944 3.88672 0.00279998 0.0160265 14 16 13934 13918 3.88286 3.66797 0.0515692 0.0160514 15 16 14964 14948 3.8922 4.02344 0.0141784 0.0160463 16 16 16061 16045 3.91673 4.28516 0.00187829 0.0159344 17 16 17016 17000 3.90576 3.73047 0.00162955 0.015976 18 16 17992 17976 3.90057 3.8125 0.00247569 0.0160194 19 16 18912 18896 3.8844 3.59375 0.00160276 0.0160639 2020-02-12 04:07:53.934573 min lat: 0.00103981 max lat: 0.170747 avg lat: 0.016074 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 20 16 19905 19889 3.88408 3.87891 0.00308883 0.016074 21 16 20846 20830 3.87415 3.67578 0.0016341 0.016113 22 16 21906 21890 3.88625 4.14062 0.00291517 0.0160796 23 16 22764 22748 3.86299 3.35156 0.00288785 0.016171 24 16 23722 23706 3.85794 3.74219 0.0305647 0.0161802 25 16 24712 24696 3.8583 3.86719 0.0013978 0.0161941 26 16 25715 25699 3.86058 3.91797 0.00147027 0.0161742 27 16 26754 26738 3.8679 4.05859 0.0194299 0.0161522 28 16 27900 27884 3.88963 4.47656 0.00175452 0.0160669 29 16 28867 28851 3.88574 3.77734 0.0019096 0.0160806 30 15 29895 29880 3.89019 4.01953 0.025952 0.0160626 31 16 30856 30840 3.88566 3.75 0.0256567 0.0160682 32 16 31790 31774 3.87824 3.64844 0.00150022 0.0161095 33 16 32802 32786 3.88049 3.95312 0.00191978 0.0160949 34 16 33744 33728 3.87458 3.67969 0.00171417 0.0161248 35 16 34745 34729 3.87559 3.91016 0.00147822 0.01612 36 16 35665 35649 3.86775 3.59375 0.00258775 0.0161447 37 16 36733 36717 3.87595 4.17188 0.00205322 0.0161162 38 16 37749 37733 3.87839 3.96875 0.0015907 0.0161134 39 16 38754 38738 3.8796 3.92578 0.0307937 0.0161055 2020-02-12 04:08:13.936306 min lat: 0.000950359 max lat: 0.170747 avg lat: 0.0161619 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 40 16 39615 39599 3.86668 3.36328 0.00341552 0.0161619 41 16 40580 40564 3.8643 3.76953 0.00199353 0.0161717 42 16 41496 41480 3.85748 3.57812 0.00180525 0.0161993 43 16 42411 42395 3.85089 3.57422 0.0257882 0.0162203 44 16 43471 43455 3.85746 4.14062 0.0309781 0.0161954 45 16 44526 44510 3.86331 4.12109 0.0014509 0.016175 46 16 45535 45519 3.865 3.94141 0.00190734 0.0161623 47 16 46486 46470 3.8618 3.71484 0.00201577 0.0161813 48 16 47421 47405 3.85743 3.65234 0.00211255 0.0162016 49 16 48478 48462 3.86296 4.12891 0.002312 0.0161749 50 16 49500 49484 3.86554 3.99219 0.0023025 0.0161678 51 16 50471 50455 3.86411 3.79297 0.0014256 0.0161667 52 16 51487 51471 3.86612 3.96875 0.00183706 0.0161642 53 16 52481 52465 3.86642 3.88281 0.00202719 0.0161618 54 16 53524 53508 3.87027 4.07422 0.0333304 0.016146 55 16 54505 54489 3.86956 3.83203 0.00313067 0.0161481 56 16 55536 55520 3.87236 4.02734 0.0329031 0.0161312 57 16 56588 56572 3.87651 4.10938 0.0106306 0.0161222 58 16 57585 57569 3.87682 3.89453 0.00180171 0.0161114 59 16 58617 58601 3.87943 4.03125 0.00226624 0.0161044 2020-02-12 04:08:33.938399 min lat: 0.000950359 max lat: 0.170747 avg lat: 0.0160916 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 60 16 59636 59620 3.8811 3.98047 0.0254604 0.0160916 Total time run: 60.0172 Total writes made: 59636 Write size: 4096 Object size: 4096 Bandwidth (MB/sec): 3.88144 Stddev Bandwidth: 0.230489 Max bandwidth (MB/sec): 4.47656 Min bandwidth (MB/sec): 3.35156 Average IOPS: 993 Stddev IOPS: 59.0052 Max IOPS: 1146 Min IOPS: 858 Average Latency(s): 0.0161019 Stddev Latency(s): 0.020806 Max latency(s): 0.170747 Min latency(s): 0.000950359 root@pve01:~#
クリーンアップ
root@pve01:~# rados -p Pool01 cleanup Removed 59636 objects root@pve01:~#
rbd benchで計測
プール内に20GBでイメージを作成
root@pve01:~# rbd create --size=20G --pool=Pool01 Test01 root@pve01:~#
ブロックサイズ1M&16スレッドで1024MBをシーケンシャル書き込み&読み込みを計測
結果はWrite:12秒 約83MB/s・Read:5秒 約182MB/s
root@pve01:~# rbd bench Test01 --io-type write --io-size 1M --io-threads 16 --io-total 1024M --io-pattern seq --pool=Pool01 bench type write io_size 1048576 io_threads 16 bytes 1073741824 pattern sequential SEC OPS OPS/SEC BYTES/SEC 1 96 107.28 112488497.79 2 176 91.78 96234355.43 3 256 89.00 93326751.16 4 336 87.56 91813639.70 5 416 86.33 90522604.25 6 496 79.36 83218531.78 7 576 80.97 84903117.52 8 672 80.25 84143182.28 9 736 79.81 83683440.00 10 816 79.17 83020863.38 11 896 79.81 83683440.16 12 976 77.94 81726695.76 elapsed: 12 ops: 1024 ops/sec: 78.94 bytes/sec: 82772031.10 root@pve01:~# rbd bench Test01 --io-type read --io-size 1M --io-threads 16 --io-total 1024M --io-pattern seq --pool=Pool01 bench type read io_size 1048576 io_threads 16 bytes 1073741824 pattern sequential SEC OPS OPS/SEC BYTES/SEC 1 192 203.92 213822679.81 2 352 190.47 199724481.16 3 544 190.47 199724481.28 4 720 187.75 196871274.50 5 912 186.94 196022899.62 elapsed: 5 ops: 1024 ops/sec: 173.91 bytes/sec: 182357135.34 root@pve01:~#
ブロックサイズ4K&16スレッドで512MBをランダム書き込み&読み込みを計測
結果はWrite:140秒 約3.8MB/s・Read:7秒 約74MB/s
root@pve01:~# rbd bench Test01 --io-type write --io-size 4K --io-threads 16 --io-total 512M --io-pattern rand --pool=Pool01 bench type write io_size 4096 io_threads 16 bytes 536870912 pattern random SEC OPS OPS/SEC BYTES/SEC 1 5680 5584.19 22872859.32 2 6144 2996.04 12271799.52 3 6592 2199.69 9009916.29 4 7104 1774.64 7268917.16 5 7664 1527.41 6256286.11 6 8288 522.42 2139851.63 7 8912 558.50 2287628.19 8 9472 576.45 2361134.40 9 10144 607.50 2488324.03 10 10816 630.89 2584130.44 11 11456 634.09 2597247.85 12 12336 686.43 2811628.68 13 12992 704.55 2885830.95 14 13888 749.98 3071934.26 15 14832 807.71 3308363.63 16 15984 904.13 3703332.89 17 17072 944.91 3870359.32 18 18096 1017.52 4167770.47 20 18880 738.45 3024673.75 21 18896 631.04 2584749.67 22 22880 1151.61 4717002.62 23 23312 994.88 4075014.86 24 23344 793.21 3248994.58 25 26784 1844.97 7556977.47 26 27216 1682.81 6892797.54 27 27232 843.39 3454536.96 28 30576 1469.22 6017942.48 29 31056 1645.02 6737985.58 30 31072 762.97 3125137.43 31 34240 1519.00 6221818.49 32 35088 1465.64 6003262.66 33 35104 892.73 3656603.19 34 35120 862.46 3532637.53 35 39200 1872.77 7670870.07 36 39296 956.83 3919177.36 37 39328 822.96 3370859.60 38 42928 1657.59 6789495.51 39 43984 1528.24 6259684.10 41 44000 727.26 2978845.41 42 44016 807.65 3308129.14 43 48640 1742.48 7137188.74 44 48752 913.98 3743658.83 45 48768 922.11 3776955.50 46 48816 1029.04 4214938.97 47 53520 1950.70 7990055.82 49 54144 900.50 3688465.67 50 54160 825.88 3382818.36 51 54176 842.35 3450263.96 52 58320 1666.16 6824605.91 53 59712 1024.47 4196209.46 54 60688 1345.37 5510619.08 55 61728 1860.33 7619927.65 56 62832 1891.57 7747862.43 57 63920 1119.08 4583754.98 58 65120 1090.30 4465865.88 59 66240 1105.07 4526374.64 60 67264 1109.84 4545904.49 61 68400 1116.26 4572181.39 62 69536 1124.08 4604212.29 63 70576 1091.18 4469459.73 64 71616 1078.63 4418062.93 65 72784 1089.16 4461214.89 66 73696 1056.64 4328003.33 67 74720 1024.48 4196286.66 68 75760 1036.78 4246642.10 69 76848 1042.21 4268887.32 70 77872 1025.78 4201613.50 71 78736 1008.79 4131985.42 72 79856 1039.65 4258422.42 73 80880 1018.28 4170857.63 74 81776 990.33 4056402.13 75 82784 983.17 4027046.12 76 83824 1018.39 4171337.68 77 84800 987.99 4046800.89 78 85664 945.43 3872499.14 79 86576 953.12 3903967.49 80 87680 976.83 4001114.90 81 88688 967.36 3962314.78 82 89680 976.76 4000811.31 83 90608 1001.60 4102551.00 84 91536 991.98 4063145.27 88 93008 599.18 2454231.02 90 93040 476.97 1953683.86 91 97856 906.81 3714305.82 92 98624 891.44 3651338.45 93 99456 883.52 3618878.59 94 100144 1391.55 5699772.29 95 100848 1611.86 6602164.66 96 101840 795.51 3258409.71 97 102704 810.15 3318372.41 98 103472 796.81 3263727.19 99 104240 824.46 3376984.31 100 105136 854.85 3501450.02 101 106496 838.00 3432427.86 102 106864 842.09 3449189.58 103 107664 833.71 3414889.76 104 108640 860.03 3522677.01 105 109232 819.84 3358058.07 106 110112 806.41 3303037.49 107 110992 818.38 3352087.11 108 111680 814.92 3337902.83 109 112672 828.25 3392506.12 110 113472 848.66 3476114.82 111 114144 804.45 3295035.88 112 115168 843.28 3454065.28 113 115920 844.60 3459495.95 114 116928 851.86 3489232.24 115 117808 862.35 3532195.17 116 118576 894.97 3665808.75 117 119376 821.22 3363700.33 118 120144 839.41 3438222.40 119 120832 767.88 3145248.44 120 121472 733.37 3003887.92 121 122288 745.96 3055471.84 122 123024 747.53 3061862.59 123 123840 744.54 3049656.18 124 124624 769.46 3151725.02 125 125456 795.51 3258409.78 126 126256 789.79 3234988.37 127 126928 781.41 3200649.15 128 127712 762.79 3124384.15 129 128416 760.21 3113813.34 130 129120 729.28 2987146.53 131 129952 738.00 3022862.09 132 130656 743.21 3044171.17 elapsed: 140 ops: 131072 ops/sec: 932.61 bytes/sec: 3819976.14 root@pve01:~# rbd bench Test01 --io-type read --io-size 4K --io-threads 16 --io-total 512M --io-pattern rand --pool=Pool01 bench type read io_size 4096 io_threads 16 bytes 536870912 pattern random SEC OPS OPS/SEC BYTES/SEC 1 17952 17967.62 73595362.04 2 36432 18260.13 74793502.77 3 53936 18007.63 73759242.35 4 72416 18125.74 74243032.94 5 90768 18170.95 74428213.04 6 108496 18122.91 74231451.86 7 127264 18166.01 74407991.31 elapsed: 7 ops: 131072 ops/sec: 18123.51 bytes/sec: 74233889.61 root@pve01:~#
イメージ削除
root@pve01:~# rbd rm --pool=Pool01 Test01 Removing image: 100% complete...done. root@pve01:~#
計測後のメモリ使用状況
ピーク値で5.6GiB
プールのpg_numを128から512へ変更
9OSDなので設定可能なPG値は全体で1OSDあたり250×3OSD×3ノード=2250
プールを512にする為には512×プールサイズ3=1536が必要という訳で
今回はHDDx4のパターンと同様にpg_num:512で設定します
設定変更は前回同様にProxmoxのシェルからコマンドで対応します
root@pve01:~# ceph osd pool set Pool01 pg_num 512 set pool 3 pg_num to 512 root@pve01:~#
各HDD3台+SSD:計9OSD(PG:512)での計測結果
Proxmox起動直後のメモリ使用状況
pve01は1.90GiB、pve01は1.94GiB、pve03は1.78GiB
CentOS7のVMから計測
[root@centos7-bench ~]# ./fio_cdm.sh /root Testing in /root Configuration: Size:1024m Loops:5 Write Only Zeroes:0 Running Benchmark, please wait... Results: Sequential Read: 162MB/s IOPS=0 Sequential Write: 28MB/s IOPS=0 512KB Read: 73MB/s IOPS=147 512KB Write: 7MB/s IOPS=14 Sequential Q32T1 Read: 158MB/s IOPS=4 Sequential Q32T1 Write: 31MB/s IOPS=0 4KB Read: 3MB/s IOPS=1000 4KB Write: 1MB/s IOPS=416 4KB Q32T1 Read: 3MB/s IOPS=1000 4KB Q32T1 Write: 1MB/s IOPS=416 4KB Q8T8 Read: 26MB/s IOPS=6833 4KB Q8T8 Write: 1MB/s IOPS=433 [root@centos7-bench ~]#
Windows10のVMから計測
------------------------------------------------------------------------------ CrystalDiskMark 7.0.0 x64 (C) 2007-2019 hiyohiyo Crystal Dew World: https://crystalmark.info/ ------------------------------------------------------------------------------ * MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s] * KB = 1000 bytes, KiB = 1024 bytes [Read] Sequential 1MiB (Q= 8, T= 1): 175.110 MB/s [ 167.0 IOPS] < 47552.41 us> Sequential 1MiB (Q= 1, T= 1): 147.657 MB/s [ 140.8 IOPS] < 7091.94 us> Random 4KiB (Q= 32, T=16): 158.712 MB/s [ 38748.0 IOPS] < 13158.36 us> Random 4KiB (Q= 1, T= 1): 8.151 MB/s [ 1990.0 IOPS] < 501.59 us> [Write] Sequential 1MiB (Q= 8, T= 1): 35.655 MB/s [ 34.0 IOPS] <232078.99 us> Sequential 1MiB (Q= 1, T= 1): 11.534 MB/s [ 11.0 IOPS] < 90118.97 us> Random 4KiB (Q= 32, T=16): 1.853 MB/s [ 452.4 IOPS] <405421.46 us> Random 4KiB (Q= 1, T= 1): 0.353 MB/s [ 86.2 IOPS] < 11566.42 us> [Mix] Read 70%/Write 30% Sequential 1MiB (Q= 8, T= 1): 73.191 MB/s [ 69.8 IOPS] <113254.28 us> Sequential 1MiB (Q= 1, T= 1): 31.879 MB/s [ 30.4 IOPS] < 32735.02 us> Random 4KiB (Q= 32, T=16): 5.398 MB/s [ 1317.9 IOPS] <222806.10 us> Random 4KiB (Q= 1, T= 1): 0.966 MB/s [ 235.8 IOPS] < 4086.15 us> Profile: Default Test: 1 GiB (x5) [Interval: 5 sec] <DefaultAffinity=DISABLED> Date: 2020/02/12 16:16:52 OS: Windows 10 Professional [10.0 Build 18363] (x64)
rados benchで計測
ブロックサイズ1M&16スレッドで60秒間書き込み
結果は103.035MB/s
root@pve01:~# rados bench -p Pool01 -t 16 -b 1M 60 write --no-cleanup hints = 1 Maintaining 16 concurrent writes of 1048576 bytes to objects of size 1048576 for up to 60 seconds or 0 objects Object prefix: benchmark_data_pve01_15914 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 100 84 83.9957 84 0.187701 0.174549 2 16 202 186 92.9941 102 0.185979 0.162335 3 16 303 287 95.6603 101 0.269485 0.161929 4 16 406 390 97.4933 103 0.0565424 0.160571 5 16 498 482 96.3933 92 0.085429 0.16317 6 16 593 577 96.1599 95 0.113384 0.164391 7 16 689 673 96.136 96 0.277184 0.164113 8 16 791 775 96.8681 102 0.186503 0.16334 9 16 895 879 97.6597 104 0.21034 0.162371 10 16 1005 989 98.8929 110 0.115867 0.160601 11 16 1105 1089 98.9929 100 0.0573031 0.16 12 16 1213 1197 99.7428 108 0.152295 0.159529 13 16 1310 1294 99.5313 97 0.120351 0.15933 14 16 1418 1402 100.136 108 0.0973263 0.158561 15 16 1524 1508 100.526 106 0.17676 0.158444 16 16 1626 1610 100.618 102 0.110538 0.157991 17 16 1729 1713 100.757 103 0.200339 0.158 18 16 1827 1811 100.604 98 0.113243 0.157961 19 16 1935 1919 100.993 108 0.117182 0.158046 2020-02-12 16:20:18.900688 min lat: 0.0395394 max lat: 0.583621 avg lat: 0.157755 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 20 16 2035 2019 100.943 100 0.165082 0.157755 21 16 2135 2119 100.897 100 0.13679 0.157912 22 16 2244 2228 101.265 109 0.200527 0.157516 23 16 2345 2329 101.253 101 0.27803 0.157047 24 16 2449 2433 101.368 104 0.109864 0.157283 25 16 2549 2533 101.313 100 0.11856 0.157368 26 16 2651 2635 101.339 102 0.172833 0.157193 27 16 2756 2740 101.474 105 0.139274 0.15718 28 16 2861 2845 101.6 105 0.0896752 0.156908 29 16 2964 2948 101.648 103 0.172784 0.156829 30 16 3065 3049 101.626 101 0.131514 0.157043 31 16 3165 3149 101.573 100 0.128491 0.15697 32 16 3269 3253 101.649 104 0.163088 0.15688 33 16 3381 3365 101.962 112 0.0633855 0.156412 34 16 3483 3467 101.963 102 0.0878763 0.156583 35 16 3582 3566 101.878 99 0.141114 0.156607 36 16 3686 3670 101.937 104 0.212936 0.156573 37 16 3784 3768 101.83 98 0.116659 0.156608 38 16 3888 3872 101.887 104 0.139855 0.156637 39 16 3997 3981 102.069 109 0.0750559 0.156332 2020-02-12 16:20:38.902271 min lat: 0.0395394 max lat: 0.583621 avg lat: 0.156352 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 40 16 4099 4083 102.067 102 0.0775366 0.156352 41 16 4202 4186 102.09 103 0.178644 0.156197 42 16 4314 4298 102.326 112 0.12585 0.156047 43 16 4418 4402 102.364 104 0.0803048 0.156113 44 16 4521 4505 102.379 103 0.157185 0.156025 45 16 4626 4610 102.437 105 0.137716 0.15592 46 16 4732 4716 102.514 106 0.153614 0.155633 47 16 4835 4819 102.524 103 0.224552 0.155768 48 16 4939 4923 102.555 104 0.202835 0.155803 49 16 5039 5023 102.502 100 0.105283 0.1557 50 16 5140 5124 102.472 101 0.216646 0.155832 51 16 5250 5234 102.62 110 0.111737 0.155675 52 16 5352 5336 102.608 102 0.148714 0.155559 53 16 5455 5439 102.615 103 0.102017 0.155588 54 16 5559 5543 102.64 104 0.149233 0.15563 55 16 5668 5652 102.756 109 0.153347 0.155445 56 16 5772 5756 102.778 104 0.122354 0.155441 57 16 5876 5860 102.799 104 0.165518 0.15534 58 16 5978 5962 102.785 102 0.0762657 0.155449 59 16 6085 6069 102.856 107 0.088037 0.15527 2020-02-12 16:20:58.903923 min lat: 0.0393852 max lat: 0.583621 avg lat: 0.155236 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 60 16 6194 6178 102.959 109 0.112724 0.155236 Total time run: 60.1251 Total writes made: 6195 Write size: 1048576 Object size: 1048576 Bandwidth (MB/sec): 103.035 Stddev Bandwidth: 4.71193 Max bandwidth (MB/sec): 112 Min bandwidth (MB/sec): 84 Average IOPS: 103 Stddev IOPS: 4.71193 Max IOPS: 112 Min IOPS: 84 Average Latency(s): 0.155273 Stddev Latency(s): 0.0689963 Max latency(s): 0.583621 Min latency(s): 0.0393852 root@pve01:~#
先ほど書き込んだデータを16スレッドでシーケンシャル読み込み
結果は166.674MB/s
root@pve01:~# rados bench -p Pool01 -t 16 60 seq hints = 1 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 199 183 182.964 183 0.0620625 0.0792351 2 16 347 331 165.475 148 0.120293 0.0925071 3 16 510 494 164.643 163 0.107141 0.0937007 4 16 694 678 169.479 184 0.187459 0.0926974 5 16 874 858 171.581 180 0.247978 0.0918876 6 16 1037 1021 170.147 163 0.00402553 0.0922867 7 16 1212 1196 170.839 175 0.470153 0.0923135 8 16 1372 1356 169.482 160 0.0137008 0.0934361 9 16 1534 1518 168.649 162 0.0329692 0.0938391 10 16 1683 1667 166.683 149 0.0166389 0.094688 11 16 1858 1842 167.437 175 0.109609 0.0946932 12 16 2016 2000 166.65 158 0.0846676 0.095084 13 16 2178 2162 166.291 162 0.232895 0.095402 14 16 2349 2333 166.627 171 0.284819 0.0953765 15 16 2502 2486 165.718 153 0.00922137 0.0959302 16 16 2679 2663 166.422 177 0.0138688 0.0953636 17 16 2849 2833 166.631 170 0.0144467 0.095199 18 16 3003 2987 165.929 154 0.448989 0.095553 19 16 3184 3168 166.721 181 0.0144444 0.0953377 2020-02-12 16:22:39.415812 min lat: 0.00278044 max lat: 0.614134 avg lat: 0.0953406 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 20 16 3347 3331 166.534 163 0.406306 0.0953406 21 16 3506 3490 166.175 159 0.0131027 0.0957429 22 16 3685 3669 166.757 179 0.138942 0.0955565 23 16 3833 3817 165.941 148 0.140002 0.0960555 24 16 4003 3987 166.11 170 0.223319 0.0959838 25 16 4167 4151 166.025 164 0.0852787 0.0959481 26 16 4336 4320 166.139 169 0.00305844 0.0959111 27 16 4499 4483 166.022 163 0.0761334 0.0959924 28 16 4662 4646 165.914 163 0.0089712 0.0960629 29 16 4825 4809 165.813 163 0.127818 0.0961544 30 16 5000 4984 166.119 175 0.0119342 0.0959418 31 16 5183 5167 166.663 183 0.277521 0.0956877 32 16 5348 5332 166.61 165 0.0129564 0.0956576 33 16 5517 5501 166.682 169 0.0122473 0.0957369 34 16 5677 5661 166.485 160 0.13254 0.0958199 35 16 5847 5831 166.586 170 0.0928981 0.0957123 36 16 6006 5990 166.375 159 0.0564166 0.0957605 37 16 6185 6169 166.715 179 0.0655861 0.0955326 Total time run: 37.1684 Total reads made: 6195 Read size: 1048576 Object size: 1048576 Bandwidth (MB/sec): 166.674 Average IOPS: 166 Stddev IOPS: 10.0985 Max IOPS: 184 Min IOPS: 148 Average Latency(s): 0.0958543 Max latency(s): 0.614134 Min latency(s): 0.00274573 root@pve01:~#
ブロックサイズ4K&16スレッドで60秒間書き込み
結果は4.22016MB/s 1080IOPS
root@pve01:~# rados bench -p Pool01 -t 16 -b 4K 60 write --no-cleanup hints = 1 Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 60 seconds or 0 objects Object prefix: benchmark_data_pve01_16842 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 0 0 0 0 0 0 - 0 1 16 1052 1036 4.04607 4.04688 0.00200278 0.0150934 2 16 2183 2167 4.2318 4.41797 0.0113091 0.0146819 3 16 3414 3398 4.42395 4.80859 0.00191924 0.0141216 4 16 4422 4406 4.30226 3.9375 0.0239537 0.0144264 5 16 5513 5497 4.29408 4.26172 0.00141641 0.0145298 6 16 6601 6585 4.28658 4.25 0.0021876 0.0145348 7 16 7691 7675 4.28241 4.25781 0.00212365 0.0145637 8 16 8729 8713 4.25388 4.05469 0.00159831 0.0146556 9 16 9757 9741 4.22736 4.01562 0.0254878 0.0147292 10 16 10918 10902 4.25808 4.53516 0.0319383 0.0146237 11 16 11910 11894 4.22318 3.875 0.00150672 0.0147825 12 16 12988 12972 4.22209 4.21094 0.0292272 0.0147692 13 16 14089 14073 4.22811 4.30078 0.00209335 0.0147664 14 16 15104 15088 4.20929 3.96484 0.0463125 0.0148067 15 16 16208 16192 4.21613 4.3125 0.00188022 0.0147966 16 16 17332 17316 4.22702 4.39062 0.0155453 0.0147843 17 16 18392 18376 4.22189 4.14062 0.00119875 0.0147795 18 16 19490 19474 4.22558 4.28906 0.00155527 0.0147641 19 16 20568 20552 4.22479 4.21094 0.0292374 0.0147749 2020-02-12 16:23:54.822925 min lat: 0.00101778 max lat: 0.133509 avg lat: 0.0148034 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 20 16 21620 21604 4.21899 4.10938 0.00173442 0.0148034 21 16 22725 22709 4.2236 4.31641 0.00218956 0.0147921 22 16 23707 23691 4.20595 3.83594 0.00192713 0.0148536 23 16 24746 24730 4.19953 4.05859 0.00137732 0.014869 24 16 25813 25797 4.1982 4.16797 0.0220107 0.014878 25 16 26976 26960 4.21198 4.54297 0.00356637 0.0148337 26 16 28088 28072 4.21703 4.34375 0.00187118 0.0148133 27 16 29280 29264 4.23327 4.65625 0.00143998 0.0147529 28 16 30286 30270 4.22242 3.92969 0.00140776 0.0147881 29 16 31402 31386 4.22713 4.35938 0.00222001 0.0147813 30 16 32436 32420 4.22085 4.03906 0.00176613 0.0147956 31 16 33518 33502 4.22101 4.22656 0.0019674 0.0147932 32 16 34665 34649 4.22911 4.48047 0.00184096 0.0147753 33 16 35748 35732 4.22914 4.23047 0.00138391 0.0147694 34 16 36830 36814 4.22904 4.22656 0.0243034 0.014768 35 16 37915 37899 4.22929 4.23828 0.0362649 0.0147654 36 16 38856 38840 4.21391 3.67578 0.00159328 0.0148268 37 16 39867 39851 4.20674 3.94922 0.0025009 0.0148545 38 16 40942 40926 4.20653 4.19922 0.0331503 0.014849 39 16 42121 42105 4.21675 4.60547 0.0024871 0.0148204 2020-02-12 16:24:14.825012 min lat: 0.00101778 max lat: 0.234147 avg lat: 0.0147795 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 40 16 43322 43306 4.22861 4.69141 0.00223187 0.0147795 41 16 44421 44405 4.23017 4.29297 0.00358861 0.0147725 42 16 45447 45431 4.22486 4.00781 0.0350306 0.0147823 43 16 46518 46502 4.22389 4.18359 0.031724 0.0147871 44 16 47590 47574 4.22304 4.1875 0.0301701 0.01479 45 16 48633 48617 4.21972 4.07422 0.0311379 0.0148055 46 15 49650 49635 4.21443 3.97656 0.0504874 0.0148255 47 16 50766 50750 4.21742 4.35547 0.00169107 0.0148164 48 16 51807 51791 4.21426 4.06641 0.0264488 0.0148224 49 16 52879 52863 4.21371 4.1875 0.00180742 0.014828 50 16 53975 53959 4.21506 4.28125 0.00232123 0.014827 51 16 54986 54970 4.20984 3.94922 0.0311146 0.0148388 52 16 56149 56133 4.21623 4.54297 0.00159122 0.0148188 53 16 57246 57230 4.21752 4.28516 0.00147916 0.0148156 54 16 58317 58301 4.21688 4.18359 0.0242346 0.0148122 55 16 59415 59399 4.21818 4.28906 0.00127209 0.0148112 56 16 60435 60419 4.214 3.98438 0.00389652 0.0148306 57 16 61576 61560 4.21826 4.45703 0.0204465 0.0148152 58 16 62729 62713 4.22317 4.50391 0.0026239 0.0147943 59 16 63775 63759 4.22084 4.08594 0.0255581 0.0147997 2020-02-12 16:24:34.827429 min lat: 0.00101778 max lat: 0.234147 avg lat: 0.0148049 sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 60 16 64836 64820 4.21955 4.14453 0.00171019 0.0148049 Total time run: 60.0141 Total writes made: 64837 Write size: 4096 Object size: 4096 Bandwidth (MB/sec): 4.22016 Stddev Bandwidth: 0.223406 Max bandwidth (MB/sec): 4.80859 Min bandwidth (MB/sec): 3.67578 Average IOPS: 1080 Stddev IOPS: 57.1918 Max IOPS: 1231 Min IOPS: 941 Average Latency(s): 0.0148095 Stddev Latency(s): 0.0183897 Max latency(s): 0.234147 Min latency(s): 0.00101778 root@pve01:~#
クリーンアップ
root@pve01:~# rados -p Pool01 cleanup Removed 64837 objects root@pve01:~#
rbd benchで計測
プール内に20GBでイメージを作成
root@pve01:~# rbd create --size=20G --pool=Pool01 Test01 root@pve01:~#
ブロックサイズ1M&16スレッドで1024MBをシーケンシャル書き込み&読み込みを計測
結果はWrite:13秒 約82MB/s・Read:6秒 約164MB/s
root@pve01:~# rbd bench Test01 --io-type write --io-size 1M --io-threads 16 --io-total 1024M --io-pattern seq --pool=Pool01 bench type write io_size 1048576 io_threads 16 bytes 1073741824 pattern sequential SEC OPS OPS/SEC BYTES/SEC 1 96 97.56 102298249.69 2 176 87.91 92180840.50 3 256 85.21 89350727.08 4 320 82.84 86862715.29 5 400 82.67 86685162.71 6 480 78.05 81838600.35 7 544 76.16 79856995.36 8 608 73.15 76702429.20 9 704 74.65 78274867.14 10 800 79.18 83021146.21 11 880 80.32 84221452.13 12 960 81.57 85529361.93 elapsed: 13 ops: 1024 ops/sec: 78.07 bytes/sec: 81863559.31 root@pve01:~# rbd bench Test01 --io-type read --io-size 1M --io-threads 16 --io-total 1024M --io-pattern seq --pool=Pool01 bench type read io_size 1048576 io_threads 16 bytes 1073741824 pattern sequential SEC OPS OPS/SEC BYTES/SEC 1 160 191.30 200593542.59 2 336 188.03 197164593.37 3 496 176.79 185380270.87 4 624 165.12 173136372.82 5 800 165.99 174048543.93 6 944 158.32 166007434.27 elapsed: 6 ops: 1024 ops/sec: 157.05 bytes/sec: 164681370.03 root@pve01:~#
ブロックサイズ4K&16スレッドで512MBをランダム書き込み&読み込みを計測
結果はWrite:124秒 約4.32MB/s・Read:6秒 約78MB/s
root@pve01:~# rbd bench Test01 --io-type write --io-size 4K --io-threads 16 --io-total 512M --io-pattern rand --pool=Pool01 bench type write io_size 4096 io_threads 16 bytes 536870912 pattern random SEC OPS OPS/SEC BYTES/SEC 1 5616 5631.90 23068258.32 2 6080 3054.05 12509402.95 3 6576 2188.54 8964258.92 4 7088 1770.66 7252607.75 5 7632 1524.69 6245144.82 6 8272 529.50 2168815.99 7 8976 574.59 2353532.42 8 9616 608.96 2494314.27 9 10352 654.88 2682404.58 10 11072 689.09 2822513.53 11 11696 685.89 2809385.57 12 12672 716.27 2933826.52 13 13552 788.45 3229480.62 14 14368 794.92 3255986.03 15 15104 806.39 3302955.25 16 16096 881.39 3610191.77 17 17024 905.14 3707461.08 18 18112 906.91 3714682.91 20 18928 746.55 3057862.76 22 18944 546.38 2237956.54 23 23104 1001.70 4102952.32 24 23472 842.64 3451455.24 25 23504 762.86 3124679.81 26 27584 1471.08 6025550.97 27 27696 1725.52 7067734.81 28 27744 887.51 3635252.52 29 31824 1922.62 7875041.53 30 32272 1595.31 6534404.30 31 32288 923.79 3783825.53 32 36176 1730.58 7088461.07 33 36384 1679.60 6879628.23 34 36784 983.33 4027715.50 35 40512 1845.85 7560581.77 36 40560 1426.18 5841639.10 37 43584 1470.98 6025144.16 38 44272 1481.57 6068495.68 39 44288 1403.12 5747164.38 40 47616 1402.82 5745944.79 41 48048 1738.13 7119382.64 42 48064 830.23 3400621.28 43 51504 1660.21 6800216.54 44 51728 1504.83 6163773.48 45 51760 778.35 3188114.40 46 55024 1463.06 5992698.43 47 55552 1420.31 5817583.61 48 55568 731.98 2998171.30 49 58896 1519.91 6225531.01 50 59856 1603.77 6569061.91 51 61024 1184.81 4852994.02 52 61664 1173.56 4806894.32 54 61680 1113.68 4561644.61 55 66048 1198.37 4908523.75 56 67136 1296.27 5309534.98 57 68240 1216.43 4982503.23 58 69280 1500.37 6145504.34 59 70576 1812.52 7424076.34 60 71904 1176.83 4820287.10 63 73040 805.66 3299993.62 64 73056 655.05 2683077.34 65 77792 1214.59 4974961.62 66 79360 1194.11 4891058.09 67 80224 1175.12 4813292.82 68 81424 1802.20 7381825.59 69 82320 1996.52 8177730.29 70 83728 1159.35 4748715.48 71 84624 1132.51 4638760.82 72 85712 1110.91 4550293.11 73 86928 1103.43 4519643.72 74 87872 1098.08 4497742.76 75 88976 1075.39 4404800.33 76 90160 1087.18 4453073.97 77 91200 1099.34 4502894.27 78 93120 1112.85 4558228.07 79 93392 1116.48 4573122.56 80 94592 1099.43 4503274.66 81 95440 1075.78 4406376.42 82 96576 1076.90 4410998.51 83 97872 1059.75 4340732.83 84 98912 1099.58 4503888.08 85 99984 1070.67 4385472.50 86 100864 1073.62 4397527.56 87 101792 1025.14 4198969.42 88 102816 1008.96 4132706.98 89 103808 979.97 4013943.10 90 104848 1002.46 4106057.32 91 105856 1008.87 4132350.21 92 106832 1023.54 4192419.45 93 107808 997.58 4086104.90 94 108784 993.59 4069755.21 95 109712 957.46 3921769.74 96 110608 948.11 3883449.02 97 111632 963.06 3944713.28 98 112656 971.14 3977775.55 99 113472 942.11 3858863.99 100 114592 981.48 4020137.83 101 115408 958.45 3925808.94 102 116384 948.11 3883449.06 103 117424 937.09 3838321.84 104 118400 980.87 4017661.22 105 119392 970.86 3976628.50 106 120400 971.95 3981088.27 107 121280 979.97 4013943.23 108 122144 963.25 3945464.73 109 123232 955.68 3914462.24 110 124240 968.81 3968236.60 111 125216 993.38 4068891.57 112 126400 932.93 3821273.38 113 126864 945.50 3872751.88 114 128016 958.32 3925263.77 115 129232 929.94 3809029.38 116 130192 948.15 3883638.51 117 130832 984.00 4030450.80 elapsed: 124 ops: 131072 ops/sec: 1055.35 bytes/sec: 4322694.37 root@pve01:~# rbd bench Test01 --io-type read --io-size 4K --io-threads 16 --io-total 512M --io-pattern rand --pool=Pool01 bench type read io_size 4096 io_threads 16 bytes 536870912 pattern random SEC OPS OPS/SEC BYTES/SEC 1 18016 18104.10 74154389.71 2 36848 18468.61 75647438.17 3 56848 18979.64 77740602.49 4 75936 19006.67 77851330.68 5 94608 18939.62 77576678.10 6 113408 19078.06 78143751.40 elapsed: 6 ops: 131072 ops/sec: 19006.63 bytes/sec: 77851140.59 root@pve01:~#
イメージ削除
root@pve01:~# rbd rm --pool=Pool01 Test01 Removing image: 100% complete...done. root@pve01:~#
計測後のメモリ使用状況
ピーク値で5.6GiB
まとめ
多数のVMを動かすことを考えるとまだまだ絶対的な速度な不足してますが
各ノードにSSD1台でこの結果なので既にHDDだけで構成したCephを使っているなら
SSD1台のコストを考えるとコストパフォーマンスはいいと思います
3OSDでシーケンシャル書き込みもGbE上限に近い100MB/s前後を記録してますし
特にVMサーバとしてはランダムアクセスが重要なのですが
WAL高速化のおかげでランダムアクセスの性能が上がり
前記事の結果でもOSインストール時間が半減してたりと
VMの操作感は数値以上で、体感で差を感じられるほどの変化がありました
今回はSATA SSDでしたがNVMe SSDを使えばもっと期待できそうです
DB/WALを別デバイスに設定するデメリットとしては
SSD書き込み回数問題とDB/WALデバイス交換問題があります
SSD書き込み回数問題自体は有名なので説明しませんが
デスクトップ用途とは違ってDB/WALに使う場合には
クラスタの3ノード全てのVM書き込みを処理するのに加えて
fsyncのおかげで通常使用よりもカウントは増えますので
今回みたいなテスト用環境なら問題ないですが後述の通り交換も面倒なので
本稼働する前にVMの規模に応じてSSDのTBWを確認する必要があります
(それもあってCephはサーバ向けSSDが推奨されています)
もう1つがDB/WALデバイス交換問題です
今回の計測環境で言うと各OSDが「HDD&SSDの一部」で構成されているわけですが
システムから見れば2デバイスであっても1つのOSDとしか見ていないので
このどちらかのデバイスが故障した時点でOSD自体の故障と判定されます
つまりDB/WALで共用しているSSDが寿命・故障した際は
該当するSSDを使って登録しているOSD全て同時に全滅します
今回の計測環境は1ノード内を1台のSSDでカバーしていますから
このSSDが壊れるだけで1ノード全滅となります
残りの2ノードにデータがあるので即データ消失ではありませんが
同時期に導入したSSDを他ノードで使っている場合は注意が必要です
更にはS.M.A.R.T.を見て故障する前に事前に交換したくなっても
DB/WALデバイスのみ交換は専用コマンドが用意されていない様子
(現時点ではマニュアルから見つけられませんでした)
ネットで見つけた解決策はSSDを丸ごとクローンするかOSD再登録の2択です
これを紹介されている時点でCephとしては用意していない可能性が高いわけですが
どちらの方法もサーバだけで完結する為には交換用SSDで使う接続ポート(SATAなど)を
用意しておく必要があり手順もなかなか面倒でこの辺りは残念です
というわけでRAIDみたいに故障したデバイスを抜いて交換するだけで済ませたいなら
現時点ではDB/WALを別デバイスで設定するのはテスト用環境以外はオススメしません
メモリ消費については
キャッシュを使わずすぐに同期書き込みする仕様のせいか
シーケンシャルでも案外メモリは思ったより使われずランダム書き込みが一番消費してました
最後にPG:128のプールにシーケンシャル書き込みベンチマーク最中のレイテンシがこちら
HDDx4のレイテンシが50~60だったので全然違うのが確認できます
コメント