Proxmox VE 6.1 + Cephで速度計測:HDDx2-3

Proxmox VE 6.1 + Cephで速度計測 各ノードHDD2~3台の記録

最初に

前記事のCephを拡張して各ノードHDD2~3台のパターンを計測します

計測環境

HDD1台で検証した前記事に詳細を掲載してますのでそちらをご覧ください

Proxmox VE 6.1 + Cephで速度計測:HDDx1 | fefcc.net
https://fefcc.net/archives/1040

今回のCeph構成

各ノードHDD2台の構成はHDD ST1000DM010 1TB 7200rpmを2台をOSD登録
各ノードHDD3台の構成はHDD ST1000DM010 1TB 7200rpmを3台をOSD登録
DB/WALともにOSDと同じHDDを指定

プールは各ノードHDD1台の構成を拡張して使用
Pool01という名でCeph/RBD形式のプール作成
Pool01のPG値は128のまま

既存のCephにOSD追加して6OSDへ

Cephは既存のプールを登録したままOSD追加が可能なので簡単に容量追加ができます
OSD追加の処理中でも(速度は低下しますが)Cephは利用可能なので
プール内のVMデータが維持されるのはもちろんのことVMを停止する必要もありません

OSD追加前の様子がこちら
各ノードHDD1台で合計3OSD

追加したいノードでCeph -> OSD画面へ移動して「作成: OSD」を選択
今回はpve01から追加していきます

pve01だけ2台になっていればOK

Ceph状態を確認するとすぐにリマップ処理が行われているのがわかります

あとはリマップ処理が終わるのを待つのみ

1THDDを1台追加した場合の使用状況推移がこちら
3OSD:948.35GB→4OSD:1.25TBで約300GB増加

このように必ずしも全ノードが同数のOSDでなくても動作はしますが
一部のデータが保存されていないノードが出てくるのであまり推奨はされていません

各ノードHDD2台で合計6OSDになった状態がこちら

使用状況のTotal Sizeは1.89TB(1.72TiB)になり
6OSDなのでクラスタ全体のPG値は250×6=1500になります

各HDD2台:計6OSDでの計測結果

Proxmox起動直後のメモリ使用状況

pve01は1.74GiB、pve01は1.73GiB、pve03は1.59GiB
3OSD構成と比較しても誤差レベルでした

CentOS7のVMから計測

[root@centos7-bench ~]# ./fio_cdm.sh /root
Testing in /root
Configuration: Size:1024m Loops:5 Write Only Zeroes:0
Running Benchmark,  please wait...


Results:

Sequential Read: 148MB/s IOPS=0
Sequential Write: 25MB/s IOPS=0

512KB Read: 73MB/s IOPS=147
512KB Write: 5MB/s IOPS=11

Sequential Q32T1 Read: 142MB/s IOPS=4
Sequential Q32T1 Write: 23MB/s IOPS=0

4KB Read: 4MB/s IOPS=1250
4KB Write: 0MB/s IOPS=74

4KB Q32T1 Read: 4MB/s IOPS=1250
4KB Q32T1 Write: 0MB/s IOPS=80

4KB Q8T8 Read: 37MB/s IOPS=9500
4KB Q8T8 Write: 1MB/s IOPS=264


[root@centos7-bench ~]#

Windows10のVMから計測

------------------------------------------------------------------------------
CrystalDiskMark 7.0.0 x64 (C) 2007-2019 hiyohiyo
                                  Crystal Dew World: https://crystalmark.info/
------------------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

[Read]
Sequential 1MiB (Q=  8, T= 1):   159.601 MB/s [    152.2 IOPS] < 51580.54 us>
Sequential 1MiB (Q=  1, T= 1):   135.920 MB/s [    129.6 IOPS] <  7709.83 us>
    Random 4KiB (Q= 32, T=16):   125.471 MB/s [  30632.6 IOPS] < 16587.69 us>
    Random 4KiB (Q=  1, T= 1):     7.120 MB/s [   1738.3 IOPS] <   574.43 us>

[Write]
Sequential 1MiB (Q=  8, T= 1):    25.794 MB/s [     24.6 IOPS] <311662.26 us>
Sequential 1MiB (Q=  1, T= 1):     7.760 MB/s [      7.4 IOPS] <133764.91 us>
    Random 4KiB (Q= 32, T=16):     1.066 MB/s [    260.3 IOPS] <545741.70 us>
    Random 4KiB (Q=  1, T= 1):     0.158 MB/s [     38.6 IOPS] < 25655.59 us>

[Mix] Read 70%/Write 30%
Sequential 1MiB (Q=  8, T= 1):    50.753 MB/s [     48.4 IOPS] <163766.61 us>
Sequential 1MiB (Q=  1, T= 1):    21.810 MB/s [     20.8 IOPS] < 48025.08 us>
    Random 4KiB (Q= 32, T=16):     2.782 MB/s [    679.2 IOPS] <272531.02 us>
    Random 4KiB (Q=  1, T= 1):     0.429 MB/s [    104.7 IOPS] <  9196.32 us>

Profile: Default
   Test: 1 GiB (x5) [Interval: 5 sec] <DefaultAffinity=DISABLED>
   Date: 2020/02/09 21:21:02
     OS: Windows 10 Professional [10.0 Build 18363] (x64)

rados benchで計測

ブロックサイズ1M&16スレッドで60秒間書き込み
結果は71.73MB/s

root@pve01:~# rados bench -p Pool01 -t 16 -b 1M 60 write --no-cleanup
hints = 1
Maintaining 16 concurrent writes of 1048576 bytes to objects of size 1048576 for up to 60 seconds or 0 objects
Object prefix: benchmark_data_pve01_16205
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16        70        54   53.9973        54    0.166681    0.259034
    2      16       142       126   62.9954        72    0.238393    0.238169
    3      16       223       207   68.9943        81    0.200345    0.224065
    4      16       294       278   69.4939        71    0.253345    0.223467
    5      16       365       349   69.7937        71    0.216933    0.223317
    6      16       447       431   71.8271        82    0.234301    0.220181
    7      16       515       499   71.2795        68    0.198107    0.220523
    8      16       580       564    70.494        65    0.133414    0.222649
    9      16       656       640   71.1051        76    0.201653    0.223094
   10      16       717       701   70.0936        61    0.273147    0.223451
   11      16       796       780   70.9026        79     0.20506    0.223114
   12      16       872       856   71.3267        76    0.150906    0.221818
   13      16       950       934   71.8395        78    0.127223     0.22168
   14      16      1021      1005   71.7789        71    0.299701    0.220322
   15      16      1091      1075     71.66        70     0.19942    0.221493
   16      16      1159      1143   71.4308        68    0.188427    0.221625
   17      16      1235      1219   71.6991        76    0.165404    0.221815
   18      16      1309      1293   71.8266        74    0.275926    0.221312
   19      16      1380      1364   71.7827        71    0.181099     0.22155
2020-02-09 21:28:38.358066 min lat: 0.0665327 max lat: 0.42292 avg lat: 0.22127
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      16      1449      1433   71.6433        69    0.234812     0.22127
   21      16      1525      1509   71.8504        76    0.219319    0.221361
   22      16      1594      1578   71.7206        69     0.15018     0.22164
   23      16      1663      1647    71.602        69    0.158402    0.221779
   24      16      1731      1715   71.4514        68    0.226752    0.222457
   25      16      1798      1782   71.2731        67    0.298069    0.223112
   26      16      1874      1858   71.4544        76    0.212011    0.222688
   27      16      1954      1938   71.7706        80    0.138644    0.222001
   28      16      2019      2003   71.5285        65     0.17797    0.221776
   29      16      2088      2072   71.4411        69    0.244765    0.222723
   30      16      2169      2153   71.7594        81    0.215854    0.222631
   31      16      2233      2217   71.5089        64    0.165947     0.22262
   32      16      2294      2278   71.1803        61    0.127047    0.223701
   33      16      2369      2353   71.2956        75    0.242882    0.223626
   34      16      2442      2426   71.3455        73    0.210134    0.223278
   35      16      2515      2499   71.3925        73    0.243776    0.223325
   36      16      2591      2575   71.5203        76    0.186243    0.223044
   37      16      2660      2644   71.4518        69    0.165274     0.22293
   38      16      2734      2718   71.5185        74    0.255108    0.222551
   39      16      2795      2779   71.2486        61    0.260296    0.223336
2020-02-09 21:28:58.360671 min lat: 0.0620772 max lat: 0.481738 avg lat: 0.224004
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   40      16      2864      2848    71.192        69     0.21892    0.224004
   41      16      2941      2925   71.3335        77    0.179598    0.223554
   42      16      3012      2996   71.3254        71    0.221595    0.223573
   43      16      3094      3078   71.5734        82    0.165702    0.223045
   44      16      3166      3150    71.583        72    0.130623    0.223021
   45      16      3238      3222   71.5921        72    0.166878    0.223004
   46      16      3313      3297    71.666        75    0.242924    0.222809
   47      16      3384      3368   71.6516        71    0.121162    0.222541
   48      16      3457      3441   71.6795        73   0.0991777    0.222378
   49      16      3536      3520   71.8288        79    0.206459     0.22237
   50      16      3605      3589   71.7721        69    0.231927    0.222256
   51      16      3679      3663   71.8157        74     0.22896     0.22226
   52      16      3751      3735   71.8191        72    0.153848    0.222213
   53      16      3823      3807   71.8224        72     0.24974    0.222416
   54      16      3891      3875   71.7514        68    0.156326    0.222408
   55      16      3965      3949   71.7922        74    0.319579    0.222451
   56      16      4033      4017   71.7243        68    0.147869    0.222437
   57      16      4104      4088   71.7116        71    0.302004    0.222709
   58      16      4172      4156   71.6474        68    0.138367    0.222872
   59      16      4245      4229   71.6701        73    0.179308    0.222916
2020-02-09 21:29:18.362748 min lat: 0.0620772 max lat: 0.485794 avg lat: 0.222904
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   60      16      4315      4299   71.6422        70    0.226698    0.222904
Total time run:         60.1701
Total writes made:      4316
Write size:             1048576
Object size:            1048576
Bandwidth (MB/sec):     71.73
Stddev Bandwidth:       5.46119
Max bandwidth (MB/sec): 82
Min bandwidth (MB/sec): 54
Average IOPS:           71
Stddev IOPS:            5.46119
Max IOPS:               82
Min IOPS:               54
Average Latency(s):     0.223058
Stddev Latency(s):      0.0603301
Max latency(s):         0.485794
Min latency(s):         0.0620772
root@pve01:~#

先ほど書き込んだデータを16スレッドでシーケンシャル読み込み
結果は152.708MB/s

root@pve01:~# rados bench -p Pool01 -t 16 60 seq
hints = 1
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16       163       147   146.978       147  0.00664183    0.093832
    2      16       326       310    154.98       163   0.0751217   0.0971496
    3      16       492       476   158.648       166   0.0144741   0.0982731
    4      16       635       619   154.732       143   0.0758463    0.100723
    5      16       794       778   155.583       159   0.0856299    0.101027
    6      16       949       933   155.483       155   0.0483979    0.100902
    7      16      1100      1084   154.841       151   0.0159092    0.101981
    8      16      1260      1244   155.484       160   0.0147741    0.101862
    9      16      1404      1388   154.207       144   0.0591117     0.10216
   10      16      1561      1545   154.485       157    0.148539    0.102394
   11      16      1714      1698   154.349       153  0.00348469    0.102949
   12      16      1872      1856   154.652       158    0.115412    0.102751
   13      16      2025      2009   154.524       153     0.10636     0.10263
   14      16      2172      2156   153.985       147   0.0199766    0.103172
   15      16      2333      2317   154.452       161    0.286223    0.102805
   16      16      2484      2468   154.236       151     0.16923    0.103173
   17      16      2629      2613   153.691       145    0.104399    0.103422
   18      16      2796      2780    154.43       167   0.0513866    0.102821
   19      16      2940      2924    153.88       144    0.145655    0.103415
2020-02-09 21:30:25.276064 min lat: 0.00274203 max lat: 0.427698 avg lat: 0.103573
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      16      3093      3077   153.835       153    0.170269    0.103573
   21      16      3230      3214   153.033       137    0.114728    0.103968
   22      16      3384      3368   153.076       154   0.0599532    0.103977
   23      16      3528      3512   152.681       144   0.0873498    0.104282
   24      16      3680      3664   152.652       152   0.0119661    0.104372
   25      16      3816      3800   151.986       136   0.0446538    0.104877
   26      16      3978      3962    152.37       162    0.329855    0.104561
   27      16      4144      4128   152.874       166   0.0559258    0.104169
   28      16      4299      4283    152.95       155    0.128285     0.10428
Total time run:       28.263
Total reads made:     4316
Read size:            1048576
Object size:          1048576
Bandwidth (MB/sec):   152.708
Average IOPS:         152
Stddev IOPS:          8.49175
Max IOPS:             167
Min IOPS:             136
Average Latency(s):   0.104669
Max latency(s):       0.427698
Min latency(s):       0.00272696
root@pve01:~#

ブロックサイズ4K&16スレッドで60秒間書き込み
結果は1.1907MB/s 304IOPS

root@pve01:~# rados bench -p Pool01 -t 16 -b 4K 60 write --no-cleanup
hints = 1
Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 60 seconds or 0 objects
Object prefix: benchmark_data_pve01_17174
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16       445       429   1.67545   1.67578   0.0500281   0.0356037
    2      16       773       757    1.4783   1.28125   0.0172264   0.0415868
    3      16      1049      1033   1.34484   1.07812   0.0148049   0.0432157
    4      16      1371      1355   1.32305   1.25781   0.0159175   0.0470511
    5      16      1733      1717   1.34123   1.41406    0.051864   0.0464846
    6      16      1995      1979   1.28822   1.02344   0.0101456   0.0465766
    7      16      2310      2294   1.27994   1.23047   0.0157812   0.0486082
    8      16      2584      2568   1.25371   1.07031   0.0164025   0.0496671
    9      16      2914      2898   1.25761   1.28906   0.0165869   0.0496375
   10      16      3198      3182   1.24278   1.10938    0.122408   0.0500057
   11      16      3462      3446   1.22354   1.03125   0.0477646   0.0509933
   12      16      3678      3662   1.19188   0.84375   0.0167316   0.0515104
   13      16      4008      3992   1.19934   1.28906   0.0737472   0.0520085
   14      16      4292      4276   1.19291   1.10938   0.0420287   0.0521601
   15      16      4638      4622   1.20346   1.35156   0.0470655   0.0515758
   16      16      4956      4940   1.20587   1.24219   0.0160688   0.0515776
   17      16      5267      5251   1.20639   1.21484   0.0327591   0.0517119
   18      16      5606      5590   1.21292   1.32422   0.0395547    0.051423
   19      16      5940      5924   1.21774   1.30469   0.0676541   0.0511385
2020-02-09 21:32:20.363004 min lat: 0.00771718 max lat: 0.551061 avg lat: 0.0510562
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      16      6281      6265   1.22345   1.33203   0.0166202   0.0510562
   21      16      6634      6618   1.23085   1.37891   0.0587667   0.0506805
   22      16      6943      6927   1.22976   1.20703   0.0161021   0.0507724
   23      16      7249      7233   1.22826   1.19531   0.0184894    0.050805
   24      16      7539      7523   1.22427   1.13281   0.0198849   0.0509466
   25      16      7840      7824   1.22232   1.17578   0.0415983   0.0510539
   26      16      8130      8114   1.21888   1.13281    0.148898   0.0512602
   27      16      8453      8437   1.22046   1.26172   0.0927031   0.0511313
   28      16      8713      8697   1.21314   1.01562    0.266455   0.0512155
   29      16      9054      9038   1.21723   1.33203   0.0568609    0.051261
   30      16      9395      9379   1.22105   1.33203   0.0697839   0.0510779
   31      16      9736      9720   1.22463   1.33203   0.0162915   0.0509812
   32      16      9958      9942   1.21346  0.867188   0.0270542   0.0514915
   33      16     10245     10229   1.21065   1.12109    0.033327   0.0515395
   34      16     10519     10503   1.20652   1.07031    0.176737   0.0516079
   35      16     10806     10790   1.20407   1.12109   0.0171315   0.0518185
   36      16     11092     11076   1.20165   1.11719   0.0166156   0.0519688
   37      16     11359     11343   1.19736   1.04297   0.0789557   0.0521708
   38      16     11654     11638   1.19617   1.15234   0.0699342   0.0521939
   39      16     11947     11931   1.19484   1.14453   0.0357696   0.0522795
2020-02-09 21:32:40.365527 min lat: 0.00610495 max lat: 0.551061 avg lat: 0.0525676
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   40      16     12182     12166   1.18792  0.917969   0.0169527   0.0525676
   41      16     12499     12483   1.18914   1.23828  0.00873827   0.0524147
   42      16     12823     12807   1.19096   1.26562   0.0712804   0.0523685
   43      16     13111     13095   1.18942     1.125    0.191711   0.0525092
   44      16     13388     13372   1.18698   1.08203   0.0166987   0.0526447
   45      16     13725     13709   1.18985   1.31641   0.0163342   0.0524994
   46      16     14051     14035   1.19166   1.27344   0.0557663    0.052392
   47      16     14267     14251   1.18426   0.84375   0.0795275   0.0527226
   48      16     14576     14560   1.18473   1.20703   0.0201785   0.0527235
   49      16     14882     14866   1.18494   1.19531   0.0422899    0.052728
   50      16     15182     15166   1.18468   1.17188   0.0279977     0.05267
   51      16     15520     15504   1.18733   1.32031   0.0687027   0.0525769
   52      16     15859     15843   1.18996   1.32422   0.0155929   0.0525014
   53      16     16150     16134   1.18895   1.13672    0.161741   0.0525362
   54      16     16405     16389   1.18538  0.996094   0.0723689   0.0526843
   55      16     16692     16676   1.18421   1.12109   0.0265968   0.0527625
   56      16     17029     17013   1.18657   1.31641    0.120635   0.0526498
   57      16     17321     17305   1.18576   1.14062   0.0171969   0.0526781
   58      16     17586     17570   1.18316   1.03516    0.017574   0.0527803
   59      16     17947     17931     1.187   1.41016   0.0264296   0.0526476
2020-02-09 21:33:00.368241 min lat: 0.00610495 max lat: 0.551061 avg lat: 0.052446
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   60      16     18316     18300   1.19124   1.44141   0.0312203    0.052446
Total time run:         60.0879
Total writes made:      18316
Write size:             4096
Object size:            4096
Bandwidth (MB/sec):     1.1907
Stddev Bandwidth:       0.153304
Max bandwidth (MB/sec): 1.67578
Min bandwidth (MB/sec): 0.84375
Average IOPS:           304
Stddev IOPS:            39.2459
Max IOPS:               429
Min IOPS:               216
Average Latency(s):     0.0524896
Stddev Latency(s):      0.047584
Max latency(s):         0.551061
Min latency(s):         0.00610495
root@pve01:~#

クリーンアップ

root@pve01:~# rados -p Pool01 cleanup
Removed 18316 objects
root@pve01:~#

rbd benchで計測

プール内に20GBでイメージを作成

root@pve01:~# rbd create --size=20G --pool=Pool01 Test01
root@pve01:~#

ブロックサイズ1M&16スレッドで1024MBをシーケンシャル書き込み&読み込みを計測
結果はWrite:19秒 約55MB/s・Read:6秒 約157MB/s

root@pve01:~# rbd bench Test01 --io-type write --io-size 1M --io-threads 16 --io-total 1024M --io-pattern seq --pool=Pool01
bench  type write io_size 1048576 io_threads 16 bytes 1073741824 pattern sequential
  SEC       OPS   OPS/SEC   BYTES/SEC
    1        80     80.53  84446717.75
    2       128     65.57  68757163.62
    3       176     62.66  65705279.06
    4       240     60.43  63368264.08
    5       288     60.08  62995693.74
    6       336     52.24  54781217.64
    7       384     52.29  54825973.49
    8       432     51.53  54031394.13
    9       496     52.20  54736534.64
   10       544     51.45  53944526.87
   11       608     53.42  56010352.18
   12       656     55.28  57968437.65
   13       704     51.32  53812209.99
   14       736     48.08  50410900.88
   15       784     48.23  50572993.79
   16       848     47.69  50010178.23
   17       896     46.73  48997511.04
   18       944     51.41  53909857.88
   19      1008     54.22  56858993.61
elapsed:    19  ops:     1024  ops/sec:    52.68  bytes/sec: 55243458.60
root@pve01:~# rbd bench Test01 --io-type read --io-size 1M --io-threads 16 --io-total 1024M --io-pattern seq --pool=Pool01
bench  type read io_size 1048576 io_threads 16 bytes 1073741824 pattern sequential
  SEC       OPS   OPS/SEC   BYTES/SEC
    1       160    181.81  190644857.94
    2       320    168.33  176508866.43
    3       496    172.97  181370243.16
    4       656    169.01  177219163.60
    5       784    159.74  167499475.68
    6       912    150.52  157827688.40
elapsed:     6  ops:     1024  ops/sec:   150.41  bytes/sec: 157713254.77
root@pve01:~#

ブロックサイズ4K&16スレッドで512MBをランダム書き込み&読み込みを計測
結果はWrite:280秒 約1.9MB/s・Read:8秒 約60MB/s

root@pve01:~# rbd bench Test01 --io-type write --io-size 4K --io-threads 16 --io-total 512M --io-pattern rand --pool=Pool01
bench  type write io_size 4096 io_threads 16 bytes 536870912 pattern random
  SEC       OPS   OPS/SEC   BYTES/SEC
    7      5920    811.35  3323288.05
    8      5968    746.49  3057607.02
    9      6080    649.04  2658487.76
   10      6112    611.07  2502950.27
   11      6176    552.84  2264439.46
   50      6240      7.46  30540.67
   51      6256      6.65  27245.36
   52      6272      4.49  18382.65
   53      6304      4.40  18040.24
   54      6336      3.74  15308.86
   55     10752    921.54  3774639.35
   56     10864    977.90  4005483.71
   57     10976    971.87  3980794.26
   58     11072   1077.73  4414371.80
   59     11248    973.80  3988697.13
   60     11376    125.80  515288.79
   61     11520    128.32  525606.56
   62     11632    126.15  516711.68
   63     11744    135.26  554032.79
   64     11920    132.59  543100.80
   65     12064    139.27  570439.08
   66     12176    134.53  551046.09
   67     12352    147.54  604310.94
   68     12528    153.96  630631.19
   69     12672    152.97  626547.13
   70     12848    156.67  641721.43
   71     12976    159.49  653251.23
   72     13184    168.28  689275.18
   73     13360    164.81  675074.58
   74     13536    169.01  692262.29
   75     13664    164.38  673296.20
   76     13840    167.83  687421.30
   77     13984    158.35  648596.23
   78     14128    157.63  645656.78
   79     14304    153.60  629127.97
   80     14480    162.67  666316.25
   81     14704    177.77  728157.37
   82     14960    198.37  812517.07
   83     15184    202.60  829864.69
   84     15328    208.63  854561.22
   85     15520    208.33  853309.42
   86     15744    207.83  851263.11
   87     15904    189.25  775163.52
   88     16128    197.65  809571.95
   89     16416    219.35  898452.24
   90     16688    233.03  954507.96
   91     16944    240.19  983799.49
   92     17264    270.69  1108758.73
   93     17584    285.71  1170252.91
   94     17856    287.53  1177730.57
   95     18160    293.92  1203902.35
   96     18432    291.76  1195034.74
   97     18608    269.87  1105395.52
   98     18880    262.13  1073678.64
   99     19232    276.08  1130806.18
  110     19440     84.68  346833.36
  111     23152    315.92  1294014.59
  116     23248    241.71  990045.17
  117     26464    400.24  1639391.65
  120     26544    336.21  1377097.31
  121     29232    901.63  3693085.40
  123     29472    490.82  2010406.50
  124     29488    747.46  3061610.36
  125     31904    673.92  2760358.67
  127     32000    808.51  3311670.02
  128     34160    687.86  2817493.29
  130     34288    714.52  2926674.01
  131     36240   1039.38  4257299.33
  133     36416    532.06  2179320.01
  134     38272    964.30  3949783.45
  136     38416    531.72  2177921.93
  137     40256    945.47  3872661.98
  138     40416    596.55  2443488.04
  139     41824    990.45  4056876.69
  141     42224    532.89  2182704.69
  142     43840    923.68  3783392.52
  143     43936    533.01  2183206.43
  144     45248    803.17  3289780.06
  146     45552    522.99  2142180.09
  147     47056    847.10  3469725.95
  148     47072    515.95  2113330.25
  149     48496    896.91  3673728.58
  150     48624    536.88  2199063.00
  151     50096    920.56  3770604.11
  152     50224    595.02  2437220.89
  153     51664    972.85  3984810.25
  154     51856    596.57  2443567.81
  155     53120    966.44  3958533.83
  156     53488    608.74  2493403.13
  157     54848   1001.71  4102983.68
  158     55152    608.07  2490663.74
  159     56416   1018.74  4172751.57
  160     56752    630.98  2584477.88
  161     57824    984.53  4032642.54
  163     58352    588.69  2411287.19
  164     59904    905.46  3708773.99
  165     59920    577.82  2366751.54
  166     61408    881.79  3611825.94
  167     61440    590.06  2416893.02
  168     62880    887.82  3636503.46
  169     62912    579.34  2372962.70
  170     64320    900.14  3686966.93
  171     64368    559.32  2290971.24
  172     65776    849.51  3479576.35
  173     65792    568.29  2327716.11
  174     67120    871.56  3569901.49
  175     67200    552.56  2263267.27
  176     68528    876.87  3591669.87
  177     68848    637.33  2610489.40
  178     70176    923.31  3781878.71
  179     70896    757.60  3103142.46
  180     71760    947.60  3881387.34
  181     72528    801.90  3284591.07
  182     73344    885.01  3625019.53
  183     74064    776.96  3182414.29
  184     74864    792.31  3245301.90
  185     75600    779.20  3191598.75
  186     76272    753.00  3084270.64
  187     77072    762.04  3121312.24
  188     77792    725.27  2970711.96
  189     77952    594.29  2434198.04
  190     78944    669.85  2743717.87
  191     79664    678.92  2780873.12
  192     80288    649.42  2660009.35
  193     81040    667.74  2735081.13
  194     81616    767.15  3142236.77
  195     82368    684.23  2802619.98
  196     83104    686.88  2813467.36
  197     83712    686.43  2811609.94
  198     84432    675.68  2767578.10
  199     85056    687.43  2815716.33
  200     85808    680.90  2788963.68
  201     86608    689.20  2822970.33
  202     87248    698.79  2862262.75
  203     88016    700.53  2869361.21
  204     88544    695.91  2850448.32
  205     89232    691.42  2832049.75
  206     89872    662.86  2715062.70
  207     90592    660.33  2704707.65
  208     91120    636.05  2605251.46
  209     91840    641.73  2628511.89
  210     92480    645.96  2645870.04
  211     93168    644.24  2638787.54
  212     93792    657.34  2692447.02
  213     94224    615.37  2520544.63
  214     94880    626.53  2566259.37
  215     95600    624.98  2559928.14
  216     96224    629.31  2577640.98
  217     96816    606.24  2483150.82
  218     97456    646.38  2647580.07
  219     98144    652.78  2673793.74
  220     98864    655.40  2684531.86
  221     99680    673.93  2760409.24
  222    100288    692.17  2835110.20
  223    101040    722.56  2959607.23
  224    101792    718.09  2941296.95
  225    102352    695.36  2848175.22
  226    103040    674.14  2761270.79
  227    103616    662.93  2715359.62
  228    104288    645.45  2643766.79
  229    105008    636.56  2607356.72
  230    105648    660.77  2706502.99
  231    106336    671.54  2750618.77
  232    106928    659.74  2702304.99
  233    107520    647.42  2651822.97
  234    107968    608.54  2492561.59
  235    108672    598.08  2449754.00
  236    109248    586.13  2400808.59
  237    109856    592.70  2427682.44
  238    110560    611.90  2506338.64
  239    111216    636.35  2606472.26
  240    111888    642.67  2632367.33
  241    112496    650.62  2664950.81
  242    113184    658.73  2698160.18
  243    113872    663.97  2719641.35
  244    114496    669.92  2743975.23
  245    115152    659.64  2701893.41
  246    115728    645.87  2645463.68
  247    116448    647.60  2652573.13
  248    117088    638.08  2613564.70
  249    117760    650.70  2665264.86
  250    118304    620.94  2543386.41
  251    118864    625.68  2562788.36
  252    119536    624.58  2558271.20
  253    120240    614.65  2517595.44
  254    120784    605.75  2481161.09
  255    121552    634.36  2598327.03
  256    122080    640.11  2621888.15
  257    122800    645.55  2644178.90
  258    123424    653.51  2676785.25
  259    124128    663.47  2717587.17
  260    124816    682.26  2794516.83
  261    125536    691.73  2833342.36
  262    126192    686.06  2810121.72
  263    126816    684.40  2803317.55
  264    127568    671.33  2749774.44
  265    128240    683.69  2800381.41
  266    128912    669.82  2743592.78
  267    129552    654.70  2681639.41
  268    130016    637.94  2613004.76
  269    130768    656.80  2690236.42
elapsed:   280  ops:   131072  ops/sec:   467.18  bytes/sec: 1913569.87
root@pve01:~# rbd bench Test01 --io-type read --io-size 4K --io-threads 16 --io-total 512M --io-pattern rand --pool=Pool01
bench  type read io_size 4096 io_threads 16 bytes 536870912 pattern random
  SEC       OPS   OPS/SEC   BYTES/SEC
    1     13840  13911.26  56980503.30
    2     28576  14324.25  58672114.86
    3     43312  14461.54  59234479.72
    4     57296  14341.94  58744583.52
    5     72304  14475.17  59290311.54
    6     86832  14597.99  59793366.21
    7    101328  14549.99  59596763.72
    8    116432  14623.59  59898220.84
elapsed:     8  ops:   131072  ops/sec: 14647.78  bytes/sec: 59997298.42
root@pve01:~#

イメージ削除

root@pve01:~# rbd rm --pool=Pool01 Test01
Removing image: 100% complete...done.
root@pve01:~#

計測後のメモリ使用状況

3OSD構成と同様にピーク値で5GiB弱

6OSDから9OSDへ

OSD追加手順は以前と同じなので省略して
各ノードHDD3台で合計9OSDへ拡張した様子がこちら

OSD追加直後の画面を記録してしまったので
一番上のosd.8の使用中と合計値が0になってますが異常ではありません
pve01のosd.6とpve02のosd.7もリマップ中なので右側にレイテンシが表示されてます

使用状況のTotal Sizeは2.84TB(2.58TiB)になり
9OSDなのでクラスタ全体のPG値は250×9=2250になります

このラインからプールに512を割り当て可能(512×3=1536)になるので
プールのpg_numを512に増やしても問題ないが
今回はとりあえずプールのPG値は128のまま計測した

各HDD3台:計9OSDでの計測結果

Proxmox起動直後のメモリ使用状況

pve01は1.80GiB、pve01は1.82GiB、pve03は1.64GiB
6OSD構成に比べて全ノードで少し増加しました

CentOS7のVMから計測

[root@centos7-bench ~]# ./fio_cdm.sh /root
Testing in /root
Configuration: Size:1024m Loops:5 Write Only Zeroes:0
Running Benchmark,  please wait...


Results:

Sequential Read: 148MB/s IOPS=0
Sequential Write: 37MB/s IOPS=0

512KB Read: 73MB/s IOPS=147
512KB Write: 6MB/s IOPS=12

Sequential Q32T1 Read: 146MB/s IOPS=4
Sequential Q32T1 Write: 36MB/s IOPS=1

4KB Read: 4MB/s IOPS=1250
4KB Write: 0MB/s IOPS=86

4KB Q32T1 Read: 3MB/s IOPS=1000
4KB Q32T1 Write: 0MB/s IOPS=78

4KB Q8T8 Read: 33MB/s IOPS=8500
4KB Q8T8 Write: 0MB/s IOPS=126


[root@centos7-bench ~]#

Windows10のVMから計測

------------------------------------------------------------------------------
CrystalDiskMark 7.0.0 x64 (C) 2007-2019 hiyohiyo
                                  Crystal Dew World: https://crystalmark.info/
------------------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

[Read]
Sequential 1MiB (Q=  8, T= 1):   160.408 MB/s [    153.0 IOPS] < 51924.64 us>
Sequential 1MiB (Q=  1, T= 1):   131.922 MB/s [    125.8 IOPS] <  7927.97 us>
    Random 4KiB (Q= 32, T=16):   142.109 MB/s [  34694.6 IOPS] < 14659.91 us>
    Random 4KiB (Q=  1, T= 1):     7.093 MB/s [   1731.7 IOPS] <   576.59 us>

[Write]
Sequential 1MiB (Q=  8, T= 1):    33.138 MB/s [     31.6 IOPS] <247359.28 us>
Sequential 1MiB (Q=  1, T= 1):     9.227 MB/s [      8.8 IOPS] <110682.31 us>
    Random 4KiB (Q= 32, T=16):     1.522 MB/s [    371.6 IOPS] <477505.40 us>
    Random 4KiB (Q=  1, T= 1):     0.158 MB/s [     38.6 IOPS] < 24816.74 us>

[Mix] Read 70%/Write 30%
Sequential 1MiB (Q=  8, T= 1):    54.948 MB/s [     52.4 IOPS] <147167.45 us>
Sequential 1MiB (Q=  1, T= 1):    24.749 MB/s [     23.6 IOPS] < 41885.18 us>
    Random 4KiB (Q= 32, T=16):     3.515 MB/s [    858.2 IOPS] <245029.82 us>
    Random 4KiB (Q=  1, T= 1):     0.469 MB/s [    114.5 IOPS] <  8700.89 us>

Profile: Default
   Test: 1 GiB (x5) [Interval: 5 sec] <DefaultAffinity=DISABLED>
   Date: 2020/02/09 23:46:34
     OS: Windows 10 Professional [10.0 Build 18363] (x64)

rados benchで計測

ブロックサイズ1M&16スレッドで60秒間書き込み
結果は79.0386MB/s

root@pve01:~# rados bench -p Pool01 -t 16 -b 1M 60 write --no-cleanup
hints = 1
Maintaining 16 concurrent writes of 1048576 bytes to objects of size 1048576 for up to 60 seconds or 0 objects
Object prefix: benchmark_data_pve01_23130
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16        88        72   71.9902        72     0.16564    0.215228
    2      16       169       153    76.492        81    0.104767    0.201348
    3      16       259       243   80.9917        90    0.146392    0.193345
    4      16       339       323   80.7417        80    0.329188    0.194219
    5      16       419       403   80.5917        80    0.185352    0.193601
    6      16       505       489   81.4916        86     0.16277    0.194242
    7      16       579       563   80.4203        74    0.206485    0.194572
    8      16       656       640   79.9918        77    0.219053    0.197334
    9      16       733       717   79.6588        77    0.236515    0.198844
   10      16       804       788   78.7922        71    0.185672    0.199997
   11      16       889       873   79.3558        85    0.197623    0.199314
   12      16       967       951   79.2421        78    0.111071    0.199652
   13      16      1049      1033   79.4536        82    0.285896    0.199345
   14      16      1128      1112   79.4206        79    0.191494    0.199678
   15      16      1212      1196   79.7253        84    0.183798      0.1994
   16      16      1291      1275   79.6795        79    0.157768    0.199617
   17      16      1363      1347   79.2274        72    0.195446    0.200265
   18      16      1439      1423   79.0477        76     0.33341    0.201031
   19      16      1511      1495   78.6765        72    0.233796    0.201837
2020-02-09 23:53:07.334454 min lat: 0.0553582 max lat: 0.472811 avg lat: 0.202945
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      16      1581      1565    78.242        70    0.128632    0.202945
   21      16      1656      1640   78.0868        75    0.221697     0.20391
   22      16      1735      1719    78.128        79     0.17049    0.203842
   23      16      1809      1793   77.9482        74    0.190351    0.204148
   24      16      1886      1870   77.9082        77    0.246939    0.204175
   25      16      1965      1949   77.9514        79    0.317125     0.20425
   26      16      2036      2020   77.6837        71    0.162932    0.204374
   27      16      2111      2095   77.5839        75    0.163913    0.205195
   28      16      2186      2170   77.4914        75     0.17129     0.20606
   29      16      2264      2248   77.5086        78    0.196872    0.205964
   30      16      2337      2321   77.3581        73    0.236624    0.205939
   31      16      2416      2400   77.4109        79    0.238466    0.205884
   32      16      2497      2481   77.5229        81    0.163186    0.205636
   33      16      2576      2560   77.5674        79    0.228701    0.205775
   34      16      2645      2629   77.3152        69    0.234228    0.206282
   35      16      2715      2699    77.106        70    0.175623     0.20695
   36      16      2800      2784   77.3251        85    0.297798    0.206251
   37      16      2886      2870   77.5593        86    0.170507    0.205774
   38      16      2970      2954   77.7287        84     0.17443    0.205348
   39      16      3047      3031   77.7096        77    0.107504    0.205023
2020-02-09 23:53:27.336681 min lat: 0.0553582 max lat: 0.540013 avg lat: 0.204824
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   40      16      3130      3114   77.8417        83    0.293891    0.204824
   41      16      3217      3201   78.0649        87     0.17514     0.20445
   42      16      3300      3284   78.1822        83    0.183305    0.204133
   43      16      3385      3369   78.3406        85    0.162006    0.203623
   44      16      3472      3456   78.5372        87    0.210051    0.203274
   45      16      3557      3541   78.6807        85     0.13023    0.203015
   46      16      3634      3618   78.6441        77    0.194956    0.202844
   47      16      3716      3700   78.7153        82    0.392102    0.202842
   48      16      3797      3781   78.7628        81    0.161462    0.202733
   49      16      3886      3870   78.9715        89     0.22085    0.202208
   50      16      3971      3955   79.0919        85    0.218483    0.201975
   51      16      4052      4036   79.1292        81    0.213389    0.201685
   52      16      4132      4116   79.1458        80    0.100467      0.2018
   53      16      4215      4199   79.2183        83    0.336192    0.201748
   54      16      4293      4277   79.1956        78    0.292019    0.201574
   55      16      4370      4354   79.1555        77    0.129269    0.201884
   56      16      4440      4424   78.9918        70    0.300838     0.20196
   57      16      4524      4508   79.0795        84    0.129737    0.201957
   58      16      4602      4586   79.0608        78    0.099031    0.201846
   59      16      4680      4664   79.0427        78    0.196407    0.202116
2020-02-09 23:53:47.338588 min lat: 0.0553582 max lat: 0.540013 avg lat: 0.202367
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   60      16      4755      4739   78.9752        75    0.154389    0.202367
Total time run:         60.1732
Total writes made:      4756
Write size:             1048576
Object size:            1048576
Bandwidth (MB/sec):     79.0386
Stddev Bandwidth:       5.19613
Max bandwidth (MB/sec): 90
Min bandwidth (MB/sec): 69
Average IOPS:           79
Stddev IOPS:            5.19613
Max IOPS:               90
Min IOPS:               69
Average Latency(s):     0.202416
Stddev Latency(s):      0.0661593
Max latency(s):         0.540013
Min latency(s):         0.0553582
root@pve01:~#

先ほど書き込んだデータを16スレッドでシーケンシャル読み込み
結果は152.694MB/s

root@pve01:~# rados bench -p Pool01 -t 16 60 seq
hints = 1
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16       177       161   160.976       161   0.0227583   0.0913365
    2      16       334       318   158.982       157    0.094527   0.0961563
    3      16       474       458   152.652       140    0.139593    0.101263
    4      16       615       599   149.735       141   0.0713489    0.104206
    5      16       779       763   152.585       164   0.0125062    0.102944
    6      16       931       915   152.486       152    0.105617    0.103223
    7      16      1084      1068   152.555       153    0.123844    0.103764
    8      16      1222      1206   150.734       138    0.217225    0.104581
    9      16      1369      1353   150.318       147   0.0783718    0.105103
   10      16      1521      1505   150.483       152   0.0115414     0.10538
   11      16      1679      1663   151.165       158    0.183321    0.105096
   12      16      1822      1806   150.484       143   0.0968277    0.105568
   13      16      1974      1958   150.599       152   0.0133907    0.105537
   14      16      2135      2119    151.34       161   0.0106423    0.104835
   15      16      2289      2273   151.517       154   0.0715924    0.104747
   16      16      2446      2430   151.858       157   0.0540856    0.104543
   17      16      2604      2588   152.218       158   0.0462262    0.104236
   18      16      2764      2748   152.649       160   0.0356474    0.104002
   19      16      2919      2903   152.772       155   0.0176023    0.103778
2020-02-09 23:54:30.363160 min lat: 0.00275774 max lat: 0.520861 avg lat: 0.103969
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      16      3073      3057   152.833       154    0.343562    0.103969
   21      16      3243      3227   153.649       170    0.106917    0.103613
   22      16      3395      3379   153.574       152    0.280434    0.103806
   23      16      3539      3523   153.157       144  0.00316049    0.104076
   24      16      3693      3677   153.191       154   0.0448456    0.103918
   25      16      3834      3818   152.703       141     0.16493    0.104326
   26      16      3986      3970   152.675       152    0.171044    0.104359
   27      16      4158      4142   153.391       172  0.00495006    0.103897
   28      16      4311      4295   153.376       153    0.132305    0.103947
   29      16      4465      4449   153.397       154    0.344788    0.103945
   30      16      4607      4591   153.017       142   0.0582578    0.104127
   31      16      4751      4735   152.726       144   0.0614718    0.104419
Total time run:       31.1472
Total reads made:     4756
Read size:            1048576
Object size:          1048576
Bandwidth (MB/sec):   152.694
Average IOPS:         152
Stddev IOPS:          8.44972
Max IOPS:             172
Min IOPS:             138
Average Latency(s):   0.104689
Max latency(s):       0.520861
Min latency(s):       0.00274626
root@pve01:~#

ブロックサイズ4K&16スレッドで60秒間書き込み
結果は1.32421MB/s 338IOPS

root@pve01:~# rados bench -p Pool01 -t 16 -b 4K 60 write --no-cleanup
hints = 1
Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 60 seconds or 0 objects
Object prefix: benchmark_data_pve01_23791
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16       502       486   1.89822   1.89844   0.0103219   0.0313584
    2      16       902       886   1.73031    1.5625   0.0315711   0.0359186
    3      15      1248      1233   1.60532   1.35547    0.130497   0.0386013
    4      16      1600      1584   1.54673   1.37109   0.0435803    0.040001
    5      16      1947      1931   1.50842   1.35547   0.0624262   0.0408456
    6      16      2280      2264   1.47378   1.30078   0.0119706    0.042123
    7      16      2633      2617   1.46019   1.37891     0.01155   0.0424989
    8      16      2968      2952   1.44123   1.30859   0.0331773   0.0429989
    9      16      3282      3266   1.41736   1.22656   0.0630744   0.0440284
   10      16      3617      3601   1.40646   1.30859   0.0147043   0.0442333
   11      16      3916      3900   1.38478   1.16797   0.0126619   0.0450905
   12      16      4287      4271   1.39013   1.44922   0.0172625   0.0449271
   13      16      4639      4623   1.38896     1.375   0.0241574   0.0449506
   14      16      4985      4969   1.38627   1.35156   0.0157414    0.044883
   15      16      5372      5356   1.39463   1.51172   0.0360859   0.0446777
   16      16      5718      5702   1.39192   1.35156   0.0137916   0.0448113
   17      16      6021      6005   1.37965   1.18359    0.118752   0.0450827
   18      16      6379      6363   1.38069   1.39844   0.0123304   0.0451727
   19      16      6725      6709   1.37915   1.35156   0.0500335   0.0451378
2020-02-09 23:55:35.959600 min lat: 0.00675178 max lat: 0.235167 avg lat: 0.0451564
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      16      7094      7078   1.38225   1.44141   0.0174515   0.0451564
   21      16      7436      7420   1.38004   1.33594    0.041436   0.0452083
   22      16      7779      7763   1.37821   1.33984   0.0170461    0.045289
   23      16      8109      8093   1.37433   1.28906   0.0358381   0.0454448
   24      16      8446      8430   1.37191   1.31641   0.0182906   0.0455392
   25      16      8763      8747   1.36656   1.23828   0.0121734   0.0456746
   26      16      9111      9095   1.36627   1.35938   0.0525774   0.0457022
   27      16      9426      9410   1.36124   1.23047   0.0144335   0.0458341
   28      16      9748      9732   1.35754   1.25781  0.00849643   0.0459322
   29      16     10060     10044   1.35275   1.21875   0.0117517   0.0461174
   30      16     10386     10370    1.3501   1.27344   0.0302381   0.0461997
   31      16     10688     10672   1.34459   1.17969    0.010182   0.0462968
   32      16     11009     10993   1.34176   1.25391   0.0349307   0.0465635
   33      16     11372     11356   1.34406   1.41797   0.0169311    0.046477
   34      16     11688     11672   1.34083   1.23438  0.00974312   0.0465929
   35      16     12043     12027   1.34214   1.38672    0.016703   0.0465258
   36      16     12345     12329   1.33762   1.17969   0.0193157   0.0466212
   37      16     12699     12683   1.33884   1.38281    0.015773   0.0466508
   38      16     13004     12988   1.33495   1.19141    0.108326   0.0467399
   39      16     13363     13347   1.33668   1.40234   0.0279975   0.0466892
2020-02-09 23:55:55.961930 min lat: 0.00641742 max lat: 0.389619 avg lat: 0.047048
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   40      16     13614     13598   1.32777  0.980469   0.0394219    0.047048
   41      16     13953     13937   1.32768   1.32422   0.0335617    0.046999
   42      16     14300     14284   1.32834   1.35547   0.0977868   0.0470156
   43      16     14613     14597   1.32588   1.22266   0.0873738   0.0470861
   44      16     14939     14923   1.32469   1.27344   0.0177279   0.0471329
   45      16     15242     15226   1.32155   1.18359   0.0534714   0.0472605
   46      16     15606     15590   1.32372   1.42188   0.0345469   0.0471906
   47      16     15980     15964   1.32664   1.46094   0.0167525     0.04707
   48      16     16309     16293   1.32577   1.28516    0.106103   0.0470467
   49      16     16635     16619    1.3247   1.27344   0.0133591   0.0471593
   50      16     16999     16983   1.32664   1.42188    0.016411   0.0470897
   51      16     17354     17338   1.32781   1.38672   0.0167391   0.0470441
   52      16     17702     17686   1.32841   1.35938   0.0168721   0.0470385
   53      16     18034     18018   1.32782   1.29688   0.0217958   0.0470402
   54      16     18380     18364   1.32825   1.35156   0.0544518   0.0470161
   55      16     18744     18728   1.32995   1.42188   0.0357476   0.0469673
   56      16     19108     19092   1.33159   1.42188   0.0233487   0.0469232
   57      16     19428     19412   1.33015      1.25   0.0180554   0.0469441
   58      16     19766     19750   1.32998   1.32031   0.0355035   0.0469846
   59      16     20109     20093   1.33015   1.33984   0.0288256   0.0469803
2020-02-09 23:56:15.964641 min lat: 0.00615572 max lat: 0.389619 avg lat: 0.0469429
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   60      16     20456     20440   1.33056   1.35547   0.0334281   0.0469429
Total time run:         60.3457
Total writes made:      20457
Write size:             4096
Object size:            4096
Bandwidth (MB/sec):     1.32421
Stddev Bandwidth:       0.122039
Max bandwidth (MB/sec): 1.89844
Min bandwidth (MB/sec): 0.980469
Average IOPS:           338
Stddev IOPS:            31.2419
Max IOPS:               486
Min IOPS:               251
Average Latency(s):     0.0470925
Stddev Latency(s):      0.0406414
Max latency(s):         0.390156
Min latency(s):         0.00615572
root@pve01:~#

クリーンアップ

root@pve01:~# rados -p Pool01 cleanup
Removed 20457 objects
root@pve01:~#

rbd benchで計測

プール内に20GBでイメージを作成

root@pve01:~# rbd create --size=20G --pool=Pool01 Test01
root@pve01:~#

ブロックサイズ1M&16スレッドで1024MBをシーケンシャル書き込み&読み込みを計測
結果はWrite:17秒 約62MB/s・Read:6秒 約156MB/s

root@pve01:~# rbd bench Test01 --io-type write --io-size 1M --io-threads 16 --io-total 1024M --io-pattern seq --pool=Pool01
bench  type write io_size 1048576 io_threads 16 bytes 1073741824 pattern sequential
  SEC       OPS   OPS/SEC   BYTES/SEC
    1        80    114.83  120407192.69
    2       144     83.86  87928383.94
    3       192     76.24  79947787.80
    4       256     72.80  76339537.06
    5       320     69.65  73033054.96
    6       384     60.17  63095405.77
    7       432     59.26  62136061.40
    8       496     59.94  62846608.74
    9       560     60.32  63245635.85
   10       624     60.22  63145406.16
   11       688     59.89  62797087.41
   12       736     60.75  63700643.50
   13       800     60.70  63649765.46
   14       864     58.01  60831685.30
   15       912     57.60  60396259.42
   16       976     57.69  60493049.36
elapsed:    17  ops:     1024  ops/sec:    59.33  bytes/sec: 62208065.43
root@pve01:~# rbd bench Test01 --io-type read --io-size 1M --io-threads 16 --io-total 1024M --io-pattern seq --pool=Pool01
bench  type read io_size 1048576 io_threads 16 bytes 1073741824 pattern sequential
  SEC       OPS   OPS/SEC   BYTES/SEC
    1       160    194.68  204141904.92
    2       320    175.36  183878817.11
    3       480    168.47  176657387.13
    4       624    163.26  171191359.90
    5       768    159.60  167357678.35
    6       912    148.73  155954790.41
elapsed:     6  ops:     1024  ops/sec:   149.18  bytes/sec: 156426606.95
root@pve01:~#

ブロックサイズ4K&16スレッドで512MBをランダム書き込み&読み込みを計測
結果はWrite:227秒 約2.35MB/s・Read:7秒 約70MB/s

root@pve01:~# rbd bench Test01 --io-type write --io-size 4K --io-threads 16 --io-total 512M --io-pattern rand --pool=Pool01
bench  type write io_size 4096 io_threads 16 bytes 536870912 pattern random
  SEC       OPS   OPS/SEC   BYTES/SEC
    1      5392   4916.23  20136889.34
    2      5488   2352.07  9634095.72
    3      5584   1719.86  7044529.98
    4      5664   1323.36  5420470.89
    5      5744   1132.05  4636857.79
    6      5824     85.37  349688.73
    7      5904     86.02  352334.72
    8      5984     84.03  344192.54
    9      6032     76.22  312197.18
   10      6096     70.79  289974.61
   11      6176     71.02  290910.77
   12      6240     68.40  280174.98
   13      6336     68.59  280934.10
   14      6432     79.74  326626.12
   51      6496      9.63  39436.67
   52      6512      8.05  32987.05
   53      6528      7.01  28709.53
   54     10912    111.81  457990.53
   55     11024    112.33  460086.57
   56     11200   1056.58  4327736.53
   57     11376   1148.22  4703122.34
   58     11520    994.40  4073047.96
   59     11664    150.76  617504.41
   60     11856    164.16  672411.90
   61     12000    161.42  661161.21
   62     12112    149.47  612221.33
   63     12272    155.37  636386.81
   64     12464    158.85  650658.28
   65     12688    163.13  668192.89
   66     12864    170.07  696624.48
   67     13056    185.38  759333.06
   68     13248    196.45  804668.39
   69     13504    211.72  867208.84
   70     13728    215.05  880837.44
   71     13936    212.02  868433.26
   72     14096    210.18  860899.35
   73     14240    197.60  809387.87
   74     14576    210.19  860940.92
   75     14864    228.29  935075.55
   76     15136    244.89  1003076.17
   77     15344    249.00  1019887.52
   78     15616    274.32  1123594.65
   79     15840    255.86  1048018.35
   80     16128    248.42  1017533.55
   81     16336    241.73  990143.74
   82     16608    255.04  1044634.95
   83     17008    277.06  1134849.80
   84     17312    296.29  1213598.44
   85     17568    293.15  1200750.91
   86     18048    343.49  1406936.58
   87     18400    356.11  1458633.46
   88     18848    370.66  1518220.51
   89     19184    375.89  1539661.71
   98     19472    144.81  593138.45
   99     23136    390.05  1597662.89
  104     23360    284.14  1163819.94
  105     26592    450.01  1843248.12
  107     26720    414.15  1696343.59
  108     26736    696.30  2852050.03
  109     29568    641.13  2626076.65
  110     29744    988.82  4050216.68
  112     29760    468.63  1919496.76
  113     32480    991.03  4059249.91
  114     32512    986.99  4042702.52
  115     34784    877.50  3594245.21
  116     35024   1000.73  4099001.36
  117     35056   1037.59  4249973.69
  118     37312    949.66  3889813.31
  119     37440    922.13  3777053.98
  120     39408    927.00  3796997.61
  121     39680    844.68  3459813.69
  122     39696    911.21  3732317.13
  123     41760    903.31  3699948.60
  124     41808    942.17  3859118.95
  125     43712    856.67  3508904.89
  126     43808    862.13  3531299.22
  127     45632   1233.55  5052627.10
  128     45824    721.06  2953464.08
  129     47520   1243.87  5094895.75
  130     47744    785.03  3215470.24
  131     49472   1232.34  5047684.66
  132     50208    884.74  3623902.41
  133     51792   1353.87  5545442.78
  134     52096    912.99  3739589.71
  135     53696   1210.71  4959067.90
  136     54416    991.15  4059766.24
  137     56064   1209.89  4955696.73
  138     57184   1091.47  4470663.20
  139     58384   1256.56  5146882.51
  140     59472   1169.20  4789048.80
  141     60656   1260.07  5161227.29
  142     61456   1068.12  4375015.26
  143     61584    877.87  3595759.64
  144     62960    915.91  3751566.33
  145     64112    932.45  3819325.36
  146     65120    890.64  3648061.70
  147     66112    944.02  3866686.47
  148     67200   1103.75  4520943.26
  149     68256   1061.72  4348811.62
  150     69216   1015.09  4157808.32
  151     69952    979.70  4012850.03
  152     70992    975.20  3994400.40
  153     71984    972.33  3982677.57
  154     73104    943.90  3866221.04
  155     73920    938.52  3844194.37
  156     74736    954.49  3909571.86
  157     75680    932.36  3818927.64
  158     76608    929.24  3806155.72
  159     77536    910.41  3729050.68
  160     78480    914.17  3744445.42
  161     79328    922.07  3776779.65
  162     80304    914.53  3745931.88
  163     81200    917.64  3758665.67
  164     82064    907.03  3715189.44
  165     82928    893.87  3661284.98
  166     83856    903.41  3700364.33
  167     84752    881.82  3611925.93
  168     85536    868.57  3557655.04
  169     86352    849.42  3479241.48
  170     87184    845.09  3461505.64
  171     87888    805.74  3300292.04
  172     88880    846.57  3467569.27
  173     89824    856.21  3507031.07
  174     90480    820.98  3362742.25
  175     91536    873.17  3576514.73
  176     92464    918.12  3760600.08
  177     93520    909.78  3726464.41
  178     94416    916.91  3755664.04
  179     95328    974.25  3990537.72
  180     96368    956.43  3917533.94
  181     97200    945.66  3873437.58
  182     98288    966.72  3959700.80
  183     99200    952.21  3900234.65
  184    100000    936.62  3836415.07
  185    100736    886.34  3630455.63
  186    101696    899.90  3685980.89
  187    102528    850.02  3481678.15
  188    103440    837.92  3432136.52
  189    104256    857.35  3511713.40
  190    105088    865.53  3545217.13
  191    105952    847.79  3472538.96
  192    106928    884.93  3624689.30
  193    107936    916.03  3752070.42
  194    108912    886.50  3631093.99
  195    109312    846.13  3465760.70
  196    110544    914.72  3746686.94
  197    111520    913.99  3743703.94
  198    112400    884.29  3622047.08
  199    113312    916.64  3754574.22
  200    114080    956.64  3918388.30
  201    115040    902.06  3694856.12
  202    115984    889.93  3645154.36
  203    116736    849.51  3479588.42
  204    117632    863.29  3536028.19
  205    118560    896.70  3672864.11
  206    119376    852.85  3493276.90
  207    119472    663.61  2718131.13
  208    121280    919.07  3764516.01
  209    121536    729.43  2987741.34
  210    123168    923.05  3780830.33
  212    123760    717.26  2937896.56
  213    125968   1098.01  4497458.40
  214    126032    709.66  2906767.50
  215    126048    801.69  3283706.35
  217    128416    681.54  2791595.04
  218    130256   1122.29  4596918.61
  220    130912    681.73  2792350.91
elapsed:   227  ops:   131072  ops/sec:   575.49  bytes/sec: 2357201.11
root@pve01:~# rbd bench Test01 --io-type read --io-size 4K --io-threads 16 --io-total 512M --io-pattern rand --pool=Pool01
bench  type read io_size 4096 io_threads 16 bytes 536870912 pattern random
  SEC       OPS   OPS/SEC   BYTES/SEC
    1     15328  15405.25  63099908.29
    2     32944  16512.63  67635724.23
    3     49216  16432.18  67306212.16
    4     66400  16620.22  68076421.46
    5     84080  16832.26  68944939.19
    6    103184  17570.78  71969901.57
    7    120592  17529.18  71799512.60
elapsed:     7  ops:   131072  ops/sec: 17066.26  bytes/sec: 69903382.98
root@pve01:~#

イメージ削除

root@pve01:~# rbd rm --pool=Pool01 Test01
Removing image: 100% complete...done.
root@pve01:~#

計測後のメモリ使用状況

ピーク値で5.6GiBを記録しているので6OSD構成より増加

まとめ

CephはOSDの数が多ければ多いほど高速になると言われている通り
3OSD→6OSD→9OSDと順にパフォーマンスが上がっているのがわかります

ただその速度増加幅がOSD3台追加した結果にしては何とも辛い数値なので
このアプローチは正直コストパフォーマンスは悪いです