Proxmox VE 6.1 + Cephで速度計測:HDDx4

Proxmox VE 6.1 + Cephで速度計測 各ノードHDD4台の記録

最初に

前記事のCephを拡張して各ノードHDD4台のパターンを計測します

3台までは同一の7200rpmHDDでしたが4台目は5400rpmHDDを使用してます
そしてOSDが増えるのでプールのPG値を変更したパターンも計測してみました

計測環境

HDD1台で検証した前記事に詳細を掲載してますのでそちらをご覧ください

Proxmox VE 6.1 + Cephで速度計測:HDDx1 | fefcc.net
https://fefcc.net/archives/1040

今回のCeph構成

各ノードに登録しているOSDは
HDD ST1000DM010 1TB 7200rpmを3台とHDD WD40EZRZ 4TB 5400rpmを1台
DB/WALともにOSDと同じHDDを指定

プールは各ノードHDD1台の構成を拡張して使用
Pool01という名でCeph/RBD形式のプール作成
Pool01のPG値は128のまま

9OSDから12OSDへ拡張

前記事と同様に既存環境へOSDを追加
各ノードHDD4台で合計12OSDへ拡張した結果がこちら

使用状況のTotal Sizeは6.64TB(6.04TiB)になり
12OSDなのでクラスタ全体のPG値は250×12=3000になります

各HDD4台:計12OSD(PG:128)での計測結果

Proxmox起動直後のメモリ使用状況

pve01は1.96GiB、pve01は1.92GiB、pve03は1.76GiB
9OSD構成に比べて更に少し増加しました

CentOS7のVMから計測

[root@centos7-bench ~]# ./fio_cdm.sh /root
Testing in /root
Configuration: Size:1024m Loops:5 Write Only Zeroes:0
Running Benchmark,  please wait...


Results:

Sequential Read: 148MB/s IOPS=0
Sequential Write: 28MB/s IOPS=0

512KB Read: 73MB/s IOPS=147
512KB Write: 6MB/s IOPS=12

Sequential Q32T1 Read: 134MB/s IOPS=4
Sequential Q32T1 Write: 27MB/s IOPS=0

4KB Read: 4MB/s IOPS=1250
4KB Write: 0MB/s IOPS=42

4KB Q32T1 Read: 4MB/s IOPS=1250
4KB Q32T1 Write: 0MB/s IOPS=64

4KB Q8T8 Read: 31MB/s IOPS=8000
4KB Q8T8 Write: 0MB/s IOPS=114


[root@centos7-bench ~]#

Windows10のVMから計測

------------------------------------------------------------------------------
CrystalDiskMark 7.0.0 x64 (C) 2007-2019 hiyohiyo
                                  Crystal Dew World: https://crystalmark.info/
------------------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

[Read]
Sequential 1MiB (Q=  8, T= 1):   168.201 MB/s [    160.4 IOPS] < 49445.50 us>
Sequential 1MiB (Q=  1, T= 1):   143.258 MB/s [    136.6 IOPS] <  7317.25 us>
    Random 4KiB (Q= 32, T=16):   124.382 MB/s [  30366.7 IOPS] < 16706.34 us>
    Random 4KiB (Q=  1, T= 1):     7.295 MB/s [   1781.0 IOPS] <   560.60 us>

[Write]
Sequential 1MiB (Q=  8, T= 1):    24.749 MB/s [     23.6 IOPS] <321442.40 us>
Sequential 1MiB (Q=  1, T= 1):     7.760 MB/s [      7.4 IOPS] <134540.40 us>
    Random 4KiB (Q= 32, T=16):     1.261 MB/s [    307.9 IOPS] <490518.05 us>
    Random 4KiB (Q=  1, T= 1):     0.181 MB/s [     44.2 IOPS] < 22559.02 us>

[Mix] Read 70%/Write 30%
Sequential 1MiB (Q=  8, T= 1):    46.559 MB/s [     44.4 IOPS] <176739.63 us>
Sequential 1MiB (Q=  1, T= 1):    21.600 MB/s [     20.6 IOPS] < 48440.56 us>
    Random 4KiB (Q= 32, T=16):     3.660 MB/s [    893.6 IOPS] <245186.48 us>
    Random 4KiB (Q=  1, T= 1):     0.571 MB/s [    139.4 IOPS] <  7143.01 us>

Profile: Default
   Test: 1 GiB (x5) [Interval: 5 sec] <DefaultAffinity=DISABLED>
   Date: 2020/02/10 1:31:15
     OS: Windows 10 Professional [10.0 Build 18363] (x64)

rados benchで計測

ブロックサイズ1M&16スレッドで60秒間書き込み
結果は78.2615MB/s

root@pve01:~# rados bench -p Pool01 -t 16 -b 1M 60 write --no-cleanup
hints = 1
Maintaining 16 concurrent writes of 1048576 bytes to objects of size 1048576 for up to 60 seconds or 0 objects
Object prefix: benchmark_data_pve01_17801
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16        72        56   55.9972        56    0.165496    0.250112
    2      16       154       138   68.9957        82    0.159491    0.221346
    3      16       238       222   73.9951        84    0.153577    0.212933
    4      16       311       295    73.745        73    0.191293    0.212819
    5      16       397       381   76.1947        86    0.171945    0.205416
    6      16       474       458    76.328        77    0.101957    0.206043
    7      16       554       538   76.8518        80    0.221669    0.205987
    8      16       630       614   76.7445        76    0.132657    0.205616
    9      16       704       688   76.4387        74    0.128847     0.20595
   10      16       783       767   76.6943        79     0.12302    0.206275
   11      16       858       842   76.5397        75    0.246698     0.20653
   12      16       936       920   76.6609        78    0.117597    0.206276
   13      16      1016      1000   76.9173        80    0.231439    0.206097
   14      16      1096      1080   77.1371        80     0.15169    0.205232
   15      16      1177      1161   77.3942        81    0.174891    0.205383
   16      16      1255      1239   77.4318        78    0.169834    0.205156
   17      16      1329      1313   77.2295        74    0.359513    0.205391
   18      16      1414      1398   77.6608        85    0.183955    0.204992
   19      16      1488      1472   77.4676        74    0.110496    0.204643
2020-02-10 01:36:20.618643 min lat: 0.0588456 max lat: 0.507087 avg lat: 0.205063
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      16      1572      1556   77.7937        84    0.198934    0.205063
   21      16      1646      1630   77.6127        74    0.128768    0.204992
   22      16      1732      1716   77.9936        86    0.139595    0.204756
   23      16      1805      1789   77.7763        73    0.123642    0.204774
   24      16      1888      1872   77.9937        83   0.0906235     0.20458
   25      16      1968      1952   78.0737        80    0.172659    0.204392
   26      16      2040      2024   77.8398        72    0.197347    0.204638
   27      16      2117      2101   77.8084        77    0.124578    0.204366
   28      16      2197      2181   77.8865        80    0.225285    0.204436
   29      16      2274      2258   77.8557        77    0.183817    0.204546
   30      16      2355      2339   77.9603        81      0.1724    0.204751
   31      16      2426      2410   77.7356        71    0.122725    0.204995
   32      16      2505      2489   77.7749        79    0.194299    0.205124
   33      16      2580      2564   77.6907        75    0.261757    0.205421
   34      16      2663      2647   77.8467        83   0.0875907    0.204936
   35      16      2743      2727    77.908        80    0.169021    0.204809
   36      16      2821      2805   77.9104        78    0.190399    0.204631
   37      16      2897      2881   77.8586        76    0.209928    0.204845
   38      16      2977      2961   77.9148        80    0.274185    0.204599
   39      16      3056      3040   77.9424        79    0.223076    0.204754
2020-02-10 01:36:40.620237 min lat: 0.0588456 max lat: 0.507087 avg lat: 0.204877
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   40      16      3129      3113   77.8187        73    0.143219    0.204877
   41      16      3208      3192   77.8474        79    0.204169     0.20515
   42      16      3276      3260   77.6127        68    0.300644    0.205497
   43      16      3352      3336   77.5749        76    0.178236    0.205643
   44      16      3436      3420   77.7207        84    0.193754    0.205442
   45      16      3511      3495   77.6601        75    0.203588    0.205416
   46      16      3594      3578    77.776        83    0.155098    0.205125
   47      16      3676      3660   77.8656        82    0.231091    0.205073
   48      16      3753      3737   77.8475        77    0.148205    0.205011
   49      16      3831      3815   77.8504        78     0.25077    0.205146
   50      16      3912      3896   77.9132        81    0.187128     0.20498
   51      16      3992      3976    77.954        80    0.230374    0.204821
   52      16      4070      4054   77.9548        78    0.177336    0.204677
   53      16      4150      4134   77.9932        80    0.226882    0.204675
   54      16      4227      4211   77.9747        77    0.182765    0.204709
   55      16      4313      4297   78.1204        86    0.182894     0.20449
   56      16      4394      4378   78.1717        81    0.263955     0.20436
   57      16      4470      4454   78.1335        76    0.211644    0.204446
   58      16      4547      4531   78.1138        77    0.184792    0.204335
   59      16      4623      4607   78.0778        76    0.133372    0.204439
2020-02-10 01:37:00.622338 min lat: 0.0588456 max lat: 0.507087 avg lat: 0.204404
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   60      16      4706      4690   78.1597        83    0.128414    0.204404
Total time run:         60.1317
Total writes made:      4706
Write size:             1048576
Object size:            1048576
Bandwidth (MB/sec):     78.2615
Stddev Bandwidth:       4.89956
Max bandwidth (MB/sec): 86
Min bandwidth (MB/sec): 56
Average IOPS:           78
Stddev IOPS:            4.89956
Max IOPS:               86
Min IOPS:               56
Average Latency(s):     0.204426
Stddev Latency(s):      0.0579736
Max latency(s):         0.507087
Min latency(s):         0.0588456
root@pve01:~#

先ほど書き込んだデータを16スレッドでシーケンシャル読み込み
結果は151.829MB/s

root@pve01:~# rados bench -p Pool01 -t 16 60 seq
hints = 1
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16       179       163   162.979       163    0.146066   0.0887821
    2      16       334       318   158.977       155    0.157372   0.0956821
    3      16       492       476   158.645       158   0.0338668   0.0973808
    4      16       632       616   153.981       140    0.107825    0.102012
    5      16       782       766   153.182       150   0.0137733     0.10315
    6      16       919       903   150.483       137  0.00345593    0.104152
    7      16      1075      1059   151.269       156   0.0843525    0.104431
    8      16      1233      1217   152.106       158    0.210866    0.103928
    9      16      1393      1377   152.982       160   0.0450929     0.10341
   10      16      1545      1529   152.882       152    0.179122    0.103876
   11      16      1705      1689   153.528       160   0.0693339    0.103302
   12      16      1851      1835   152.899       146  0.00299264    0.103563
   13      16      1996      1980   152.291       145  0.00916029    0.104225
   14      16      2162      2146   153.269       166   0.0903366    0.103499
   15      16      2319      2303   153.517       157    0.013828    0.103621
   16      16      2472      2456   153.483       153   0.0799746    0.103672
   17      16      2613      2597   152.748       141   0.0604253    0.104167
   18      16      2769      2753   152.928       156  0.00954085    0.104141
   19      16      2924      2908   153.037       155  0.00837231    0.103778
2020-02-10 01:37:42.778832 min lat: 0.00279779 max lat: 0.463324 avg lat: 0.104138
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      16      3072      3056   152.784       148    0.273736    0.104138
   21      16      3221      3205   152.604       149    0.018459    0.104435
   22      16      3373      3357   152.576       152   0.0430277    0.104178
   23      16      3520      3504   152.333       147    0.242463    0.104565
   24      16      3681      3665   152.693       161   0.0175145    0.104325
   25      16      3831      3815   152.585       150   0.0103998    0.104412
   26      16      3983      3967   152.562       152    0.262054    0.104396
   27      16      4121      4105   152.022       138    0.345315    0.104779
   28      16      4265      4249   151.735       144    0.201918    0.105102
   29      16      4421      4405   151.881       156   0.0443349     0.10486
   30      16      4575      4559   151.951       154  0.00309545    0.104876
Total time run:       30.9954
Total reads made:     4706
Read size:            1048576
Object size:          1048576
Bandwidth (MB/sec):   151.829
Average IOPS:         151
Stddev IOPS:          7.3975
Max IOPS:             166
Min IOPS:             137
Average Latency(s):   0.105261
Max latency(s):       0.463324
Min latency(s):       0.00279779
root@pve01:~#

ブロックサイズ4K&16スレッドで60秒間書き込み
結果は1.37528MB/s 352IOPS

root@pve01:~# rados bench -p Pool01 -t 16 -b 4K 60 write --no-cleanup
hints = 1
Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 60 seconds or 0 objects
Object prefix: benchmark_data_pve01_19047
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16       412       396   1.54674   1.54688   0.0219786   0.0392206
    2      16       780       764   1.49201    1.4375   0.0700195   0.0417246
    3      16      1117      1101   1.43339   1.31641  0.00792616   0.0431849
    4      16      1466      1450   1.41582   1.36328    0.035507   0.0438561
    5      16      1786      1770   1.38263      1.25   0.0302969   0.0450807
    6      16      2157      2141   1.39371   1.44922    0.101288   0.0446493
    7      16      2503      2487   1.38767   1.35156   0.0948331   0.0449212
    8      16      2831      2815   1.37435   1.28125   0.0226422   0.0452966
    9      16      3180      3164   1.37309   1.36328    0.018731   0.0453217
   10      16      3503      3487   1.36194   1.26172   0.0222619   0.0457924
   11      16      3869      3853   1.36808   1.42969   0.0187413   0.0456161
   12      16      4204      4188   1.36312   1.30859   0.0948958   0.0457482
   13      16      4572      4556   1.36883    1.4375   0.0218892   0.0456125
   14      16      4929      4913   1.37066   1.39453   0.0168531   0.0455475
   15      16      5307      5291   1.37771   1.47656   0.0180912   0.0453181
   16      16      5637      5621   1.37216   1.28906   0.0297729   0.0455184
   17      16      5962      5946   1.36611   1.26953   0.0514815    0.045652
   18      16      6287      6271   1.36073   1.26953   0.0324946   0.0458158
   19      16      6676      6660   1.36909   1.51953   0.0415566   0.0456135
2020-02-10 01:38:47.411570 min lat: 0.00559087 max lat: 0.227312 avg lat: 0.0453695
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      16      7060      7044   1.37562       1.5   0.0341274   0.0453695
   21      16      7454      7438   1.38339   1.53906    0.058937    0.045118
   22      16      7844      7828   1.38974   1.52344   0.0226823   0.0449424
   23      16      8196      8180   1.38909     1.375    0.123987   0.0449226
   24      16      8519      8503   1.38378   1.26172   0.0100244   0.0450275
   25      16      8864      8848   1.38233   1.34766   0.0197402   0.0450687
   26      16      9255      9239   1.38789   1.52734    0.021905   0.0450016
   27      16      9627      9611    1.3903   1.45312    0.010088   0.0448937
   28      16      9966      9950   1.38794   1.32422   0.0246682   0.0449382
   29      16     10271     10255   1.38115   1.19141   0.0720025     0.04513
   30      16     10642     10626   1.38341   1.44922   0.0169297   0.0451129
   31      16     10977     10961   1.38099   1.30859    0.102894   0.0451629
   32      16     11326     11310   1.38044   1.36328   0.0222394   0.0452338
   33      16     11708     11692   1.38382   1.49219   0.0626263   0.0451206
   34      16     12042     12026   1.38149   1.30469   0.0405936   0.0452114
   35      16     12386     12370    1.3804   1.34375   0.0210983   0.0452566
   36      16     12737     12721   1.38014   1.37109   0.0936307   0.0452485
   37      16     13087     13071   1.37978   1.36719   0.0412265   0.0452646
   38      16     13420     13404    1.3777   1.30078   0.0227441   0.0453229
   39      16     13762     13746   1.37663   1.33594    0.088562   0.0453544
2020-02-10 01:39:07.414329 min lat: 0.00559087 max lat: 0.227312 avg lat: 0.0452414
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   40      16     14158     14142   1.38088   1.54688   0.0225702   0.0452414
   41      16     14521     14505   1.38178   1.41797   0.0226132    0.045193
   42      16     14882     14866   1.38245   1.41016   0.0222648   0.0451686
   43      16     15196     15180   1.37882   1.22656   0.0168413   0.0453136
   44      16     15547     15531   1.37864   1.37109   0.0318043   0.0453123
   45      16     15913     15897   1.37977   1.42969   0.0566436   0.0452802
   46      16     16270     16254   1.38009   1.39453   0.0371996   0.0452358
   47      16     16626     16610   1.38031   1.39062   0.0193016   0.0452585
   48      16     16984     16968   1.38068   1.39844   0.0941691    0.045236
   49      16     17309     17293   1.37841   1.26953   0.0146899   0.0452944
   50      16     17644     17628   1.37701   1.30859   0.0366679   0.0453443
   51      16     17975     17959   1.37536   1.29297   0.0226249    0.045377
   52      16     18334     18318   1.37588   1.40234    0.118096   0.0454173
   53      16     18679     18663   1.37534   1.34766   0.0138271   0.0453933
   54      16     19080     19064   1.37888   1.56641     0.03536   0.0453034
   55      16     19456     19440   1.38051   1.46875   0.0302081   0.0452346
   56      16     19753     19737   1.37657   1.16016   0.0206638    0.045377
   57      16     20102     20086   1.37633   1.36328   0.0113361   0.0453688
   58      16     20450     20434   1.37604   1.35938   0.0727052   0.0454109
   59      16     20789     20773   1.37516   1.32422   0.0174198   0.0454218
2020-02-10 01:39:27.416769 min lat: 0.00559087 max lat: 0.227312 avg lat: 0.0453787
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   60      16     21157     21141   1.37619    1.4375   0.0645447   0.0453787
Total time run:         60.093
Total writes made:      21157
Write size:             4096
Object size:            4096
Bandwidth (MB/sec):     1.37528
Stddev Bandwidth:       0.0933819
Max bandwidth (MB/sec): 1.56641
Min bandwidth (MB/sec): 1.16016
Average IOPS:           352
Stddev IOPS:            23.9058
Max IOPS:               401
Min IOPS:               297
Average Latency(s):     0.0454356
Stddev Latency(s):      0.0324911
Max latency(s):         0.227312
Min latency(s):         0.00559087
root@pve01:~#

クリーンアップ

root@pve01:~# rados -p Pool01 cleanup
Removed 21157 objects
root@pve01:~#

rbd benchで計測

プール内に20GBでイメージを作成

root@pve01:~# rbd create --size=20G --pool=Pool01 Test01
root@pve01:~#

ブロックサイズ1M&16スレッドで1024MBをシーケンシャル書き込み&読み込みを計測
結果はWrite:17秒 約63MB/s・Read:6秒 約158MB/s

root@pve01:~# rbd bench Test01 --io-type write --io-size 1M --io-threads 16 --io-total 1024M --io-pattern seq --pool=Pool01
bench  type write io_size 1048576 io_threads 16 bytes 1073741824 pattern sequential
  SEC       OPS   OPS/SEC   BYTES/SEC
    1        80    124.35  130387700.05
    2       144     84.56  88670986.56
    3       192     76.92  80656493.37
    4       256     72.03  75530021.11
    5       304     67.97  71268269.01
    6       368     57.28  60059249.75
    7       432     58.30  61129130.76
    8       496     59.61  62500873.46
    9       560     58.96  61822042.79
   10       624     61.54  64525192.48
   11       688     62.69  65738832.08
   12       752     62.64  65687352.59
   13       800     61.49  64472986.55
   14       864     61.84  64840204.13
   15       928     59.93  62845907.26
   16       976     59.60  62495424.37
elapsed:    17  ops:     1024  ops/sec:    60.06  bytes/sec: 62980947.77
root@pve01:~# rbd bench Test01 --io-type read --io-size 1M --io-threads 16 --io-total 1024M --io-pattern seq --pool=Pool01
bench  type read io_size 1048576 io_threads 16 bytes 1073741824 pattern sequential
  SEC       OPS   OPS/SEC   BYTES/SEC
    1       160    178.13  186783427.20
    2       288    156.37  163968316.84
    3       448    159.12  166844953.46
    4       624    163.26  171189262.14
    5       800    163.98  171946127.33
    6       928    155.59  163143087.99
elapsed:     6  ops:     1024  ops/sec:   151.12  bytes/sec: 158456175.35
root@pve01:~#

ブロックサイズ4K&16スレッドで512MBをランダム書き込み&読み込みを計測
結果はWrite:240秒 約2.23MB/s・Read:8秒 約60MB/s

root@pve01:~# rbd bench Test01 --io-type write --io-size 4K --io-threads 16 --io-total 512M --io-pattern rand --pool=Pool01
bench  type write io_size 4096 io_threads 16 bytes 536870912 pattern random
  SEC       OPS   OPS/SEC   BYTES/SEC
    1      5360   4199.83  17202513.73
    2      5424   2277.97  9330549.02
    3      5472   1714.93  7024359.74
    4      5536   1329.45  5445422.57
    5      5616   1083.87  4439521.07
    6      5680     65.25  267265.03
    7      5744     67.39  276046.27
    8      5808     68.01  278583.21
    9      5872     67.63  277013.09
   10      5936     65.36  267701.73
   11      6000     62.89  257599.78
   12      6064     64.72  265102.68
   13      6112     60.90  249425.93
   14      6160     58.58  239951.36
   15      6240     58.73  240559.17
   16      6304     63.28  259186.98
   58      6368      6.54  26786.29
   60      6384      5.77  23645.20
   61      6400      5.08  20824.52
   62      6416      3.75  15367.68
   63     10752     94.51  387128.72
   64     10944    830.76  3402785.51
   65     11072    916.30  3753181.39
   66     11136    996.59  4082044.62
   67     11296    995.07  4075790.42
   68     11424    138.27  566337.72
   69     11584    128.72  527219.34
   70     11728    140.77  576581.10
   71     11920    157.42  644806.19
   72     12080    158.44  648976.31
   73     12224    158.85  650648.96
   74     12384    160.25  656383.84
   75     12576    164.85  675207.66
   76     12704    152.64  625220.94
   77     12864    155.06  635113.67
   78     13072    170.14  696883.66
   79     13280    179.48  735149.88
   80     13488    186.72  764824.23
   81     13648    192.96  790365.55
   82     13824    195.27  799837.59
   83     14032    190.62  780778.71
   84     14224    187.00  765940.61
   85     14384    180.93  741088.07
   86     14672    205.12  840171.27
   87     14944    222.21  910185.54
   88     15184    231.50  948231.87
   89     15440    246.34  1009022.53
   90     15664    256.40  1050214.07
   91     15968    255.51  1046569.79
   92     16256    263.44  1079063.31
   93     16576    279.06  1143023.67
   94     16896    291.42  1193662.02
   95     17264    317.20  1299233.87
   96     17632    335.47  1374086.50
   97     18016    348.64  1428024.79
   98     18400    366.25  1500161.15
   99     18784    373.11  1528248.20
  100     19216    389.76  1596459.52
  107     19456    155.04  635056.30
  108     19488    134.40  550499.32
  109     23248    439.91  1801869.87
  112     23264    324.44  1328891.85
  113     23296    314.80  1289429.38
  114     26688   1045.65  4282977.82
  116     26704    806.77  3304510.41
  117     26720    432.68  1772275.30
  118     29776   1242.70  5090090.33
  119     29824   1056.27  4326473.56
  120     29840    578.11  2367951.17
  121     32592   1461.71  5987157.98
  122     32752   1147.59  4700546.50
  123     32784    540.21  2212692.07
  124     35360   1132.52  4638817.19
  126     35568    892.18  3654351.78
  127     38000    898.30  3679447.08
  129     38176    765.64  3136053.72
  130     40672   1248.84  5115253.30
  132     40880    678.11  2777518.97
  133     43504   1236.86  5066168.88
  134     43616    709.06  2904318.72
  135     43712    985.01  4034615.20
  136     46544    978.63  4008456.26
  137     46608   1179.52  4831329.76
  138     46672    631.05  2584781.31
  139     49776   1508.26  6177849.51
  140     50480   1141.65  4676215.31
  141     50512    672.97  2756490.70
  142     50848    861.75  3529743.08
  143     54000   1430.07  5857586.07
  145     54656    748.44  3065593.47
  146     54672    793.31  3249382.22
  147     58048   1464.95  6000428.46
  148     58304   1093.85  4480419.19
  150     58320    603.66  2472610.70
  151     62000   1343.03  5501061.35
  153     62912   1078.49  4417496.11
  155     62928    583.15  2388583.74
  156     67152   1231.58  5044534.65
  158     68128   1179.37  4830680.25
  161     68144    606.61  2484676.35
  162     72368   1159.34  4748676.19
  163     73312   1365.54  5593258.33
  164     74208    881.96  3612525.25
  165     75184   1102.46  4515656.53
  166     76192   1653.18  6771419.16
  167     77216    966.47  3958652.51
  168     78160    963.39  3946065.37
  169     79056    956.55  3918034.90
  170     79808    928.48  3803038.97
  171     80384    829.08  3395912.57
  172     81120    778.90  3190369.88
  173     82288    832.22  3408790.42
  174     83360    867.01  3551262.51
  175     84288    898.84  3681648.01
  176     85248    979.03  4010091.32
  177     86224   1007.06  4124927.70
  178     87664    973.17  3986097.06
  179     88112    955.71  3914601.79
  180     89008    941.70  3857209.37
  181     89728    883.94  3620626.14
  182     90416    845.81  3464433.66
  183     91216    784.42  3212985.83
  184     91920    757.33  3102015.34
  185     92704    709.65  2906720.38
  186     93312    720.81  2952426.86
  187     93984    720.49  2951117.32
  188     94752    694.94  2846475.28
  189     95264    632.35  2590103.29
  190     95760    631.38  2586129.38
  191     96448    630.20  2581296.79
  192     97376    669.27  2741331.11
  193     98384    709.35  2905481.63
  194     99008    792.52  3246147.17
  195     99872    829.67  3398318.35
  196    100864    886.71  3631967.66
  197    101632    858.72  3517325.58
  198    102560    862.06  3531007.20
  199    103280    856.42  3507898.71
  200    104112    846.61  3467718.06
  201    104944    821.22  3363727.60
  202    105760    807.48  3307433.22
  203    106560    818.97  3354490.57
  204    107472    835.03  3420265.31
  205    108224    823.02  3371109.92
  206    109072    818.36  3352021.89
  207    109936    855.70  3504958.67
  208    110848    827.77  3390527.32
  209    111536    820.65  3361362.06
  210    112448    846.12  3465704.75
  211    113184    811.33  3323217.20
  212    114000    816.03  3342462.80
  213    114800    819.20  3355458.42
  214    115584    796.82  3263766.00
  215    116320    770.67  3156660.96
  216    117216    793.04  3248307.99
  217    118144    801.82  3284274.59
  218    118688    773.85  3169704.98
  219    119568    809.07  3313931.30
  220    120480    836.65  3426923.61
  224    121808    536.43  2197203.71
  225    121840    442.09  1810789.19
  226    121856    388.79  1592488.70
  227    126320    842.70  3451705.21
  228    127200    839.55  3438780.04
  229    128032   1458.23  5972929.28
  230    128704   1541.72  6314868.59
  231    129552   1555.31  6370559.33
  232    130224    782.65  3205719.80
elapsed:   240  ops:   131072  ops/sec:   545.08  bytes/sec: 2232629.67
root@pve01:~# rbd bench Test01 --io-type read --io-size 4K --io-threads 16 --io-total 512M --io-pattern rand --pool=Pool01
bench  type read io_size 4096 io_threads 16 bytes 536870912 pattern random
  SEC       OPS   OPS/SEC   BYTES/SEC
    1     15344  15421.05  63164628.70
    2     28896  14484.37  59327994.50
    3     43648  14534.69  59534081.70
    4     58608  14670.07  60088593.22
    5     74128  14840.06  60784892.12
    6     89168  14764.19  60474130.83
    7    104336  15087.38  61797903.35
    8    119088  15111.56  61896938.28
elapsed:     8  ops:   131072  ops/sec: 14833.25  bytes/sec: 60756994.84
root@pve01:~#

イメージ削除

root@pve01:~# rbd rm --pool=Pool01 Test01
Removing image: 100% complete...done.
root@pve01:~#

計測後のメモリ使用状況

ピーク値で5.6GiBと速度が変わらないせいか9OSDを同レベルの結果

プールのpg_numを128から512へ変更

ここまでの計測は3OSDをベースにpg_num:128で行ってきました

ここからはPG値を増やすと計測結果にどのような変化があるかを確認していきます

12OSDまで大きくなると1024を試したいところですが全体のPG値が3000なので
PG不足のため今回は512に変更することにします

pg_numを変更する際でもプールは利用可能なのでVMを起動したままでも処理が可能です
しかし現実にはIOPSが極端に低下、もしくは一時的に0になるケースもあるようなので
安全の為にVMを停止してから実行することが推奨されています

また既存プールのpg_numを変更する注意点としては
一度設定すると減らすことはできません

pg_numを変更する場合はProxmoxのシェルからコマンドで対応します
(Pool01の部分に変更したいプール名を指定)

root@pve01:~# ceph osd pool set Pool01 pg_num 512
set pool 4 pg_num to 512
root@pve01:~#

コマンドを実行するとすぐにWEBGUIに反映されます

そして裏ではリマップ処理が行われていました
PGsの数値が128から増えているのがわかります

あとはひたすら待つだけです
完了するといつも通り全て緑になります(PGも512)

各HDD4台:計12OSD(PG:512)での計測結果

Proxmox起動直後のメモリ使用状況

pve01は2.00GiB、pve01は1.91GiB、pve03は1.88GiB
PG:128に比べると減っているノードもあるので誤差の範囲内だと思います

CentOS7のVMから計測

[root@centos7-bench ~]# ./fio_cdm.sh /root
Testing in /root
Configuration: Size:1024m Loops:5 Write Only Zeroes:0
Running Benchmark,  please wait...


Results:

Sequential Read: 167MB/s IOPS=0
Sequential Write: 26MB/s IOPS=0

512KB Read: 89MB/s IOPS=178
512KB Write: 6MB/s IOPS=13

Sequential Q32T1 Read: 135MB/s IOPS=4
Sequential Q32T1 Write: 26MB/s IOPS=0

4KB Read: 3MB/s IOPS=1000
4KB Write: 0MB/s IOPS=56

4KB Q32T1 Read: 3MB/s IOPS=1000
4KB Q32T1 Write: 0MB/s IOPS=56

4KB Q8T8 Read: 31MB/s IOPS=8000
4KB Q8T8 Write: 0MB/s IOPS=96


[root@centos7-bench ~]#

Windows10のVMから計測

------------------------------------------------------------------------------
CrystalDiskMark 7.0.0 x64 (C) 2007-2019 hiyohiyo
                                  Crystal Dew World: https://crystalmark.info/
------------------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

[Read]
Sequential 1MiB (Q=  8, T= 1):   180.573 MB/s [    172.2 IOPS] < 45960.04 us>
Sequential 1MiB (Q=  1, T= 1):   147.644 MB/s [    140.8 IOPS] <  7087.28 us>
    Random 4KiB (Q= 32, T=16):   143.350 MB/s [  34997.6 IOPS] < 14520.04 us>
    Random 4KiB (Q=  1, T= 1):     7.221 MB/s [   1762.9 IOPS] <   566.34 us>

[Write]
Sequential 1MiB (Q=  8, T= 1):    23.280 MB/s [     22.2 IOPS] <353682.79 us>
Sequential 1MiB (Q=  1, T= 1):     7.130 MB/s [      6.8 IOPS] <141298.37 us>
    Random 4KiB (Q= 32, T=16):     1.191 MB/s [    290.8 IOPS] <493584.28 us>
    Random 4KiB (Q=  1, T= 1):     0.180 MB/s [     43.9 IOPS] < 22189.37 us>

[Mix] Read 70%/Write 30%
Sequential 1MiB (Q=  8, T= 1):    48.450 MB/s [     46.2 IOPS] <170087.97 us>
Sequential 1MiB (Q=  1, T= 1):    20.974 MB/s [     20.0 IOPS] < 48525.71 us>
    Random 4KiB (Q= 32, T=16):     4.117 MB/s [   1005.1 IOPS] <246383.30 us>
    Random 4KiB (Q=  1, T= 1):     0.556 MB/s [    135.7 IOPS] <  7308.08 us>

Profile: Default
   Test: 1 GiB (x5) [Interval: 5 sec] <DefaultAffinity=DISABLED>
   Date: 2020/02/10 3:00:00
     OS: Windows 10 Professional [10.0 Build 18363] (x64)

rados benchで計測

ブロックサイズ1M&16スレッドで60秒間書き込み
結果は75.797MB/s

root@pve01:~# rados bench -p Pool01 -t 16 -b 1M 60 write --no-cleanup
hints = 1
Maintaining 16 concurrent writes of 1048576 bytes to objects of size 1048576 for up to 60 seconds or 0 objects
Object prefix: benchmark_data_pve01_16994
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16        61        45   45.0003        45    0.179009    0.313645
    2      16       134       118   58.9981        73    0.146351    0.252613
    3      16       209       193   64.3304        75   0.0970457     0.24142
    4      16       270       254   63.4966        61    0.208428    0.243244
    5      16       344       328   65.5962        74    0.313737     0.23769
    6      16       412       396    65.996        68    0.259696    0.238033
    7      16       478       462   65.9957        66    0.134319    0.236217
    8      16       555       539   67.3704        77    0.217593    0.234341
    9      16       628       612   67.9956        73    0.266892    0.231898
   10      16       703       687   68.6954        75    0.212791    0.229813
   11      16       782       766   69.6316        79    0.123396     0.22836
   12      16       850       834   69.4953        68    0.364031    0.229183
   13      16       920       904   69.5337        70    0.233671    0.228082
   14      16       998       982    70.138        78    0.200291    0.226404
   15      16      1071      1055   70.3284        73    0.216116    0.225258
   16      16      1142      1126   70.3697        71    0.248266    0.225412
   17      16      1218      1202   70.7006        76    0.186702    0.224394
   18      16      1296      1280   71.1058        78    0.310638    0.223829
   19      16      1371      1355   71.3105        75    0.078892    0.223243
2020-02-10 03:09:47.222286 min lat: 0.0667525 max lat: 0.560139 avg lat: 0.222428
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      16      1448      1432   71.5947        77    0.151228    0.222428
   21      16      1524      1508   71.8041        76    0.133552    0.222046
   22      16      1598      1582   71.9036        74    0.200115    0.221884
   23      16      1672      1656   71.9946        74    0.167725    0.221046
   24      16      1751      1735   72.2862        79    0.279627    0.220434
   25      16      1823      1807   72.2745        72     0.17569    0.220422
   26      16      1898      1882   72.3792        75    0.315959     0.21977
   27      16      1978      1962   72.6612        80    0.194949     0.21922
   28      16      2054      2038   72.7802        76    0.177378    0.219383
   29      16      2130      2114   72.8911        76    0.151689     0.21887
   30      16      2207      2191   73.0279        77    0.115531    0.218245
   31      16      2287      2271   73.2526        80    0.138093    0.217774
   32      16      2367      2351   73.4633        80    0.255374       0.217
   33      16      2446      2430   73.6309        79    0.300343    0.216503
   34      16      2518      2502   73.5827        72    0.179602    0.216418
   35      16      2591      2575   73.5659        73    0.100665    0.216636
   36      16      2672      2656   73.7723        81    0.157614    0.216181
   37      16      2751      2735   73.9134        79   0.0901166    0.215817
   38      16      2825      2809   73.9155        74    0.292583    0.215703
   39      16      2906      2890    74.097        81    0.220832    0.215078
2020-02-10 03:10:07.223844 min lat: 0.0667525 max lat: 0.560139 avg lat: 0.214557
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   40      16      2992      2976   74.3943        86    0.322516    0.214557
   41      16      3068      3052   74.4334        76    0.361068    0.214233
   42      16      3152      3136    74.661        84    0.235363    0.214126
   43      16      3226      3210   74.6455        74    0.225949    0.213862
   44      16      3302      3286   74.6761        76    0.166667    0.213423
   45      16      3384      3368   74.8387        82    0.182444     0.21335
   46      16      3462      3446   74.9073        78    0.146709    0.213126
   47      16      3539      3523   74.9517        77    0.165489    0.212974
   48      16      3617      3601    75.015        78    0.129591    0.212937
   49      16      3696      3680   75.0962        79     0.11229    0.212658
   50      16      3776      3760   75.1942        80    0.241682    0.212631
   51      16      3845      3829   75.0727        69    0.155895    0.212475
   52      16      3929      3913   75.2442        84   0.0754374     0.21227
   53      16      4007      3991   75.2961        78    0.146067    0.212184
   54      16      4086      4070   75.3646        79    0.328053    0.211903
   55      16      4164      4148   75.4124        78   0.0978352    0.211787
   56      16      4245      4229   75.5121        81    0.191431    0.211554
   57      16      4324      4308   75.5732        79    0.155393    0.211252
   58      16      4405      4389   75.6666        81   0.0990196    0.211114
   59      16      4482      4466   75.6891        77    0.109239    0.211034
2020-02-10 03:10:27.225386 min lat: 0.0667525 max lat: 0.560139 avg lat: 0.210973
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   60      16      4557      4541   75.6775        75    0.166502    0.210973
Total time run:         60.1343
Total writes made:      4558
Write size:             1048576
Object size:            1048576
Bandwidth (MB/sec):     75.797
Stddev Bandwidth:       6.00986
Max bandwidth (MB/sec): 86
Min bandwidth (MB/sec): 45
Average IOPS:           75
Stddev IOPS:            6.00986
Max IOPS:               86
Min IOPS:               45
Average Latency(s):     0.211022
Stddev Latency(s):      0.0626341
Max latency(s):         0.560139
Min latency(s):         0.0667525
root@pve01:~#

先ほど書き込んだデータを16スレッドでシーケンシャル読み込み
結果は160.485MB/s

root@pve01:~# rados bench -p Pool01 -t 16 60 seq
hints = 1
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16       173       157   156.973       157   0.0574327   0.0939676
    2      16       306       290   144.974       133   0.0604245     0.10393
    3      16       463       447   148.977       157   0.0566775    0.104088
    4      16       649       633   158.228       186    0.292006    0.098729
    5      16       836       820   163.978       187    0.217264   0.0957541
    6      16       996       980   163.313       160   0.0416325   0.0960492
    7      16      1175      1159    165.55       179     0.28379   0.0950192
    8      16      1326      1310    163.73       151   0.0836956   0.0967349
    9      16      1481      1465   162.758       155    0.265245   0.0975261
   10      16      1641      1625   162.481       160   0.0899419    0.097037
   11      16      1788      1772   161.072       147   0.0630237   0.0985439
   12      16      1950      1934   161.148       162    0.252735   0.0984893
   13      16      2094      2078   159.827       144  0.00313608   0.0991515
   14      16      2239      2223   158.767       145    0.353117   0.0998608
   15      16      2402      2386   159.048       163    0.280573   0.0999204
   16      16      2573      2557   159.794       171    0.104912   0.0994631
   17      16      2741      2725   160.276       168   0.0891733    0.099362
   18      16      2904      2888   160.426       163    0.146705   0.0991488
   19      16      3067      3051   160.561       163    0.176214   0.0989968
2020-02-10 03:11:26.189370 min lat: 0.00282085 max lat: 0.441131 avg lat: 0.0994188
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      16      3220      3204   160.182       153   0.0122941   0.0994188
   21      16      3389      3373   160.601       169   0.0552028   0.0990401
   22      16      3548      3532   160.528       159   0.0534175   0.0991134
   23      16      3717      3701   160.896       169    0.117185   0.0989803
   24      16      3878      3862   160.899       161    0.144505   0.0990599
   25      16      4048      4032   161.263       170   0.0781308   0.0988304
   26      16      4203      4187   161.022       155    0.154898   0.0990267
   27      16      4358      4342   160.798       155    0.112947   0.0990486
   28      16      4516      4500   160.698       158   0.0431214   0.0991056
Total time run:       28.4014
Total reads made:     4558
Read size:            1048576
Object size:          1048576
Bandwidth (MB/sec):   160.485
Average IOPS:         160
Stddev IOPS:          11.9221
Max IOPS:             187
Min IOPS:             133
Average Latency(s):   0.0995795
Max latency(s):       0.441131
Min latency(s):       0.00282085
root@pve01:~#

ブロックサイズ4K&16スレッドで60秒間書き込み
結果は1.36973MB/s 350IOPS

root@pve01:~# rados bench -p Pool01 -t 16 -b 4K 60 write --no-cleanup
hints = 1
Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 60 seconds or 0 objects
Object prefix: benchmark_data_pve01_18518
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16       469       453   1.76934   1.76953   0.0285695   0.0341653
    2      16       855       839    1.6385   1.50781   0.0886942   0.0380841
    3      16      1243      1227   1.59751   1.51562   0.0225074   0.0389533
    4      16      1563      1547    1.5106      1.25   0.0192213   0.0407375
    5      16      1900      1884   1.47174   1.31641    0.018719   0.0423084
    6      16      2266      2250    1.4647   1.42969   0.0383777   0.0424462
    7      16      2617      2601    1.4513   1.37109   0.0199866   0.0427303
    8      16      2994      2978   1.45395   1.47266   0.0167087   0.0429101
    9      16      3350      3334    1.4469   1.39062   0.0226296   0.0430536
   10      16      3707      3691   1.44165   1.39453   0.0387178   0.0432296
   11      16      3998      3982   1.41391   1.13672    0.117072    0.044058
   12      16      4348      4332      1.41   1.36719    0.015178   0.0442681
   13      16      4727      4711   1.41541   1.48047    0.030594   0.0441222
   14      16      5062      5046   1.40778   1.30859    0.018518   0.0443593
   15      16      5403      5387   1.40272   1.33203     0.02676   0.0444248
   16      16      5762      5746   1.40269   1.40234   0.0297216   0.0444263
   17      16      6105      6089   1.39897   1.33984   0.0110362    0.044478
   18      16      6461      6445   1.39851   1.39062   0.0190955   0.0446487
   19      16      6827      6811   1.40014   1.42969   0.0473343   0.0444944
2020-02-10 03:12:40.498270 min lat: 0.0059555 max lat: 0.339018 avg lat: 0.0446279
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      16      7177      7161   1.39848   1.36719   0.0718719   0.0446279
   21      16      7523      7507   1.39624   1.35156    0.102084   0.0446394
   22      16      7867      7851   1.39385   1.34375   0.0422839   0.0447869
   23      16      8192      8176   1.38843   1.26953   0.0371815   0.0449834
   24      16      8509      8493   1.38217   1.23828    0.042311   0.0451412
   25      16      8837      8821   1.37813   1.28125   0.0889875   0.0453092
   26      16      9172      9156   1.37545   1.30859   0.0274299   0.0454203
   27      16      9534      9518   1.37687   1.41406   0.0723538   0.0452903
   28      16      9896      9880    1.3782   1.41406    0.106638   0.0453199
   29      16     10270     10254   1.38104   1.46094   0.0149348   0.0452156
   30      16     10666     10650   1.38656   1.54688   0.0541185   0.0450589
   31      16     11049     11033   1.39009   1.49609   0.0383536   0.0449283
   32      16     11405     11389    1.3901   1.39062   0.0455414   0.0449231
   33      16     11761     11745   1.39011   1.39062   0.0313031    0.044905
   34      16     12103     12087   1.38851   1.33594   0.0588919    0.044975
   35      16     12468     12452   1.38957   1.42578   0.0220484    0.044962
   36      16     12798     12782   1.38677   1.28906     0.12836   0.0450133
   37      16     13147     13131   1.38613   1.36328   0.0262662   0.0450799
   38      16     13498     13482   1.38573   1.37109    0.022454   0.0450754
   39      16     13852     13836   1.38565   1.38281     0.03527   0.0450479
2020-02-10 03:13:00.500887 min lat: 0.0059555 max lat: 0.339018 avg lat: 0.045066
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   40      16     14208     14192   1.38577   1.39062   0.0833164    0.045066
   41      16     14553     14537   1.38484   1.34766    0.082726   0.0450829
   42      16     14905     14889    1.3846     1.375   0.0169701   0.0451043
   43      16     15240     15224   1.38283   1.30859   0.0148114   0.0451582
   44      16     15579     15563   1.38149   1.32422   0.0389806   0.0452232
   45      16     15951     15935   1.38308   1.45312   0.0694952   0.0451314
   46      16     16297     16281   1.38239   1.35156   0.0328218   0.0451822
   47      16     16652     16636   1.38248   1.38672   0.0203276   0.0451807
   48      16     16979     16963   1.38029   1.27734    0.134524   0.0452518
   49      16     17307     17291   1.37827   1.28125   0.0752695   0.0453283
   50      16     17678     17662   1.37968   1.44922  0.00985989   0.0452673
   51      16     18021     18005    1.3789   1.33984   0.0225295   0.0453002
   52      16     18359     18343   1.37777   1.32031   0.0133397    0.045347
   53      16     18691     18675   1.37623   1.29688   0.0238378   0.0453851
   54      16     19063     19047   1.37765   1.45312   0.0170903   0.0453498
   55      16     19442     19426   1.37952   1.48047   0.0219286   0.0452925
   56      16     19764     19748   1.37735   1.25781   0.0865308   0.0453386
   57      16     20075     20059   1.37449   1.21484    0.126323   0.0454297
   58      16     20419     20403   1.37396   1.34375   0.0221119   0.0454755
   59      16     20736     20720   1.37165   1.23828   0.0220878   0.0455134
2020-02-10 03:13:20.503382 min lat: 0.00556043 max lat: 0.339018 avg lat: 0.0455463
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   60      16     21068     21052   1.37041   1.29688   0.0190516   0.0455463
Total time run:         60.0853
Total writes made:      21069
Write size:             4096
Object size:            4096
Bandwidth (MB/sec):     1.36973
Stddev Bandwidth:       0.0962262
Max bandwidth (MB/sec): 1.76953
Min bandwidth (MB/sec): 1.13672
Average IOPS:           350
Stddev IOPS:            24.6339
Max IOPS:               453
Min IOPS:               291
Average Latency(s):     0.0456012
Stddev Latency(s):      0.0322419
Max latency(s):         0.339018
Min latency(s):         0.00556043
root@pve01:~#

クリーンアップ

root@pve01:~# rados -p Pool01 cleanup
Removed 21069 objects
root@pve01:~#

rbd benchで計測

プール内に20GBでイメージを作成

root@pve01:~# rbd create --size=20G --pool=Pool01 Test01
root@pve01:~#

ブロックサイズ1M&16スレッドで1024MBをシーケンシャル書き込み&読み込みを計測
結果はWrite:19秒 約54MB/s・Read:6秒 約174MB/s

root@pve01:~# rbd bench Test01 --io-type write --io-size 1M --io-threads 16 --io-total 1024M --io-pattern seq --pool=Pool01
bench  type write io_size 1048576 io_threads 16 bytes 1073741824 pattern sequential
  SEC       OPS   OPS/SEC   BYTES/SEC
    1        80     78.69  82509278.47
    2       112     63.62  66707302.71
    3       160     58.20  61027034.77
    4       224     57.75  60551804.62
    5       272     56.65  59398890.68
    6       320     49.87  52297028.40
    7       384     51.17  53650691.23
    8       432     52.39  54932025.24
    9       480     50.55  53007541.22
   10       528     50.04  52468762.55
   11       576     50.71  53175552.61
   12       624     50.85  53316377.80
   13       672     49.92  52340537.36
   14       736     51.82  54338095.34
   15       784     53.11  55690911.07
   16       832     51.82  54338095.49
   17       896     53.12  55704507.66
   18       944     53.75  56365035.50
   19      1008     52.79  55358517.11
elapsed:    19  ops:     1024  ops/sec:    51.99  bytes/sec: 54514660.87
root@pve01:~# rbd bench Test01 --io-type read --io-size 1M --io-threads 16 --io-total 1024M --io-pattern seq --pool=Pool01
bench  type read io_size 1048576 io_threads 16 bytes 1073741824 pattern sequential
  SEC       OPS   OPS/SEC   BYTES/SEC
    1       160    191.30  200593221.87
    2       320    173.91  182357474.58
    3       496    172.50  180882885.92
    4       656    171.95  180304320.03
    5       832    173.48  181909788.83
    6      1008    170.28  178549206.57
elapsed:     6  ops:     1024  ops/sec:   166.34  bytes/sec: 174418585.64
root@pve01:~#

ブロックサイズ4K&16スレッドで512MBをランダム書き込み&読み込みを計測
結果はWrite:263秒 約2.03MB/s・Read:8秒 約62MB/s

root@pve01:~# rbd bench Test01 --io-type write --io-size 4K --io-threads 16 --io-total 512M --io-pattern rand --pool=Pool01
bench  type write io_size 4096 io_threads 16 bytes 536870912 pattern random
  SEC       OPS   OPS/SEC   BYTES/SEC
    1      5344   5296.34  21693805.32
    2      5408   2690.42  11019975.14
    3      5472   1802.86  7384496.97
    4      5552   1386.43  5678805.38
    5      5632   1108.30  4539591.05
    6      5712     72.96  298830.01
    7      5792     75.29  308398.68
    8      5872     77.94  319245.52
    9      5920     72.84  298356.80
   10      6016     75.77  310345.95
   11      6112     77.16  316043.21
   12      6192     78.93  323277.03
   13      6256     79.01  323628.25
   14      6336     83.00  339964.63
   15      6432     80.81  330983.44
   54      6480      8.53  34952.66
   55      6496      7.01  28706.19
   56      6512      5.92  24238.45
   57     10832    104.48  427943.32
   58     10976    106.13  434694.07
   59     11104    964.92  3952323.75
   60     11248   1056.92  4329139.95
   61     11376   1024.41  4195989.11
   62     11552    144.34  591231.48
   63     11712    148.86  609748.65
   64     11840    151.44  620287.52
   65     11984    146.96  601956.34
   66     12128    150.64  617013.64
   67     12272    144.81  593134.08
   68     12416    138.14  565841.43
   69     12544    140.46  575324.80
   70     12736    149.20  611137.32
   71     12880    151.49  620493.98
   72     13024    150.04  614551.50
   73     13216    163.13  668176.24
   74     13360    162.03  663675.74
   75     13552    164.91  675479.19
   76     13728    167.98  688062.69
   77     13888    175.89  720455.05
   78     14032    165.05  676025.69
   79     14208    170.96  700270.26
   80     14416    173.77  711760.91
   81     14624    178.91  732816.43
   82     14864    193.49  792549.23
   83     15120    214.85  880008.18
   84     15344    227.56  932084.45
   85     15536    224.71  920431.56
   86     15760    229.77  941133.82
   87     16032    232.85  953755.00
   88     16272    233.38  955936.05
   89     16528    237.17  971468.31
   90     16800    252.59  1034620.99
   91     17040    256.20  1049395.15
   92     17360    263.49  1079242.53
   93     17696    285.94  1171202.96
   94     18064    307.19  1258266.77
   95     18368    313.59  1284480.66
   96     18656    322.42  1320631.98
   97     18960    317.71  1301325.02
   98     19296    319.74  1309646.84
  107     19392     97.27  398431.18
  108     19440     82.41  337548.19
  109     23184    348.41  1427079.73
  111     23200    295.18  1209043.66
  113     23216    257.75  1055760.69
  114     26432   1097.92  4497080.60
  117     26496    709.56  2906357.16
  118     26512    369.77  1514580.41
  119     29296    811.92  3325616.93
  121     29328    721.76  2956337.59
  122     31696    650.83  2665792.04
  124     31792    781.11  3199409.38
  125     33888   1035.93  4243189.68
  127     34000    538.70  2206506.03
  128     35888   1037.95  4251461.92
  129     36000    568.40  2328161.97
  130     36016    756.97  3100567.89
  131     37728    649.51  2660408.54
  132     37776    773.12  3166706.88
  133     39312    669.26  2741288.57
  134     39584    835.03  3420265.54
  135     39968    845.87  3464700.02
  136     41248    703.42  2881223.18
  137     41520    704.28  2884711.63
  138     41536    453.13  1856016.22
  139     42944    668.24  2737130.77
  140     42976    598.24  2450383.74
  141     44320    616.86  2526640.24
  142     45392    948.07  3883301.90
  143     45600    722.60  2959783.39
  144     46720    758.22  3105661.97
  145     46848    727.81  2981091.17
  146     48000    732.47  3000196.71
  147     48064    536.54  2197650.64
  148     49232    727.55  2980044.89
  149     50192    698.30  2860224.62
  150     50432    729.63  2988554.55
  151     51520    691.27  2831429.90
  152     52352    852.81  3493100.27
  153     52768    799.26  3273773.91
  154     53872    734.22  3007379.97
  155     54784    915.81  3751146.29
  156     55568    826.11  3383732.10
  157     56384    799.98  3276736.62
  158     57232    881.50  3610622.11
  159     57952    806.95  3305252.53
  160     58640    768.73  3148698.27
  161     59408    766.76  3140642.14
  162     60112    754.64  3091010.67
  163     60864    745.47  3053445.06
  164     61616    739.89  3030584.37
  165     62336    739.78  3030128.76
  166     63072    739.29  3028138.39
  167     63856    745.21  3052374.11
  168     64656    740.61  3033541.35
  169     65312    738.59  3025284.44
  170     65984    727.26  2978851.50
  171     66608    683.67  2800304.72
  172     67312    694.52  2844755.30
  173     68208    707.56  2898149.56
  174     68912    719.99  2949063.01
  175     69648    724.67  2968246.44
  176     70416    781.59  3201409.41
  177     71136    766.01  3137580.40
  178     71952    758.49  3106792.49
  179     72736    763.56  3127556.19
  180     73408    766.08  3137869.29
  181     74192    751.58  3078462.83
  182     75008    770.69  3156728.83
  183     75760    767.73  3144610.23
  184     76384    729.00  2985995.08
  185     77280    774.39  3171881.14
  186     78128    798.69  3271417.34
  187     78864    774.28  3171460.04
  188     79776    799.98  3276736.73
  189     80496    820.42  3360420.35
  190     80752    672.33  2753880.21
  191     81760    702.77  2878557.43
  192     82592    742.02  3039329.86
  193     83488    750.79  3075254.55
  194     84336    764.32  3130640.20
  195     85232    921.79  3775663.74
  196     85968    867.25  3552274.39
  197     86896    866.33  3548479.02
  198     87680    838.38  3434020.12
  199     88592    841.76  3447832.19
  200     89328    823.80  3374274.38
  201     90160    831.73  3406766.01
  202     91008    809.43  3315438.39
  203     91728    806.36  3302846.23
  204     92560    815.10  3338663.61
  205     93424    809.47  3315591.37
  206     94176    807.71  3308370.61
  207     94976    791.68  3242740.32
  208     95840    811.99  3325913.73
  209     96592    798.08  3268953.57
  210     97424    802.55  3287256.02
  211     98320    826.14  3383870.94
  212     99088    831.03  3403885.85
  213     99904    822.66  3369599.80
  214    100704    824.36  3376589.26
  215    101488    819.34  3356012.71
  216    102288    796.77  3263577.23
  217    103152    820.00  3358721.36
  218    103984    806.31  3302639.88
  219    104816    830.36  3401136.37
  220    105696    839.57  3438873.85
  221    106528    841.25  3445774.85
  222    107392    847.98  3473341.05
  223    108144    844.83  3460404.46
  224    108992    829.87  3399158.65
  225    109824    823.61  3373496.04
  226    110528    811.01  3321914.89
  227    111392    789.87  3235324.70
  228    112240    814.62  3336693.06
  229    112752    755.01  3092502.66
  230    113632    764.03  3129467.79
  231    114512    796.15  3261021.10
  232    115408    784.97  3215250.02
  233    116240    805.79  3300500.48
  234    116976    842.09  3449196.71
  235    117904    855.07  3502356.87
  236    118688    836.52  3426395.54
  237    119616    844.96  3460971.09
  238    120304    808.90  3313261.02
  239    121056    820.58  3361093.75
  240    122000    821.15  3363450.86
  241    122720    781.99  3203016.67
  242    123360    765.32  3134736.04
  243    124096    758.39  3106346.58
  244    125088    801.89  3284557.28
  245    125920    780.24  3195862.03
  246    126624    805.27  3298365.54
  247    127520    840.73  3443620.04
  248    128336    845.28  3462261.90
  249    128928    728.92  2985637.29
  250    129360    685.25  2806766.69
  251    130176    709.82  2907416.46
  252    131040    691.27  2831430.18
elapsed:   263  ops:   131072  ops/sec:   497.24  bytes/sec: 2036710.18
root@pve01:~# rbd bench Test01 --io-type read --io-size 4K --io-threads 16 --io-total 512M --io-pattern rand --pool=Pool01
bench  type read io_size 4096 io_threads 16 bytes 536870912 pattern random
  SEC       OPS   OPS/SEC   BYTES/SEC
    1     14720  14794.90  60599900.69
    2     30272  15174.06  62152943.04
    3     41664  13911.62  56981980.51
    4     57760  14458.18  59220712.78
    5     74512  14905.31  61052169.50
    6     91168  15289.31  62625003.46
    7    106512  15247.71  62454613.16
    8    121744  16015.69  65600281.00
elapsed:     8  ops:   131072  ops/sec: 15247.73  bytes/sec: 62454704.64
root@pve01:~#

イメージ削除

root@pve01:~# rbd rm --pool=Pool01 Test01
Removing image: 100% complete...done.
root@pve01:~#

計測後のメモリ使用状況

5.6GiBぐらいでPG:128に比べて特に変化なし

まとめ

まず5400rpmのHDDを追加したパターンについてです

そもそもCephは同期書き込みな仕様上、一番低速なデバイスに引っ張られやすく
HDDとSSDの混在は推奨されていません

ただ今回使っているST1000DM010とWD40EZRZは性能だけで見れば
全ての項目でST1000DM010の方が高性能というわけではなく(特に4K周り)
単純にOSD増加の速度アップ分も期待して案外大丈夫かなと思いきや
特に書き込み速度がかなり落ちてしまいました

理由は書き込みベンチマーク中のOSD画面を見れば
一番右の「Apply/Commit Latency(ms)」の項目で3.64TB表記のWD40EZRZだけ
桁違いな数値が出ているのでレイテンシ問題だと思われます

ST1000DM010だけで構成すればレイテンシは150msぐらいは記録することから
今回の構成ではWD40EZRZの反応が遅くボトルネックになってしまっていて
ST1000DM010に負荷がかかりきっていないことが予想されます

Cephは10GbEケーブルすら光ファイバー推奨なぐらいレイテンシが重要な要素なので
単純性能に加えて回転数の影響から発生するレイテンシを気にする必要がある様子

ただこの程度の落ち具合で大容量化が可能という見方もできると思います

次にpg_numを512に変更したパターンについてです

書き込み速度が若干低下してしまってますが
それよりも明らかな読み込み速度アップが確認できました

特にCentOS7での計測時がわかりやすく
pg_num:128の時は3~12OSD全てのケースでSequential Read: 148MB/sになっていて
明らかに頭打ちしている様子でしたが
pg_num:512にするだけでSequential Read: 167MB/sに上がっています

Ceph公式によると基本的にはOSD数を参考に設定すればいいものの
細かくいうとストレージ使用率によって最適なPG値が変化するらしく
使用率が少ない場合はOSDが多くてもPGは少なくする方がいいと言われているので
ベンチマークVMしかないような使用率1%程度の環境では512は多すぎたかもしれません

この結果だけ見るとPG値が不足時は読み込み速度に、多すぎると書き込み速度に
反映されやすい様子なのでどちらを優先するかにもよると思います