Proxmox VE 6.1 + Cephで速度計測:SSD+HDDx3

Proxmox VE 6.1 + Cephで速度計測 各ノードSSD+HDD3台の記録

最初に

前記事の続きでDB/WALにSSDを使ったパターンを測定していきます

計測環境

以前の記事にスペックや計測方法を掲載してますのでそちらをご覧ください

Proxmox VE 6.1 + Cephで速度計測:HDDx1 | fefcc.net
https://fefcc.net/archives/1040

今回のCeph構成

各ノードに登録しているOSDはHDD ST1000DM010 1TB 7200rpmを3台
各OSDのDB/WALはSSD SATA Crucial MX500 500GBを共有で登録

Pool01という名でCeph/RBD形式のプール作成
Pool01のPG値は128のまま

6OSDから9OSDへ拡張

拡張手順は前記事と同じなので省略します
9OSDになった状態がこちら

プールのpg_numは128のままにしています

各HDD3台+SSD:計9OSDでの計測結果

Proxmox起動直後のメモリ使用状況

pve01は1.83GiB、pve01は1.82GiB、pve03は1.70GiB

CentOS7のVMから計測

[root@centos7-bench ~]# ./fio_cdm.sh /root
Testing in /root
Configuration: Size:1024m Loops:5 Write Only Zeroes:0
Running Benchmark,  please wait...


Results:

Sequential Read: 167MB/s IOPS=0
Sequential Write: 31MB/s IOPS=0

512KB Read: 89MB/s IOPS=178
512KB Write: 9MB/s IOPS=19

Sequential Q32T1 Read: 159MB/s IOPS=4
Sequential Q32T1 Write: 31MB/s IOPS=0

4KB Read: 3MB/s IOPS=1000
4KB Write: 1MB/s IOPS=454

4KB Q32T1 Read: 3MB/s IOPS=1000
4KB Q32T1 Write: 1MB/s IOPS=454

4KB Q8T8 Read: 33MB/s IOPS=8500
4KB Q8T8 Write: 1MB/s IOPS=443


[root@centos7-bench ~]#

Windows10のVMから計測

------------------------------------------------------------------------------
CrystalDiskMark 7.0.0 x64 (C) 2007-2019 hiyohiyo
                                  Crystal Dew World: https://crystalmark.info/
------------------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

[Read]
Sequential 1MiB (Q=  8, T= 1):   175.559 MB/s [    167.4 IOPS] < 47565.99 us>
Sequential 1MiB (Q=  1, T= 1):   149.546 MB/s [    142.6 IOPS] <  7009.63 us>
    Random 4KiB (Q= 32, T=16):   171.832 MB/s [  41951.2 IOPS] < 12167.36 us>
    Random 4KiB (Q=  1, T= 1):     7.871 MB/s [   1921.6 IOPS] <   519.55 us>

[Write]
Sequential 1MiB (Q=  8, T= 1):    41.949 MB/s [     40.0 IOPS] <194496.06 us>
Sequential 1MiB (Q=  1, T= 1):    13.423 MB/s [     12.8 IOPS] < 75841.40 us>
    Random 4KiB (Q= 32, T=16):     1.993 MB/s [    486.6 IOPS] <409282.24 us>
    Random 4KiB (Q=  1, T= 1):     0.395 MB/s [     96.4 IOPS] < 10267.84 us>

[Mix] Read 70%/Write 30%
Sequential 1MiB (Q=  8, T= 1):    83.253 MB/s [     79.4 IOPS] <100251.29 us>
Sequential 1MiB (Q=  1, T= 1):    34.602 MB/s [     33.0 IOPS] < 30072.45 us>
    Random 4KiB (Q= 32, T=16):     5.622 MB/s [   1372.6 IOPS] <220665.57 us>
    Random 4KiB (Q=  1, T= 1):     1.012 MB/s [    247.1 IOPS] <  4003.04 us>

Profile: Default
   Test: 1 GiB (x5) [Interval: 5 sec] <DefaultAffinity=DISABLED>
   Date: 2020/02/12 4:02:39
     OS: Windows 10 Professional [10.0 Build 18363] (x64)

rados benchで計測

ブロックサイズ1M&16スレッドで60秒間書き込み
結果は103.14MB/s

root@pve01:~# rados bench -p Pool01 -t 16 -b 1M 60 write --no-cleanup
hints = 1
Maintaining 16 concurrent writes of 1048576 bytes to objects of size 1048576 for up to 60 seconds or 0 objects
Object prefix: benchmark_data_pve01_12417
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16        95        79   78.9943        79    0.151648    0.186919
    2      16       202       186   92.9919       107    0.146749    0.165073
    3      16       305       289   96.3254       103     0.19436    0.160054
    4      16       407       391   97.7422       102    0.173736    0.159574
    5      16       512       496   99.1917       105   0.0796191    0.157412
    6      16       614       598    99.658       102    0.189238    0.158253
    7      16       717       701   100.134       103    0.170663    0.157495
    8      16       827       811   101.366       110    0.220368    0.156527
    9      16       933       917    101.88       106   0.0990909    0.155447
   10      16      1044      1028   102.791       111    0.103954    0.154413
   11      16      1143      1127   102.446        99    0.291773    0.154736
   12      16      1250      1234   102.824       107    0.123457    0.154235
   13      16      1356      1340   103.068       106   0.0947886    0.153908
   14      16      1462      1446   103.277       106    0.195154    0.154174
   15      16      1564      1548   103.191       102     0.17496     0.15424
   16      16      1665      1649   103.054       101    0.415889    0.154329
   17      16      1765      1749   102.873       100   0.0910751    0.154555
   18      16      1866      1850   102.769       101    0.166857    0.154938
   19      16      1973      1957   102.991       107    0.131006    0.154545
2020-02-12 04:05:22.078779 min lat: 0.0419693 max lat: 0.437319 avg lat: 0.154428
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      16      2079      2063   103.141       106   0.0902121    0.154428
   21      16      2183      2167   103.181       104    0.166472    0.154477
   22      16      2286      2270   103.173       103   0.0953462    0.154507
   23      16      2387      2371   103.078       101    0.103126    0.154517
   24      16      2488      2472   102.991       101    0.116258    0.154877
   25      16      2592      2576   103.031       104    0.137099    0.154807
   26      16      2692      2676   102.914       100    0.150684    0.155049
   27      16      2794      2778    102.88       102   0.0801669    0.154879
   28      16      2902      2886   103.062       108    0.107229    0.154967
   29      16      3005      2989    103.06       103    0.300001    0.154834
   30      16      3103      3087   102.891        98    0.140845    0.154911
   31      16      3208      3192   102.959       105    0.118837    0.154922
   32      16      3307      3291   102.835        99   0.0533002    0.154924
   33      16      3407      3391   102.749       100    0.128777    0.155309
   34      16      3516      3500   102.932       109    0.285367    0.155085
   35      16      3621      3605   102.991       105   0.0580663    0.154891
   36      16      3727      3711   103.074       106    0.289533    0.154987
   37      16      3822      3806   102.856        95    0.188138    0.154945
   38      16      3930      3914   102.991       108    0.220447    0.154921
   39      16      4037      4021   103.094       107    0.108037    0.154935
2020-02-12 04:05:42.080542 min lat: 0.0362313 max lat: 0.475229 avg lat: 0.154997
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   40      16      4140      4124   103.091       103    0.162846    0.154997
   41      16      4241      4225    103.04       101    0.247305    0.155041
   42      16      4342      4326   102.991       101    0.334396    0.155022
   43      16      4440      4424   102.875        98    0.223068    0.155165
   44      16      4549      4533   103.014       109     0.19159    0.155047
   45      16      4645      4629   102.858        96    0.131663    0.155224
   46      16      4747      4731   102.839       102    0.167792    0.155293
   47      16      4852      4836   102.885       105   0.0679347    0.155206
   48      16      4952      4936   102.824       100    0.186389    0.155217
   49      16      5057      5041   102.869       105    0.163051     0.15523
   50      16      5163      5147   102.931       106    0.110075    0.155175
   51      16      5266      5250   102.932       103    0.191388    0.155178
   52      16      5368      5352   102.914       102    0.142332    0.155127
   53      16      5474      5458   102.972       106     0.11335    0.155106
   54      16      5584      5568   103.102       110   0.0522058       0.155
   55      16      5686      5670   103.082       102    0.212901    0.154987
   56      16      5787      5771   103.045       101     0.14213    0.155005
   57      16      5888      5872   103.009       101    0.167801    0.154976
   58      16      5994      5978    103.06       106    0.170768    0.155027
   59      16      6101      6085   103.127       107    0.184237    0.154861
2020-02-12 04:06:02.082082 min lat: 0.0362313 max lat: 0.501009 avg lat: 0.155036
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   60      16      6198      6182   103.025        97   0.0932122    0.155036
Total time run:         60.1027
Total writes made:      6199
Write size:             1048576
Object size:            1048576
Bandwidth (MB/sec):     103.14
Stddev Bandwidth:       4.76558
Max bandwidth (MB/sec): 111
Min bandwidth (MB/sec): 79
Average IOPS:           103
Stddev IOPS:            4.76558
Max IOPS:               111
Min IOPS:               79
Average Latency(s):     0.155081
Stddev Latency(s):      0.0675867
Max latency(s):         0.501009
Min latency(s):         0.0362313
root@pve01:~#

先ほど書き込んだデータを16スレッドでシーケンシャル読み込み
結果は166.588MB/s

root@pve01:~# rados bench -p Pool01 -t 16 60 seq
hints = 1
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16       179       163    162.98       163    0.106703   0.0907979
    2      16       337       321   160.482       158  0.00305732   0.0950809
    3      16       504       488   162.649       167   0.0693441   0.0949025
    4      16       679       663   165.732       175   0.0112448   0.0940464
    5      16       849       833    166.58       170  0.00308126   0.0942842
    6      16      1017      1001   166.814       168   0.0850132   0.0946196
    7      16      1178      1162   165.981       161    0.125311   0.0949395
    8      16      1336      1320   164.982       158    0.247367   0.0957647
    9      16      1520      1504   167.093       184    0.209451   0.0947676
   10      16      1696      1680   167.982       176   0.0031072   0.0944238
   11      16      1863      1847   167.892       167  0.00771158   0.0943504
   12      16      2050      2034   169.483       187    0.154345   0.0937686
   13      16      2217      2201   169.291       167    0.276421    0.093776
   14      16      2374      2358   168.412       157  0.00453055   0.0944787
   15      16      2540      2524   168.251       166   0.0546916   0.0942288
   16      16      2713      2697   168.547       173    0.144331   0.0941479
   17      16      2876      2860    168.22       163    0.285592   0.0945412
   18      16      3049      3033   168.484       173   0.0751339   0.0943583
   19      16      3218      3202   168.511       169    0.171937   0.0944731
2020-02-12 04:06:54.846803 min lat: 0.00275273 max lat: 0.479925 avg lat: 0.0942669
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      16      3389      3373   168.635       171   0.0033018   0.0942669
   21      16      3566      3550   169.032       177    0.258975   0.0941919
   22      16      3723      3707   168.485       157   0.0678435   0.0944179
   23      16      3882      3866   168.072       159  0.00304002   0.0946061
   24      16      4051      4035    168.11       169  0.00303442   0.0947492
   25      16      4207      4191   167.625       156   0.0102796   0.0951243
   26      16      4359      4343   167.024       152    0.109681   0.0954078
   27      16      4520      4504     166.8       161    0.104861   0.0955588
   28      16      4680      4664   166.557       160    0.239691   0.0956712
   29      16      4855      4839   166.848       175    0.331598   0.0955015
   30      16      5020      5004   166.786       165     0.24207   0.0955813
   31      16      5184      5168   166.696       164   0.0793535   0.0956012
   32      16      5352      5336   166.736       168  0.00715408   0.0956675
   33      16      5520      5504   166.774       168  0.00939843   0.0956116
   34      16      5690      5674   166.868       170  0.00313725   0.0955374
   35      16      5861      5845   166.986       171    0.114454   0.0955115
   36      16      6022      6006    166.82       161  0.00324846   0.0956329
   37      16      6184      6168   166.689       162    0.250121   0.0956618
Total time run:       37.2116
Total reads made:     6199
Read size:            1048576
Object size:          1048576
Bandwidth (MB/sec):   166.588
Average IOPS:         166
Stddev IOPS:          7.68789
Max IOPS:             187
Min IOPS:             152
Average Latency(s):   0.0959081
Max latency(s):       0.479925
Min latency(s):       0.00275273
root@pve01:~#

ブロックサイズ4K&16スレッドで60秒間書き込み
結果は3.88144MB/s 993IOPS

root@pve01:~# rados bench -p Pool01 -t 16 -b 4K 60 write --no-cleanup
hints = 1
Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 60 seconds or 0 objects
Object prefix: benchmark_data_pve01_13098
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16      1054      1038   4.05428   4.05469  0.00286729   0.0154049
    2      16      2096      2080   4.06192   4.07031    0.026003   0.0151394
    3      16      3037      3021   3.93311   3.67578  0.00166826   0.0158184
    4      16      3919      3903   3.81096   3.44531  0.00131153   0.0163322
    5      16      4816      4800   3.74942   3.50391  0.00186369   0.0164931
    6      16      5856      5840   3.80155    4.0625  0.00356462   0.0164175
    7      16      6977      6961   3.88396   4.37891  0.00245743   0.0160592
    8      16      7985      7969   3.89057    3.9375  0.00169477   0.0160347
    9      16      8955      8939   3.87921   3.78906   0.0324712    0.016041
   10      16      9958      9942   3.88304   3.91797  0.00180067   0.0160666
   11      16     10994     10978   3.89789   4.04688  0.00331544   0.0160296
   12      16     12000     11984   3.90052   3.92969    0.059654   0.0159171
   13      16     12995     12979   3.89944   3.88672  0.00279998   0.0160265
   14      16     13934     13918   3.88286   3.66797   0.0515692   0.0160514
   15      16     14964     14948    3.8922   4.02344   0.0141784   0.0160463
   16      16     16061     16045   3.91673   4.28516  0.00187829   0.0159344
   17      16     17016     17000   3.90576   3.73047  0.00162955    0.015976
   18      16     17992     17976   3.90057    3.8125  0.00247569   0.0160194
   19      16     18912     18896    3.8844   3.59375  0.00160276   0.0160639
2020-02-12 04:07:53.934573 min lat: 0.00103981 max lat: 0.170747 avg lat: 0.016074
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      16     19905     19889   3.88408   3.87891  0.00308883    0.016074
   21      16     20846     20830   3.87415   3.67578   0.0016341    0.016113
   22      16     21906     21890   3.88625   4.14062  0.00291517   0.0160796
   23      16     22764     22748   3.86299   3.35156  0.00288785    0.016171
   24      16     23722     23706   3.85794   3.74219   0.0305647   0.0161802
   25      16     24712     24696    3.8583   3.86719   0.0013978   0.0161941
   26      16     25715     25699   3.86058   3.91797  0.00147027   0.0161742
   27      16     26754     26738    3.8679   4.05859   0.0194299   0.0161522
   28      16     27900     27884   3.88963   4.47656  0.00175452   0.0160669
   29      16     28867     28851   3.88574   3.77734   0.0019096   0.0160806
   30      15     29895     29880   3.89019   4.01953    0.025952   0.0160626
   31      16     30856     30840   3.88566      3.75   0.0256567   0.0160682
   32      16     31790     31774   3.87824   3.64844  0.00150022   0.0161095
   33      16     32802     32786   3.88049   3.95312  0.00191978   0.0160949
   34      16     33744     33728   3.87458   3.67969  0.00171417   0.0161248
   35      16     34745     34729   3.87559   3.91016  0.00147822     0.01612
   36      16     35665     35649   3.86775   3.59375  0.00258775   0.0161447
   37      16     36733     36717   3.87595   4.17188  0.00205322   0.0161162
   38      16     37749     37733   3.87839   3.96875   0.0015907   0.0161134
   39      16     38754     38738    3.8796   3.92578   0.0307937   0.0161055
2020-02-12 04:08:13.936306 min lat: 0.000950359 max lat: 0.170747 avg lat: 0.0161619
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   40      16     39615     39599   3.86668   3.36328  0.00341552   0.0161619
   41      16     40580     40564    3.8643   3.76953  0.00199353   0.0161717
   42      16     41496     41480   3.85748   3.57812  0.00180525   0.0161993
   43      16     42411     42395   3.85089   3.57422   0.0257882   0.0162203
   44      16     43471     43455   3.85746   4.14062   0.0309781   0.0161954
   45      16     44526     44510   3.86331   4.12109   0.0014509    0.016175
   46      16     45535     45519     3.865   3.94141  0.00190734   0.0161623
   47      16     46486     46470    3.8618   3.71484  0.00201577   0.0161813
   48      16     47421     47405   3.85743   3.65234  0.00211255   0.0162016
   49      16     48478     48462   3.86296   4.12891    0.002312   0.0161749
   50      16     49500     49484   3.86554   3.99219   0.0023025   0.0161678
   51      16     50471     50455   3.86411   3.79297   0.0014256   0.0161667
   52      16     51487     51471   3.86612   3.96875  0.00183706   0.0161642
   53      16     52481     52465   3.86642   3.88281  0.00202719   0.0161618
   54      16     53524     53508   3.87027   4.07422   0.0333304    0.016146
   55      16     54505     54489   3.86956   3.83203  0.00313067   0.0161481
   56      16     55536     55520   3.87236   4.02734   0.0329031   0.0161312
   57      16     56588     56572   3.87651   4.10938   0.0106306   0.0161222
   58      16     57585     57569   3.87682   3.89453  0.00180171   0.0161114
   59      16     58617     58601   3.87943   4.03125  0.00226624   0.0161044
2020-02-12 04:08:33.938399 min lat: 0.000950359 max lat: 0.170747 avg lat: 0.0160916
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   60      16     59636     59620    3.8811   3.98047   0.0254604   0.0160916
Total time run:         60.0172
Total writes made:      59636
Write size:             4096
Object size:            4096
Bandwidth (MB/sec):     3.88144
Stddev Bandwidth:       0.230489
Max bandwidth (MB/sec): 4.47656
Min bandwidth (MB/sec): 3.35156
Average IOPS:           993
Stddev IOPS:            59.0052
Max IOPS:               1146
Min IOPS:               858
Average Latency(s):     0.0161019
Stddev Latency(s):      0.020806
Max latency(s):         0.170747
Min latency(s):         0.000950359
root@pve01:~#

クリーンアップ

root@pve01:~# rados -p Pool01 cleanup
Removed 59636 objects
root@pve01:~#

rbd benchで計測

プール内に20GBでイメージを作成

root@pve01:~# rbd create --size=20G --pool=Pool01 Test01
root@pve01:~#

ブロックサイズ1M&16スレッドで1024MBをシーケンシャル書き込み&読み込みを計測
結果はWrite:12秒 約83MB/s・Read:5秒 約182MB/s

root@pve01:~# rbd bench Test01 --io-type write --io-size 1M --io-threads 16 --io-total 1024M --io-pattern seq --pool=Pool01
bench  type write io_size 1048576 io_threads 16 bytes 1073741824 pattern sequential
  SEC       OPS   OPS/SEC   BYTES/SEC
    1        96    107.28  112488497.79
    2       176     91.78  96234355.43
    3       256     89.00  93326751.16
    4       336     87.56  91813639.70
    5       416     86.33  90522604.25
    6       496     79.36  83218531.78
    7       576     80.97  84903117.52
    8       672     80.25  84143182.28
    9       736     79.81  83683440.00
   10       816     79.17  83020863.38
   11       896     79.81  83683440.16
   12       976     77.94  81726695.76
elapsed:    12  ops:     1024  ops/sec:    78.94  bytes/sec: 82772031.10
root@pve01:~# rbd bench Test01 --io-type read --io-size 1M --io-threads 16 --io-total 1024M --io-pattern seq --pool=Pool01
bench  type read io_size 1048576 io_threads 16 bytes 1073741824 pattern sequential
  SEC       OPS   OPS/SEC   BYTES/SEC
    1       192    203.92  213822679.81
    2       352    190.47  199724481.16
    3       544    190.47  199724481.28
    4       720    187.75  196871274.50
    5       912    186.94  196022899.62
elapsed:     5  ops:     1024  ops/sec:   173.91  bytes/sec: 182357135.34
root@pve01:~#

ブロックサイズ4K&16スレッドで512MBをランダム書き込み&読み込みを計測
結果はWrite:140秒 約3.8MB/s・Read:7秒 約74MB/s

root@pve01:~# rbd bench Test01 --io-type write --io-size 4K --io-threads 16 --io-total 512M --io-pattern rand --pool=Pool01
bench  type write io_size 4096 io_threads 16 bytes 536870912 pattern random
  SEC       OPS   OPS/SEC   BYTES/SEC
    1      5680   5584.19  22872859.32
    2      6144   2996.04  12271799.52
    3      6592   2199.69  9009916.29
    4      7104   1774.64  7268917.16
    5      7664   1527.41  6256286.11
    6      8288    522.42  2139851.63
    7      8912    558.50  2287628.19
    8      9472    576.45  2361134.40
    9     10144    607.50  2488324.03
   10     10816    630.89  2584130.44
   11     11456    634.09  2597247.85
   12     12336    686.43  2811628.68
   13     12992    704.55  2885830.95
   14     13888    749.98  3071934.26
   15     14832    807.71  3308363.63
   16     15984    904.13  3703332.89
   17     17072    944.91  3870359.32
   18     18096   1017.52  4167770.47
   20     18880    738.45  3024673.75
   21     18896    631.04  2584749.67
   22     22880   1151.61  4717002.62
   23     23312    994.88  4075014.86
   24     23344    793.21  3248994.58
   25     26784   1844.97  7556977.47
   26     27216   1682.81  6892797.54
   27     27232    843.39  3454536.96
   28     30576   1469.22  6017942.48
   29     31056   1645.02  6737985.58
   30     31072    762.97  3125137.43
   31     34240   1519.00  6221818.49
   32     35088   1465.64  6003262.66
   33     35104    892.73  3656603.19
   34     35120    862.46  3532637.53
   35     39200   1872.77  7670870.07
   36     39296    956.83  3919177.36
   37     39328    822.96  3370859.60
   38     42928   1657.59  6789495.51
   39     43984   1528.24  6259684.10
   41     44000    727.26  2978845.41
   42     44016    807.65  3308129.14
   43     48640   1742.48  7137188.74
   44     48752    913.98  3743658.83
   45     48768    922.11  3776955.50
   46     48816   1029.04  4214938.97
   47     53520   1950.70  7990055.82
   49     54144    900.50  3688465.67
   50     54160    825.88  3382818.36
   51     54176    842.35  3450263.96
   52     58320   1666.16  6824605.91
   53     59712   1024.47  4196209.46
   54     60688   1345.37  5510619.08
   55     61728   1860.33  7619927.65
   56     62832   1891.57  7747862.43
   57     63920   1119.08  4583754.98
   58     65120   1090.30  4465865.88
   59     66240   1105.07  4526374.64
   60     67264   1109.84  4545904.49
   61     68400   1116.26  4572181.39
   62     69536   1124.08  4604212.29
   63     70576   1091.18  4469459.73
   64     71616   1078.63  4418062.93
   65     72784   1089.16  4461214.89
   66     73696   1056.64  4328003.33
   67     74720   1024.48  4196286.66
   68     75760   1036.78  4246642.10
   69     76848   1042.21  4268887.32
   70     77872   1025.78  4201613.50
   71     78736   1008.79  4131985.42
   72     79856   1039.65  4258422.42
   73     80880   1018.28  4170857.63
   74     81776    990.33  4056402.13
   75     82784    983.17  4027046.12
   76     83824   1018.39  4171337.68
   77     84800    987.99  4046800.89
   78     85664    945.43  3872499.14
   79     86576    953.12  3903967.49
   80     87680    976.83  4001114.90
   81     88688    967.36  3962314.78
   82     89680    976.76  4000811.31
   83     90608   1001.60  4102551.00
   84     91536    991.98  4063145.27
   88     93008    599.18  2454231.02
   90     93040    476.97  1953683.86
   91     97856    906.81  3714305.82
   92     98624    891.44  3651338.45
   93     99456    883.52  3618878.59
   94    100144   1391.55  5699772.29
   95    100848   1611.86  6602164.66
   96    101840    795.51  3258409.71
   97    102704    810.15  3318372.41
   98    103472    796.81  3263727.19
   99    104240    824.46  3376984.31
  100    105136    854.85  3501450.02
  101    106496    838.00  3432427.86
  102    106864    842.09  3449189.58
  103    107664    833.71  3414889.76
  104    108640    860.03  3522677.01
  105    109232    819.84  3358058.07
  106    110112    806.41  3303037.49
  107    110992    818.38  3352087.11
  108    111680    814.92  3337902.83
  109    112672    828.25  3392506.12
  110    113472    848.66  3476114.82
  111    114144    804.45  3295035.88
  112    115168    843.28  3454065.28
  113    115920    844.60  3459495.95
  114    116928    851.86  3489232.24
  115    117808    862.35  3532195.17
  116    118576    894.97  3665808.75
  117    119376    821.22  3363700.33
  118    120144    839.41  3438222.40
  119    120832    767.88  3145248.44
  120    121472    733.37  3003887.92
  121    122288    745.96  3055471.84
  122    123024    747.53  3061862.59
  123    123840    744.54  3049656.18
  124    124624    769.46  3151725.02
  125    125456    795.51  3258409.78
  126    126256    789.79  3234988.37
  127    126928    781.41  3200649.15
  128    127712    762.79  3124384.15
  129    128416    760.21  3113813.34
  130    129120    729.28  2987146.53
  131    129952    738.00  3022862.09
  132    130656    743.21  3044171.17
elapsed:   140  ops:   131072  ops/sec:   932.61  bytes/sec: 3819976.14
root@pve01:~# rbd bench Test01 --io-type read --io-size 4K --io-threads 16 --io-total 512M --io-pattern rand --pool=Pool01
bench  type read io_size 4096 io_threads 16 bytes 536870912 pattern random
  SEC       OPS   OPS/SEC   BYTES/SEC
    1     17952  17967.62  73595362.04
    2     36432  18260.13  74793502.77
    3     53936  18007.63  73759242.35
    4     72416  18125.74  74243032.94
    5     90768  18170.95  74428213.04
    6    108496  18122.91  74231451.86
    7    127264  18166.01  74407991.31
elapsed:     7  ops:   131072  ops/sec: 18123.51  bytes/sec: 74233889.61
root@pve01:~#

イメージ削除

root@pve01:~# rbd rm --pool=Pool01 Test01
Removing image: 100% complete...done.
root@pve01:~#

計測後のメモリ使用状況

ピーク値で5.6GiB

プールのpg_numを128から512へ変更

9OSDなので設定可能なPG値は全体で1OSDあたり250×3OSD×3ノード=2250
プールを512にする為には512×プールサイズ3=1536が必要という訳で
今回はHDDx4のパターンと同様にpg_num:512で設定します

設定変更は前回同様にProxmoxのシェルからコマンドで対応します

root@pve01:~# ceph osd pool set Pool01 pg_num 512
set pool 3 pg_num to 512
root@pve01:~#

リマップ処理が終わって正常になった状態がこちら

各HDD3台+SSD:計9OSD(PG:512)での計測結果

Proxmox起動直後のメモリ使用状況

pve01は1.90GiB、pve01は1.94GiB、pve03は1.78GiB

CentOS7のVMから計測

[root@centos7-bench ~]# ./fio_cdm.sh /root
Testing in /root
Configuration: Size:1024m Loops:5 Write Only Zeroes:0
Running Benchmark,  please wait...


Results:

Sequential Read: 162MB/s IOPS=0
Sequential Write: 28MB/s IOPS=0

512KB Read: 73MB/s IOPS=147
512KB Write: 7MB/s IOPS=14

Sequential Q32T1 Read: 158MB/s IOPS=4
Sequential Q32T1 Write: 31MB/s IOPS=0

4KB Read: 3MB/s IOPS=1000
4KB Write: 1MB/s IOPS=416

4KB Q32T1 Read: 3MB/s IOPS=1000
4KB Q32T1 Write: 1MB/s IOPS=416

4KB Q8T8 Read: 26MB/s IOPS=6833
4KB Q8T8 Write: 1MB/s IOPS=433


[root@centos7-bench ~]#

Windows10のVMから計測

------------------------------------------------------------------------------
CrystalDiskMark 7.0.0 x64 (C) 2007-2019 hiyohiyo
                                  Crystal Dew World: https://crystalmark.info/
------------------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

[Read]
Sequential 1MiB (Q=  8, T= 1):   175.110 MB/s [    167.0 IOPS] < 47552.41 us>
Sequential 1MiB (Q=  1, T= 1):   147.657 MB/s [    140.8 IOPS] <  7091.94 us>
    Random 4KiB (Q= 32, T=16):   158.712 MB/s [  38748.0 IOPS] < 13158.36 us>
    Random 4KiB (Q=  1, T= 1):     8.151 MB/s [   1990.0 IOPS] <   501.59 us>

[Write]
Sequential 1MiB (Q=  8, T= 1):    35.655 MB/s [     34.0 IOPS] <232078.99 us>
Sequential 1MiB (Q=  1, T= 1):    11.534 MB/s [     11.0 IOPS] < 90118.97 us>
    Random 4KiB (Q= 32, T=16):     1.853 MB/s [    452.4 IOPS] <405421.46 us>
    Random 4KiB (Q=  1, T= 1):     0.353 MB/s [     86.2 IOPS] < 11566.42 us>

[Mix] Read 70%/Write 30%
Sequential 1MiB (Q=  8, T= 1):    73.191 MB/s [     69.8 IOPS] <113254.28 us>
Sequential 1MiB (Q=  1, T= 1):    31.879 MB/s [     30.4 IOPS] < 32735.02 us>
    Random 4KiB (Q= 32, T=16):     5.398 MB/s [   1317.9 IOPS] <222806.10 us>
    Random 4KiB (Q=  1, T= 1):     0.966 MB/s [    235.8 IOPS] <  4086.15 us>

Profile: Default
   Test: 1 GiB (x5) [Interval: 5 sec] <DefaultAffinity=DISABLED>
   Date: 2020/02/12 16:16:52
     OS: Windows 10 Professional [10.0 Build 18363] (x64)

rados benchで計測

ブロックサイズ1M&16スレッドで60秒間書き込み
結果は103.035MB/s

root@pve01:~# rados bench -p Pool01 -t 16 -b 1M 60 write --no-cleanup
hints = 1
Maintaining 16 concurrent writes of 1048576 bytes to objects of size 1048576 for up to 60 seconds or 0 objects
Object prefix: benchmark_data_pve01_15914
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16       100        84   83.9957        84    0.187701    0.174549
    2      16       202       186   92.9941       102    0.185979    0.162335
    3      16       303       287   95.6603       101    0.269485    0.161929
    4      16       406       390   97.4933       103   0.0565424    0.160571
    5      16       498       482   96.3933        92    0.085429     0.16317
    6      16       593       577   96.1599        95    0.113384    0.164391
    7      16       689       673    96.136        96    0.277184    0.164113
    8      16       791       775   96.8681       102    0.186503     0.16334
    9      16       895       879   97.6597       104     0.21034    0.162371
   10      16      1005       989   98.8929       110    0.115867    0.160601
   11      16      1105      1089   98.9929       100   0.0573031        0.16
   12      16      1213      1197   99.7428       108    0.152295    0.159529
   13      16      1310      1294   99.5313        97    0.120351     0.15933
   14      16      1418      1402   100.136       108   0.0973263    0.158561
   15      16      1524      1508   100.526       106     0.17676    0.158444
   16      16      1626      1610   100.618       102    0.110538    0.157991
   17      16      1729      1713   100.757       103    0.200339       0.158
   18      16      1827      1811   100.604        98    0.113243    0.157961
   19      16      1935      1919   100.993       108    0.117182    0.158046
2020-02-12 16:20:18.900688 min lat: 0.0395394 max lat: 0.583621 avg lat: 0.157755
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      16      2035      2019   100.943       100    0.165082    0.157755
   21      16      2135      2119   100.897       100     0.13679    0.157912
   22      16      2244      2228   101.265       109    0.200527    0.157516
   23      16      2345      2329   101.253       101     0.27803    0.157047
   24      16      2449      2433   101.368       104    0.109864    0.157283
   25      16      2549      2533   101.313       100     0.11856    0.157368
   26      16      2651      2635   101.339       102    0.172833    0.157193
   27      16      2756      2740   101.474       105    0.139274     0.15718
   28      16      2861      2845     101.6       105   0.0896752    0.156908
   29      16      2964      2948   101.648       103    0.172784    0.156829
   30      16      3065      3049   101.626       101    0.131514    0.157043
   31      16      3165      3149   101.573       100    0.128491     0.15697
   32      16      3269      3253   101.649       104    0.163088     0.15688
   33      16      3381      3365   101.962       112   0.0633855    0.156412
   34      16      3483      3467   101.963       102   0.0878763    0.156583
   35      16      3582      3566   101.878        99    0.141114    0.156607
   36      16      3686      3670   101.937       104    0.212936    0.156573
   37      16      3784      3768    101.83        98    0.116659    0.156608
   38      16      3888      3872   101.887       104    0.139855    0.156637
   39      16      3997      3981   102.069       109   0.0750559    0.156332
2020-02-12 16:20:38.902271 min lat: 0.0395394 max lat: 0.583621 avg lat: 0.156352
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   40      16      4099      4083   102.067       102   0.0775366    0.156352
   41      16      4202      4186    102.09       103    0.178644    0.156197
   42      16      4314      4298   102.326       112     0.12585    0.156047
   43      16      4418      4402   102.364       104   0.0803048    0.156113
   44      16      4521      4505   102.379       103    0.157185    0.156025
   45      16      4626      4610   102.437       105    0.137716     0.15592
   46      16      4732      4716   102.514       106    0.153614    0.155633
   47      16      4835      4819   102.524       103    0.224552    0.155768
   48      16      4939      4923   102.555       104    0.202835    0.155803
   49      16      5039      5023   102.502       100    0.105283      0.1557
   50      16      5140      5124   102.472       101    0.216646    0.155832
   51      16      5250      5234    102.62       110    0.111737    0.155675
   52      16      5352      5336   102.608       102    0.148714    0.155559
   53      16      5455      5439   102.615       103    0.102017    0.155588
   54      16      5559      5543    102.64       104    0.149233     0.15563
   55      16      5668      5652   102.756       109    0.153347    0.155445
   56      16      5772      5756   102.778       104    0.122354    0.155441
   57      16      5876      5860   102.799       104    0.165518     0.15534
   58      16      5978      5962   102.785       102   0.0762657    0.155449
   59      16      6085      6069   102.856       107    0.088037     0.15527
2020-02-12 16:20:58.903923 min lat: 0.0393852 max lat: 0.583621 avg lat: 0.155236
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   60      16      6194      6178   102.959       109    0.112724    0.155236
Total time run:         60.1251
Total writes made:      6195
Write size:             1048576
Object size:            1048576
Bandwidth (MB/sec):     103.035
Stddev Bandwidth:       4.71193
Max bandwidth (MB/sec): 112
Min bandwidth (MB/sec): 84
Average IOPS:           103
Stddev IOPS:            4.71193
Max IOPS:               112
Min IOPS:               84
Average Latency(s):     0.155273
Stddev Latency(s):      0.0689963
Max latency(s):         0.583621
Min latency(s):         0.0393852
root@pve01:~#

先ほど書き込んだデータを16スレッドでシーケンシャル読み込み
結果は166.674MB/s

root@pve01:~# rados bench -p Pool01 -t 16 60 seq
hints = 1
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16       199       183   182.964       183   0.0620625   0.0792351
    2      16       347       331   165.475       148    0.120293   0.0925071
    3      16       510       494   164.643       163    0.107141   0.0937007
    4      16       694       678   169.479       184    0.187459   0.0926974
    5      16       874       858   171.581       180    0.247978   0.0918876
    6      16      1037      1021   170.147       163  0.00402553   0.0922867
    7      16      1212      1196   170.839       175    0.470153   0.0923135
    8      16      1372      1356   169.482       160   0.0137008   0.0934361
    9      16      1534      1518   168.649       162   0.0329692   0.0938391
   10      16      1683      1667   166.683       149   0.0166389    0.094688
   11      16      1858      1842   167.437       175    0.109609   0.0946932
   12      16      2016      2000    166.65       158   0.0846676    0.095084
   13      16      2178      2162   166.291       162    0.232895    0.095402
   14      16      2349      2333   166.627       171    0.284819   0.0953765
   15      16      2502      2486   165.718       153  0.00922137   0.0959302
   16      16      2679      2663   166.422       177   0.0138688   0.0953636
   17      16      2849      2833   166.631       170   0.0144467    0.095199
   18      16      3003      2987   165.929       154    0.448989    0.095553
   19      16      3184      3168   166.721       181   0.0144444   0.0953377
2020-02-12 16:22:39.415812 min lat: 0.00278044 max lat: 0.614134 avg lat: 0.0953406
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      16      3347      3331   166.534       163    0.406306   0.0953406
   21      16      3506      3490   166.175       159   0.0131027   0.0957429
   22      16      3685      3669   166.757       179    0.138942   0.0955565
   23      16      3833      3817   165.941       148    0.140002   0.0960555
   24      16      4003      3987    166.11       170    0.223319   0.0959838
   25      16      4167      4151   166.025       164   0.0852787   0.0959481
   26      16      4336      4320   166.139       169  0.00305844   0.0959111
   27      16      4499      4483   166.022       163   0.0761334   0.0959924
   28      16      4662      4646   165.914       163   0.0089712   0.0960629
   29      16      4825      4809   165.813       163    0.127818   0.0961544
   30      16      5000      4984   166.119       175   0.0119342   0.0959418
   31      16      5183      5167   166.663       183    0.277521   0.0956877
   32      16      5348      5332    166.61       165   0.0129564   0.0956576
   33      16      5517      5501   166.682       169   0.0122473   0.0957369
   34      16      5677      5661   166.485       160     0.13254   0.0958199
   35      16      5847      5831   166.586       170   0.0928981   0.0957123
   36      16      6006      5990   166.375       159   0.0564166   0.0957605
   37      16      6185      6169   166.715       179   0.0655861   0.0955326
Total time run:       37.1684
Total reads made:     6195
Read size:            1048576
Object size:          1048576
Bandwidth (MB/sec):   166.674
Average IOPS:         166
Stddev IOPS:          10.0985
Max IOPS:             184
Min IOPS:             148
Average Latency(s):   0.0958543
Max latency(s):       0.614134
Min latency(s):       0.00274573
root@pve01:~#

ブロックサイズ4K&16スレッドで60秒間書き込み
結果は4.22016MB/s 1080IOPS

root@pve01:~# rados bench -p Pool01 -t 16 -b 4K 60 write --no-cleanup
hints = 1
Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 60 seconds or 0 objects
Object prefix: benchmark_data_pve01_16842
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
    0       0         0         0         0         0           -           0
    1      16      1052      1036   4.04607   4.04688  0.00200278   0.0150934
    2      16      2183      2167    4.2318   4.41797   0.0113091   0.0146819
    3      16      3414      3398   4.42395   4.80859  0.00191924   0.0141216
    4      16      4422      4406   4.30226    3.9375   0.0239537   0.0144264
    5      16      5513      5497   4.29408   4.26172  0.00141641   0.0145298
    6      16      6601      6585   4.28658      4.25   0.0021876   0.0145348
    7      16      7691      7675   4.28241   4.25781  0.00212365   0.0145637
    8      16      8729      8713   4.25388   4.05469  0.00159831   0.0146556
    9      16      9757      9741   4.22736   4.01562   0.0254878   0.0147292
   10      16     10918     10902   4.25808   4.53516   0.0319383   0.0146237
   11      16     11910     11894   4.22318     3.875  0.00150672   0.0147825
   12      16     12988     12972   4.22209   4.21094   0.0292272   0.0147692
   13      16     14089     14073   4.22811   4.30078  0.00209335   0.0147664
   14      16     15104     15088   4.20929   3.96484   0.0463125   0.0148067
   15      16     16208     16192   4.21613    4.3125  0.00188022   0.0147966
   16      16     17332     17316   4.22702   4.39062   0.0155453   0.0147843
   17      16     18392     18376   4.22189   4.14062  0.00119875   0.0147795
   18      16     19490     19474   4.22558   4.28906  0.00155527   0.0147641
   19      16     20568     20552   4.22479   4.21094   0.0292374   0.0147749
2020-02-12 16:23:54.822925 min lat: 0.00101778 max lat: 0.133509 avg lat: 0.0148034
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   20      16     21620     21604   4.21899   4.10938  0.00173442   0.0148034
   21      16     22725     22709    4.2236   4.31641  0.00218956   0.0147921
   22      16     23707     23691   4.20595   3.83594  0.00192713   0.0148536
   23      16     24746     24730   4.19953   4.05859  0.00137732    0.014869
   24      16     25813     25797    4.1982   4.16797   0.0220107    0.014878
   25      16     26976     26960   4.21198   4.54297  0.00356637   0.0148337
   26      16     28088     28072   4.21703   4.34375  0.00187118   0.0148133
   27      16     29280     29264   4.23327   4.65625  0.00143998   0.0147529
   28      16     30286     30270   4.22242   3.92969  0.00140776   0.0147881
   29      16     31402     31386   4.22713   4.35938  0.00222001   0.0147813
   30      16     32436     32420   4.22085   4.03906  0.00176613   0.0147956
   31      16     33518     33502   4.22101   4.22656   0.0019674   0.0147932
   32      16     34665     34649   4.22911   4.48047  0.00184096   0.0147753
   33      16     35748     35732   4.22914   4.23047  0.00138391   0.0147694
   34      16     36830     36814   4.22904   4.22656   0.0243034    0.014768
   35      16     37915     37899   4.22929   4.23828   0.0362649   0.0147654
   36      16     38856     38840   4.21391   3.67578  0.00159328   0.0148268
   37      16     39867     39851   4.20674   3.94922   0.0025009   0.0148545
   38      16     40942     40926   4.20653   4.19922   0.0331503    0.014849
   39      16     42121     42105   4.21675   4.60547   0.0024871   0.0148204
2020-02-12 16:24:14.825012 min lat: 0.00101778 max lat: 0.234147 avg lat: 0.0147795
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   40      16     43322     43306   4.22861   4.69141  0.00223187   0.0147795
   41      16     44421     44405   4.23017   4.29297  0.00358861   0.0147725
   42      16     45447     45431   4.22486   4.00781   0.0350306   0.0147823
   43      16     46518     46502   4.22389   4.18359    0.031724   0.0147871
   44      16     47590     47574   4.22304    4.1875   0.0301701     0.01479
   45      16     48633     48617   4.21972   4.07422   0.0311379   0.0148055
   46      15     49650     49635   4.21443   3.97656   0.0504874   0.0148255
   47      16     50766     50750   4.21742   4.35547  0.00169107   0.0148164
   48      16     51807     51791   4.21426   4.06641   0.0264488   0.0148224
   49      16     52879     52863   4.21371    4.1875  0.00180742    0.014828
   50      16     53975     53959   4.21506   4.28125  0.00232123    0.014827
   51      16     54986     54970   4.20984   3.94922   0.0311146   0.0148388
   52      16     56149     56133   4.21623   4.54297  0.00159122   0.0148188
   53      16     57246     57230   4.21752   4.28516  0.00147916   0.0148156
   54      16     58317     58301   4.21688   4.18359   0.0242346   0.0148122
   55      16     59415     59399   4.21818   4.28906  0.00127209   0.0148112
   56      16     60435     60419     4.214   3.98438  0.00389652   0.0148306
   57      16     61576     61560   4.21826   4.45703   0.0204465   0.0148152
   58      16     62729     62713   4.22317   4.50391   0.0026239   0.0147943
   59      16     63775     63759   4.22084   4.08594   0.0255581   0.0147997
2020-02-12 16:24:34.827429 min lat: 0.00101778 max lat: 0.234147 avg lat: 0.0148049
  sec Cur ops   started  finished  avg MB/s  cur MB/s last lat(s)  avg lat(s)
   60      16     64836     64820   4.21955   4.14453  0.00171019   0.0148049
Total time run:         60.0141
Total writes made:      64837
Write size:             4096
Object size:            4096
Bandwidth (MB/sec):     4.22016
Stddev Bandwidth:       0.223406
Max bandwidth (MB/sec): 4.80859
Min bandwidth (MB/sec): 3.67578
Average IOPS:           1080
Stddev IOPS:            57.1918
Max IOPS:               1231
Min IOPS:               941
Average Latency(s):     0.0148095
Stddev Latency(s):      0.0183897
Max latency(s):         0.234147
Min latency(s):         0.00101778
root@pve01:~#

クリーンアップ

root@pve01:~# rados -p Pool01 cleanup
Removed 64837 objects
root@pve01:~#

rbd benchで計測

プール内に20GBでイメージを作成

root@pve01:~# rbd create --size=20G --pool=Pool01 Test01
root@pve01:~#

ブロックサイズ1M&16スレッドで1024MBをシーケンシャル書き込み&読み込みを計測
結果はWrite:13秒 約82MB/s・Read:6秒 約164MB/s

root@pve01:~# rbd bench Test01 --io-type write --io-size 1M --io-threads 16 --io-total 1024M --io-pattern seq --pool=Pool01
bench  type write io_size 1048576 io_threads 16 bytes 1073741824 pattern sequential
  SEC       OPS   OPS/SEC   BYTES/SEC
    1        96     97.56  102298249.69
    2       176     87.91  92180840.50
    3       256     85.21  89350727.08
    4       320     82.84  86862715.29
    5       400     82.67  86685162.71
    6       480     78.05  81838600.35
    7       544     76.16  79856995.36
    8       608     73.15  76702429.20
    9       704     74.65  78274867.14
   10       800     79.18  83021146.21
   11       880     80.32  84221452.13
   12       960     81.57  85529361.93
elapsed:    13  ops:     1024  ops/sec:    78.07  bytes/sec: 81863559.31
root@pve01:~# rbd bench Test01 --io-type read --io-size 1M --io-threads 16 --io-total 1024M --io-pattern seq --pool=Pool01
bench  type read io_size 1048576 io_threads 16 bytes 1073741824 pattern sequential
  SEC       OPS   OPS/SEC   BYTES/SEC
    1       160    191.30  200593542.59
    2       336    188.03  197164593.37
    3       496    176.79  185380270.87
    4       624    165.12  173136372.82
    5       800    165.99  174048543.93
    6       944    158.32  166007434.27
elapsed:     6  ops:     1024  ops/sec:   157.05  bytes/sec: 164681370.03
root@pve01:~#

ブロックサイズ4K&16スレッドで512MBをランダム書き込み&読み込みを計測
結果はWrite:124秒 約4.32MB/s・Read:6秒 約78MB/s

root@pve01:~# rbd bench Test01 --io-type write --io-size 4K --io-threads 16 --io-total 512M --io-pattern rand --pool=Pool01
bench  type write io_size 4096 io_threads 16 bytes 536870912 pattern random
  SEC       OPS   OPS/SEC   BYTES/SEC
    1      5616   5631.90  23068258.32
    2      6080   3054.05  12509402.95
    3      6576   2188.54  8964258.92
    4      7088   1770.66  7252607.75
    5      7632   1524.69  6245144.82
    6      8272    529.50  2168815.99
    7      8976    574.59  2353532.42
    8      9616    608.96  2494314.27
    9     10352    654.88  2682404.58
   10     11072    689.09  2822513.53
   11     11696    685.89  2809385.57
   12     12672    716.27  2933826.52
   13     13552    788.45  3229480.62
   14     14368    794.92  3255986.03
   15     15104    806.39  3302955.25
   16     16096    881.39  3610191.77
   17     17024    905.14  3707461.08
   18     18112    906.91  3714682.91
   20     18928    746.55  3057862.76
   22     18944    546.38  2237956.54
   23     23104   1001.70  4102952.32
   24     23472    842.64  3451455.24
   25     23504    762.86  3124679.81
   26     27584   1471.08  6025550.97
   27     27696   1725.52  7067734.81
   28     27744    887.51  3635252.52
   29     31824   1922.62  7875041.53
   30     32272   1595.31  6534404.30
   31     32288    923.79  3783825.53
   32     36176   1730.58  7088461.07
   33     36384   1679.60  6879628.23
   34     36784    983.33  4027715.50
   35     40512   1845.85  7560581.77
   36     40560   1426.18  5841639.10
   37     43584   1470.98  6025144.16
   38     44272   1481.57  6068495.68
   39     44288   1403.12  5747164.38
   40     47616   1402.82  5745944.79
   41     48048   1738.13  7119382.64
   42     48064    830.23  3400621.28
   43     51504   1660.21  6800216.54
   44     51728   1504.83  6163773.48
   45     51760    778.35  3188114.40
   46     55024   1463.06  5992698.43
   47     55552   1420.31  5817583.61
   48     55568    731.98  2998171.30
   49     58896   1519.91  6225531.01
   50     59856   1603.77  6569061.91
   51     61024   1184.81  4852994.02
   52     61664   1173.56  4806894.32
   54     61680   1113.68  4561644.61
   55     66048   1198.37  4908523.75
   56     67136   1296.27  5309534.98
   57     68240   1216.43  4982503.23
   58     69280   1500.37  6145504.34
   59     70576   1812.52  7424076.34
   60     71904   1176.83  4820287.10
   63     73040    805.66  3299993.62
   64     73056    655.05  2683077.34
   65     77792   1214.59  4974961.62
   66     79360   1194.11  4891058.09
   67     80224   1175.12  4813292.82
   68     81424   1802.20  7381825.59
   69     82320   1996.52  8177730.29
   70     83728   1159.35  4748715.48
   71     84624   1132.51  4638760.82
   72     85712   1110.91  4550293.11
   73     86928   1103.43  4519643.72
   74     87872   1098.08  4497742.76
   75     88976   1075.39  4404800.33
   76     90160   1087.18  4453073.97
   77     91200   1099.34  4502894.27
   78     93120   1112.85  4558228.07
   79     93392   1116.48  4573122.56
   80     94592   1099.43  4503274.66
   81     95440   1075.78  4406376.42
   82     96576   1076.90  4410998.51
   83     97872   1059.75  4340732.83
   84     98912   1099.58  4503888.08
   85     99984   1070.67  4385472.50
   86    100864   1073.62  4397527.56
   87    101792   1025.14  4198969.42
   88    102816   1008.96  4132706.98
   89    103808    979.97  4013943.10
   90    104848   1002.46  4106057.32
   91    105856   1008.87  4132350.21
   92    106832   1023.54  4192419.45
   93    107808    997.58  4086104.90
   94    108784    993.59  4069755.21
   95    109712    957.46  3921769.74
   96    110608    948.11  3883449.02
   97    111632    963.06  3944713.28
   98    112656    971.14  3977775.55
   99    113472    942.11  3858863.99
  100    114592    981.48  4020137.83
  101    115408    958.45  3925808.94
  102    116384    948.11  3883449.06
  103    117424    937.09  3838321.84
  104    118400    980.87  4017661.22
  105    119392    970.86  3976628.50
  106    120400    971.95  3981088.27
  107    121280    979.97  4013943.23
  108    122144    963.25  3945464.73
  109    123232    955.68  3914462.24
  110    124240    968.81  3968236.60
  111    125216    993.38  4068891.57
  112    126400    932.93  3821273.38
  113    126864    945.50  3872751.88
  114    128016    958.32  3925263.77
  115    129232    929.94  3809029.38
  116    130192    948.15  3883638.51
  117    130832    984.00  4030450.80
elapsed:   124  ops:   131072  ops/sec:  1055.35  bytes/sec: 4322694.37
root@pve01:~# rbd bench Test01 --io-type read --io-size 4K --io-threads 16 --io-total 512M --io-pattern rand --pool=Pool01
bench  type read io_size 4096 io_threads 16 bytes 536870912 pattern random
  SEC       OPS   OPS/SEC   BYTES/SEC
    1     18016  18104.10  74154389.71
    2     36848  18468.61  75647438.17
    3     56848  18979.64  77740602.49
    4     75936  19006.67  77851330.68
    5     94608  18939.62  77576678.10
    6    113408  19078.06  78143751.40
elapsed:     6  ops:   131072  ops/sec: 19006.63  bytes/sec: 77851140.59
root@pve01:~#

イメージ削除

root@pve01:~# rbd rm --pool=Pool01 Test01
Removing image: 100% complete...done.
root@pve01:~#

計測後のメモリ使用状況

ピーク値で5.6GiB

まとめ

多数のVMを動かすことを考えるとまだまだ絶対的な速度な不足してますが
各ノードにSSD1台でこの結果なので既にHDDだけで構成したCephを使っているなら
SSD1台のコストを考えるとコストパフォーマンスはいいと思います

3OSDでシーケンシャル書き込みもGbE上限に近い100MB/s前後を記録してますし
特にVMサーバとしてはランダムアクセスが重要なのですが
WAL高速化のおかげでランダムアクセスの性能が上がり
前記事の結果でもOSインストール時間が半減してたりと
VMの操作感は数値以上で、体感で差を感じられるほどの変化がありました
今回はSATA SSDでしたがNVMe SSDを使えばもっと期待できそうです

DB/WALを別デバイスに設定するデメリットとしては
SSD書き込み回数問題とDB/WALデバイス交換問題があります

SSD書き込み回数問題自体は有名なので説明しませんが
デスクトップ用途とは違ってDB/WALに使う場合には
クラスタの3ノード全てのVM書き込みを処理するのに加えて
fsyncのおかげで通常使用よりもカウントは増えますので
今回みたいなテスト用環境なら問題ないですが後述の通り交換も面倒なので
本稼働する前にVMの規模に応じてSSDのTBWを確認する必要があります
(それもあってCephはサーバ向けSSDが推奨されています)

もう1つがDB/WALデバイス交換問題です
今回の計測環境で言うと各OSDが「HDD&SSDの一部」で構成されているわけですが
システムから見れば2デバイスであっても1つのOSDとしか見ていないので
このどちらかのデバイスが故障した時点でOSD自体の故障と判定されます

つまりDB/WALで共用しているSSDが寿命・故障した際は
該当するSSDを使って登録しているOSD全て同時に全滅します

今回の計測環境は1ノード内を1台のSSDでカバーしていますから
このSSDが壊れるだけで1ノード全滅となります
残りの2ノードにデータがあるので即データ消失ではありませんが
同時期に導入したSSDを他ノードで使っている場合は注意が必要です

更にはS.M.A.R.T.を見て故障する前に事前に交換したくなっても
DB/WALデバイスのみ交換は専用コマンドが用意されていない様子
(現時点ではマニュアルから見つけられませんでした)
ネットで見つけた解決策はSSDを丸ごとクローンするかOSD再登録の2択です
これを紹介されている時点でCephとしては用意していない可能性が高いわけですが
どちらの方法もサーバだけで完結する為には交換用SSDで使う接続ポート(SATAなど)を
用意しておく必要があり手順もなかなか面倒でこの辺りは残念です

というわけでRAIDみたいに故障したデバイスを抜いて交換するだけで済ませたいなら
現時点ではDB/WALを別デバイスで設定するのはテスト用環境以外はオススメしません

メモリ消費については
キャッシュを使わずすぐに同期書き込みする仕様のせいか
シーケンシャルでも案外メモリは思ったより使われずランダム書き込みが一番消費してました

最後にPG:128のプールにシーケンシャル書き込みベンチマーク最中のレイテンシがこちら

HDDx4のレイテンシが50~60だったので全然違うのが確認できます