1 ------
2 TOPIC:
3 ------
4 Dynamic backoff throttle is been introduced in the Jewel timeframe to produce
5 a stable and improved performance out from filestore. This should also improve
6 the average and 99th latency significantly.
7
8 -----------
9 WHAT IS IT?
10 -----------
11 The old throttle scheme in the filestore is to allow all the ios till the
12 outstanding io/bytes reached some limit. Once crossed the threshold, it will not
13 allow any more io to go through till the outstanding ios/bytes goes below the
14 threshold. That's why once it is crossed the threshold, the write behavior
15 becomes very spiky.
16 The idea of the new throttle is, based on some user configurable parameters
17 it can start throttling (inducing delays) early and with proper parameter
18 value we can prevent the outstanding io/bytes to reach to the threshold mark.
19 This dynamic backoff throttle is implemented in the following scenarios within
20 filestore.
21
22 1. Filestore op worker queue holds the transactions those are yet to be applied.
23 This queue needs to be bounded and a backoff throttle is used to gradually
24 induce delays to the queueing threads if the ratio of current outstanding
25 bytes|ops and filestore_queue_max_(bytes|ops) is more than
26 filestore_queue_low_threshhold. The throttles will block io once the
27 outstanding bytes|ops reaches max value (filestore_queue_max_(bytes|ops)).
28
29 User configurable config option for adjusting delay is the following.
30
31 filestore_queue_low_threshhold:
32 Valid values should be between 0-1 and shouldn't be > *_high_threshold.
33
34 filestore_queue_high_threshhold:
35 Valid values should be between 0-1 and shouldn't be < *_low_threshold.
36
37 filestore_expected_throughput_ops:
38 Shouldn't be less than or equal to 0.
39
40 filestore_expected_throughput_bytes:
41 Shouldn't be less than or equal to 0.
42
43 filestore_queue_high_delay_multiple:
44 Shouldn't be less than 0 and shouldn't be > *_max_delay_multiple.
45
46 filestore_queue_max_delay_multiple:
47 Shouldn't be less than 0 and shouldn't be < *_high_delay_multiple
48
49 2. journal usage throttle is implemented to gradually slow down queue_transactions
50 callers as the journal fills up. We don't want journal to become full as this
51 will again induce spiky behavior. The configs work very similarly to the
52 Filestore op worker queue throttle.
53
54 journal_throttle_low_threshhold
55 journal_throttle_high_threshhold
56 filestore_expected_throughput_ops
57 filestore_expected_throughput_bytes
58 journal_throttle_high_multiple
59 journal_throttle_max_multiple
60
61 This scheme will not be inducing any delay between [0, low_threshold].
62 In [low_threshhold, high_threshhold), delays should be injected based on a line
63 from 0 at low_threshold to high_multiple * (1/expected_throughput) at high_threshhold.
64 In [high_threshhold, 1), we want delays injected based on a line from
65 (high_multiple * (1/expected_throughput)) at high_threshhold to
66 (high_multiple * (1/expected_throughput)) + (max_multiple * (1/expected_throughput))
67 at 1.
68 Now setting *_high_multiple and *_max_multiple to 0 should give us similar effect
69 like old throttle (this is default). Setting filestore_queue_max_ops and
70 filestore_queue_max_bytes to zero should disable the entire backoff throttle.
71
72 -------------------------
73 HOW TO PICK PROPER VALUE ?
74 -------------------------
75
76 General guideline is the following.
77
78 filestore_queue_max_bytes and filestore_queue_max_ops:
79 -----------------------------------------------------
80
81 This directly depends on how filestore op_wq will be growing in the memory. So,
82 user needs to calculate how much memory he can give to each OSD. Bigger value
83 meaning current/max ratio will be smaller and throttle will be relaxed a bit.
84 This could help in case of 100% write but in case of mixed read/write this can
85 induce more latencies for reads on the objects those are not yet been applied
86 i.e yet to reach in it's final filesystem location. Ideally, once we reach the
87 max throughput of the backend, increasing further will only increase memory
88 usage and latency.
89
90 filestore_expected_throughput_bytes and filestore_expected_throughput_ops:
91 -------------------------------------------------------------------------
92
93 This directly co-relates to how much delay needs to be induced per throttle.
94 Ideally, this should be a bit above (or same) your backend device throughput.
95
96 *_low_threshhold *_high_threshhold :
97 ----------------------------------
98
99 If your backend is fast you may want to set this threshold value higher so that
100 initial throttle would be less and it can give a chance to back end transaction
101 to catch up.
102
103 *_high_multiple and *_max_multiple:
104 ----------------------------------
105
106 This is the delay tunables and one should do all effort here so that the current
107 doesn't reach to the max. For HDD and smaller block size, the old throttle may
108 make sense, in that case we should set these delay multiple to 0.
109
110 It is recommended that user should use ceph_smalliobenchfs tool to see the effect
111 of these parameters (and decide) on your HW before deploying OSDs.
112
113 Need to supply the following parameters.
114
115 1. --journal-path
116
117 2. --filestore-path
118 (Just mount the empty data partition and give the path here)
119
120 3. --op-dump-file
121
122 'ceph_smalliobenchfs --help' will show all the options that user can tweak.
123 In case user wants to supply any ceph.conf parameter related to filestore,
124 it can be done by adding '--' in front, for ex. --debug_filestore
125
126 Once test is done, analyze the throughput and latency information dumped in the
127 file provided in the option --op-dump-file.
ceph Dynamic backoff throttle
猜你喜欢
转载自my.oschina.net/itfanr/blog/1809576
今日推荐
周排行