AutoTikv Profile

AutoTikv is a tool for automatic tuning of TiKV database. Its design was inspired by the SIGMOD of a paper 2017: Automatic Database Management System Through the Tuning Large-Scale Machine Learning , using machine learning model for database parameters for automatic tuning.

Design goals

Substantially the entire tuning process below:

AutoTiKV support after the restart tikv modify the parameters (you can also choose not to restart). Need to adjust parameters and metric may be declared to be viewed in controller.py years.

The following is a statement knob model:

"Write-Buffer-size" : # The name of Knob 
    {
         "changebyyml": True, # indicates True modify tikv-ansible / conf / adjusting tikv.yml
         "set_func": None, if # changebyyml == False, then the function name specified modification parameter (in function also define controller.py, generally using tikv- CTL adjusting command line ) 
        "MINVAL": 64, # IF type =! enum , Indicate min Possible value
         "MAXVAL": 1024, # IF of the type =! enum , Indicate Possible max value
         "EnumVal": [], # IF of the type == enum, list all valid values
        "type": "int",                          # int / enum / real
        "default": 64                           # default value
    },

The following statement is a metric model:

"write_latency" : 
    {
      "read_func" : read_write_latency, # statement to see the index function (function is also defined in controller.py in)
      "lessisbetter": 1, # model types within less value of the this Metric IS of Better (1 : yes)
      "Calc ":" INS " , # INS represents the value of this parameter is the result after the benchmark to view. inc indicates that the parameter is the incremental, and need after the benchmark value before subtraction as a result. 
    },

10 (specifically, size can be adjusted) is randomly generated knob to benchmark, it is recommended after the ML model parameters to the beginning of the benchmark.

ML model

AutoTikv and OtterTune also used as a Gaussian process regression to recommend new knob. That estimate f: X-> Y (for example, parameter X, the estimated value of the latency Y), the problem becomes finding the right of X, so that f (X) value as small as possible. F above and we do find the right gradient descent to minimize the X of Y (if Y is the bigger the better, such as throughput, direct access to all the negative Y). As shown below:

One of the benefits is Gaussian regression: 1 and neural network like compared to a Gaussian process model parameter model belongs to no, to solve the problem of the relative complexity of the algorithm and compare with other algorithms to reduce the amount of calculation; but also in the training sample. NN perform better than a few cases. 2. It is not only possible to estimate a value corresponding to Y at a given X, X can mean estimate m (X) and standard deviation s (X).

There is also a Gaussian regression nature: the search for new recommended values of time, will consider exploration (exploration) and use (exploitation):

That exploration explore new data points are few points in an unknown area.
Namely the use of these data using the training machine learning models in a sufficient number of data points known area is estimated, and then find a good point

In the proposed process, it is necessary to explore new areas, but also the use of data on the known area were recommended, namely the need to balance the exploration and use might otherwise fall into local optimum and can not find the global optimum point. For example, the area has been known to use data recommendation, although able to find the best spot in this zone, but unknown area may have better points undetected. And has been exploring in turn makes the search process is very inefficient. The core idea of this balance between the two is this: When enough data for a long time, we recommend the use of these data; and when the lack of data, we explore the fewest points in the region, to explore the most unknown areas give us the maximum amount of information.

Using the above characteristics of the Gaussian process regression: It estimated average m (X) and the standard deviation S (X), if not more surrounding data X, it is estimated standard deviation of S (X) will be too large (this X other data points and a large difference), if the data are intuitively understood that much, no uncertainty will be larger, reflected in the standard deviation is too large. Conversely, when the data is sufficient, standard deviation will be small, because uncertainty is reduced.

In recommending use of the confidence interval bound Upper Confidence Bound to balance the exploration and use. Let's assume that we need to find the X Y value as large as possible. Then U (X) = m (X) + k * s (X), where k> 0 is the adjustable coefficient. We just need to find the X U (X) can be as large as possible.

When U (X) large, it may be m (X) large, it may s (X) large.
If S (X) large, then no more data around the X, need to explore new point unknown region.
If m (X) large, i.e., the estimated mean value Y is large, the need to find a good effect using known data points. Formula coefficient k affects the proportion of the exploration and use of, the greater k, the more encouraged to explore new areas.

In the code implementation, we start random generated 40 candidate knobs, and then calculate their U (X) and predicted values with Gaussian regression model to find U (X) as the maximum that a result of this recommendation.

Ref：https://mp.weixin.qq.com/s/y8VIieK0LO37SjRRyPhtrw

Experimental results

AutoTiKV currently supports the following tuning parameters:

to sum up

Guess you like