clickhouse (seven, cluster monitoring)

Introduction

The importance of cluster monitoring for a production application is self-evident. Today we will look at the practice of clickhouse cluster monitoring. For those who are not familiar with clickhouse related components, it is necessary to introduce some tools in this section.

chproxy

First of all, when using the ck cluster, we usually use chproxy, an open source proxy tool, which is mainly used for load balancing, concurrency control, user access control and other operations.

gravitation

grafana supports multiple data sources and rich chart display. It is a beautiful and powerful visual monitoring indicator display tool.

prometheus

Monitoring requires us to collect system data, so we also need a place to store the continuously collected data, so here we use prometheus, an excellent time series database.

Installation configuration

chproxy

• Download
  We select the latest version in releases .

# 下载
wget https://github.com/Vertamedia/chproxy/releases/download/v1.14.0/chproxy-linux-amd64-v1.14.0.tar.gz

# 解压
tar zxvf chproxy-linux-amd64-v1.14.0.tar.gz

• Configuration
  Then create a new configuration file config.yml, there is a sample configuration reference in github .
  I configure two nodes here, the user name is default, and the password is blank. You can customize the listening port, the 19000 I configured here, will be used in subsequent prometheus.

# config.yml简单配置，
server:
  http:
      listen_addr: ":19000"
      allowed_networks: ["192.168.9.225","192.168.9.226"]

users:
  - name: "default"
    to_cluster: "dc_sit"
    to_user: "default"
    password: "" 


clusters:
  - name: "dc_sit"

    nodes: [
      "dc-sit-225:8123",
      "dc-sit-226:8123"
    ]

    heartbeat_interval: 30s

    users:
      - name: "default"
        password: ""

• Start
  command: ./chproxy -config=config.yml
  If the log prints as follows successful, it is a normal start. Otherwise, handle the exception according to the prompt.

INFO: 2020/05/12 04:58:13 main.go:53: Loading config "config.yml": successful
INFO: 2020/05/12 04:58:13 main.go:152: Serving http on ":19000"

• test

# 外网限制
sun_mac:~ sun$  echo 'SELECT 1' | curl 'http://dc-sit-225:19000/?user=default' --data-binary @-
http connections are not allowed from 192.168.64.49:62164

# 内网通过
[root@dc-sit-225 ~]# echo 'SELECT 1' | curl 'http://dc-sit-225:19000/?user=default' --data-binary @-
1

• chproxy removes network restrictions
  By default, chproxy has a security check, because the account password is configured in the proxy, it allowed_networks: ["0.0.0.0/0"]is not recommended to restrict this kind of open network. But if you want to ensure network security, you can also hack_me_please: truedisable all security-related checks by configuring the outermost layer . By commenting out the previous allowed_networksconfiguration, you can open it to the outside world. This section has instructions on github .

prometheus

• download

wget https://github.com/prometheus/prometheus/releases/download/v2.18.1/prometheus-2.18.1.linux-amd64.tar.gz

• Configuration
  Modify the configuration file after decompression prometheus.yml, mainly modify scrape_configspart of the configuration, here we specify the namingclickhouse-chproxy

scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'clickhouse-chproxy'
    scrape_interval: 10s
    static_configs:
    - targets: ['dc-sit-225:19000']
      labels:
          instance: 'dc_sit'

• Start
  command:./prometheus --config.file=prometheus.yml
• After verification
  , open http://localhost:9090/targets to check whether the chproxy status is normal.
  

gravitation

• To install
  grafana, we can use a simple docker installation.

# 获取镜像
docker pull grafana/grafana
# 后台启动容器，将23000映射到容器的3000
docker run -d -p 23000:3000 --name=my-grafana -v /data/grafana:/var/lib/grafana grafana/grafana

• Login
  Open http://localhost:23000/ , account/password: admin/admin.
  

Monitoring configuration

Configure DataSources

Settings -> Data Sources -> Add data source -> select prometheus type.

• settings
  are modified according to their own configuration, I named them here Prometheus-225, and then Save & Test
  

Import template

First download the exporter template of chproxy and save it as chproxy.json. Before selecting the + sign import in grafana , the

name can be customized, and the prometheus data source selects the Prometheus-225 we configured above, and then import it.

View effect

The dashboard can see multiple indicators such as the number of requests in each period, request duration, number of concurrent users, and the status of each node. It is very convenient for us to get the running status of the cluster.

end

Because of the space factor, this section mainly describes the cluster monitoring part. There are also some important modules, such as the alarm component alertmanager provided by prometheus, and the alarm hook service.
In addition, the combination of grafana and prometheus is also commonly used to monitor the status of each node in the cluster, node_exporter , and clickhouse single component monitoring clickhouse_exporter . If the above cluster can be configured, I believe there will be no big problem for a single node.