ClickHouse 集群快速搭建

本文介绍 ClickHouse 集群的搭建方法，为了方便和便于读者尝试，我们使用 docker 的方式来启动 ClickHouse Server 和 Zookeeper，并且后续会提供一套自动化脚本来方便读者搭建一套集群来尝试使用 ClickHouse。这里使用的 ClickHouse docker 镜像版本为latest，因为只是测试使用，新版本会有更多的新功能让我们来体验。Zookeeper docker 镜像为 3.6.1 版本，这个版本是生产上运行比较稳定的且提供 Prometheus 监控的版本。

所需文件

docker-compose.yml

docker compose 文件，用来描述各个 docker 间的拓扑关系，这里不过多介绍 docker compose 相关的内容，专注于我们的集群拓扑关系。本例子是创建一个 1 ZK 节点 4 CK 节点的 ClickHouse 集群，其中 ClickHouse 集群为 2 分片 2 副本。由于只是为了测试功能，ZK 没有采用集群式部署，所以本例子也只是用于测试，不适用于生产环境。

version: '3.5'
services:
  zookeeper:
    image: zookeeper:3.6.1
    container_name: zookeeper
    hostname: zookeeper
    networks:
      clickhouse-network:
        ipv4_address: 172.1.0.10
  clickhouse-s1-r1:
    image: clickhouse/clickhouse-server
    container_name: clickhouse-s1-r1
    hostname: clickhouse-s1-r1
    networks:
      clickhouse-network:
        ipv4_address: 172.1.0.11
    ports:
      - "127.0.0.1:8123:8123"
      - "127.0.0.1:9000:9000"
    volumes:
      - ${
    
    PWD}/clickhouse-s1-r1:/etc/clickhouse-server
    depends_on:
      - zookeeper
  clickhouse-s1-r2:
    image: clickhouse/clickhouse-server
    container_name: clickhouse-s1-r2
    hostname: clickhouse-s1-r2
    networks:
      clickhouse-network:
        ipv4_address: 172.1.0.12
    volumes:
      - ${
    
    PWD}/clickhouse-s1-r2:/etc/clickhouse-server
    depends_on:
      - zookeeper
  clickhouse-s2-r1:
    image: clickhouse/clickhouse-server
    container_name: clickhouse-s2-r1
    hostname: clickhouse-s2-r1
    networks:
      clickhouse-network:
        ipv4_address: 172.1.0.13
    volumes:
      - ${
    
    PWD}/clickhouse-s2-r1:/etc/clickhouse-server
    depends_on:
      - zookeeper
  clickhouse-s2-r2:
    image: clickhouse/clickhouse-server
    container_name: clickhouse-s2-r2
    hostname: clickhouse-s2-r2
    networks:
      clickhouse-network:
        ipv4_address: 172.1.0.14
    volumes:
      - ${
    
    PWD}/clickhouse-s2-r2:/etc/clickhouse-server
    depends_on:
      - zookeeper

networks:
  clickhouse-network:
    name: clickhouse-network
    ipam:
      config:
        - subnet: 172.1.0.0/24

config.xml

ClickHouse 的主配置文件，具体可以参考官网，这里主要的改动是精简了一些配置，包括一些系统表配置等，然后指定了集群配置 remote_servers 为 2 分片 2 副本，并且指定了节点的宏配置 macro。

<?xml version="1.0"?>
<clickhouse>
    <logger>
        <level>debug</level>
        <console>true</console>
        <log remove="remove"/>
        <errorlog remove="remove"/>
    </logger>

    <query_log>
        <database>system</database>
        <table>query_log</table>
    </query_log>

    <listen_host>0.0.0.0</listen_host>
    <http_port>8123</http_port>
    <tcp_port>9000</tcp_port>
    <interserver_http_host>clickhouse-s${SHARD}-r${REPLICA}</interserver_http_host>
    <interserver_http_port>9009</interserver_http_port>

    <max_connections>4096</max_connections>
    <keep_alive_timeout>3</keep_alive_timeout>
    <max_concurrent_queries>100</max_concurrent_queries>
    <uncompressed_cache_size>8589934592</uncompressed_cache_size>
    <mark_cache_size>5368709120</mark_cache_size>

    <path>/var/lib/clickhouse/</path>
    <tmp_path>/var/lib/clickhouse/tmp/</tmp_path>
    <user_files_path>/var/lib/clickhouse/user_files/</user_files_path>

    <users_config>users.xml</users_config>
    <default_profile>default</default_profile>
    <default_database>default</default_database>
    <mlock_executable>false</mlock_executable>

    <remote_servers>
        <default>
            <shard>
                <replica>
                    <host>clickhouse-s1-r1</host>
                    <port>9000</port>
                </replica>
                <replica>
                    <host>clickhouse-s1-r2</host>
                    <port>9000</port>
                </replica>
            </shard>
            <shard>
                <replica>
                    <host>clickhouse-s2-r1</host>
                    <port>9000</port>
                </replica>
                <replica>
                    <host>clickhouse-s2-r2</host>
                    <port>9000</port>
                </replica>
            </shard>
        </default>
    </remote_servers>

    <zookeeper>
        <node index="1">
            <host>zookeeper</host>
            <port>2181</port>
        </node>
    </zookeeper>

    <macros>
        <cluster>default</cluster>
        <shard>${SHARD}</shard>
        <replica>clickhouse-s${SHARD}-r${REPLICA}</replica>
    </macros>

    <distributed_ddl>
        <path>/clickhouse/task_queue/ddl</path>
    </distributed_ddl>

    <format_schema_path>/var/lib/clickhouse/format_schemas/</format_schema_path>
</clickhouse>

users.xml

ClickHouse 的用户配置文件，指定用户的权限、资源限制等配置。这里指定了 default 用户不需要密码登录。

<?xml version="1.0"?>
<clickhouse>
    <profiles>
        <default>
            <max_memory_usage>10000000000</max_memory_usage>
            <use_uncompressed_cache>0</use_uncompressed_cache>
            <load_balancing>in_order</load_balancing>
            <log_queries>1</log_queries>
        </default>
    </profiles>

    <users>
        <default>
            <password></password>
            <profile>default</profile>
            <networks>
                <ip>::/0</ip>
            </networks>
            <quota>default</quota>
        </default>
    </users>

    <quotas>
        <default>
            <interval>
                <duration>3600</duration>
                <queries>0</queries>
                <errors>0</errors>
                <result_rows>0</result_rows>
                <read_rows>0</read_rows>
                <execution_time>0</execution_time>
            </interval>
        </default>
    </quotas>
</clickhouse>

Makefile

为了方便配置文件的统一管理，这里使用 Makefile config 来复用一份配置文件。

.PHONY: config
config:
	rm -rf clickhouse-s1-r1 clickhouse-s1-r2 clickhouse-s2-r1 clickhouse-s2-r2
	mkdir -p clickhouse-s1-r1 clickhouse-s1-r2 clickhouse-s2-r1 clickhouse-s2-r2
	REPLICA=1 SHARD=1 envsubst < config.xml > clickhouse-s1-r1/config.xml
	REPLICA=2 SHARD=1 envsubst < config.xml > clickhouse-s1-r2/config.xml
	REPLICA=1 SHARD=2 envsubst < config.xml > clickhouse-s2-r1/config.xml
	REPLICA=2 SHARD=2 envsubst < config.xml > clickhouse-s2-r2/config.xml
	cp users.xml clickhouse-s1-r1/users.xml
	cp users.xml clickhouse-s1-r2/users.xml
	cp users.xml clickhouse-s2-r1/users.xml
	cp users.xml clickhouse-s2-r2/users.xml

.PHONY: up
up:
	docker-compose up -d

.PHONY: start
start:
	docker-compose start

.PHONY: down
down:
	docker-compose down

体验集群

启动集群

通过一下命令可以执行 Makefile 中的 config 和 up 命令，先将 config.xml 和 users.xml 复制到对应的目录，然后启动集群。

make config up

查看集群

执行 docker compose ps 就可以查看当前的 ClickHouse 集群了，可以看到以下五个容器都是 running 状态。

[root@centos clickhouse-cluster]# docker compose ps    
NAME                COMMAND                  SERVICE             STATUS              PORTS
clickhouse-s1-r1    "/entrypoint.sh"         clickhouse-s1-r1    running             127.0.0.1:8123->8123/tcp, 127.0.0.1:9000->9000/tcp, 9009/tcp
clickhouse-s1-r2    "/entrypoint.sh"         clickhouse-s1-r2    running             8123/tcp, 9000/tcp, 9009/tcp
clickhouse-s2-r1    "/entrypoint.sh"         clickhouse-s2-r1    running             8123/tcp, 9000/tcp, 9009/tcp
clickhouse-s2-r2    "/entrypoint.sh"         clickhouse-s2-r2    running             8123/tcp, 9000/tcp, 9009/tcp
zookeeper           "/docker-entrypoint.…"   zookeeper           running             2181/tcp, 2888/tcp, 3888/tcp, 8080/tcp

登录 ClickHouse

docker exec -it clickhouse-s1-r1 clickhouse-client -h localhost

通过 clusterAllReplicas 函数来查询所有节点的 system.macros 表，可以看到有 4 个节点的值。

SELECT
    _shard_num,
    *
FROM clusterAllReplicas('default', system.macros)
ORDER BY _shard_num ASC

┌─_shard_num─┬─macro───┬─substitution─────┐
│          1 │ cluster │ default          │
│          1 │ replica │ clickhouse-s1-r1 │
│          1 │ shard   │ 1                │
└────────────┴─────────┴──────────────────┘
┌─_shard_num─┬─macro───┬─substitution─────┐
│          2 │ cluster │ default          │
│          2 │ replica │ clickhouse-s1-r2 │
│          2 │ shard   │ 1                │
└────────────┴─────────┴──────────────────┘
┌─_shard_num─┬─macro───┬─substitution─────┐
│          3 │ cluster │ default          │
│          3 │ replica │ clickhouse-s2-r1 │
│          3 │ shard   │ 2                │
└────────────┴─────────┴──────────────────┘
┌─_shard_num─┬─macro───┬─substitution─────┐
│          4 │ cluster │ default          │
│          4 │ replica │ clickhouse-s2-r2 │
│          4 │ shard   │ 2                │
└────────────┴─────────┴──────────────────┘

小结

我们可以通过几个文件快速搭建一个分布式的 ClickHouse 集群，方便体验 ClickHouse 完整功能。并且后续的复杂操作都会提供脚本来辅助我们快速完成复杂的搭建和修改配置等操作。

---

欢迎添加微信：xideyantu，讨论技术问题。