ClickHouse 集群部署及资源权限管控文档

文章目录

一、ClickHouse集群部署

1.1 版本说明

软硬件信息	参数
配置	2C4G
操作系统版本	CentOS Linux release 7.7.1908 (Core)
内核版本	3.10.0-1062.el7.x86_64
Zookeeper版本	apache-zookeeper-3.6.1
ClickHouse版本	clickhouse-client-20.10.3.30-2 clickhouse-common-static-20.10.3.30-2 clickhouse-common-static-dbg-20.10.3.30-2 clickhouse-server-20.10.3.30-2 clickhouse-test-20.10.3.30-2

1.2 服务器规划

服务器	IP	端口	角色
db01	192.168.137.129	2181/2888/3888	Zookeeper
db02	192.168.137.130	2181/2888/3888	Zookeeper
db03	192.168.137.131	2181/2888/3888	Zookeeper
db01	192.168.137.129	9000(server)/8123(http)/9009(inter)	ClickHouse
db02	192.168.137.130	9000(server)/8123(http)/9009(inter)	ClickHouse
db03	192.168.137.131	9000(server)/8123(http)/9009(inter)	ClickHouse

生产环境建议数据库服务器与zookeeper服务器隔离，防止因单个组件导致服务器异常进而导致其余服务受到影响

1.3 配置目录规划

软件	目录
Zookeeper软件安装目录	/usr/local/zookeeper/
Zookeeper数据文件目录	/data/zookeeper/data/
Zookeeper日志文件目录	/data/zookeeper/log/
ClickHouse配置文件目录	/etc/clickhouse-server/
ClickHouse数据目录	/data/clickhouse/data/,/data2/clickhouse/data/
ClickHouse日志目录	/data/clickhouse/logs

二、基础环境配置

2.1 配置映射

cat>>/etc/hosts <<EOF
192.168.137.129 db01
192.168.137.130 db02
192.168.137.131 db03
EOF

2.2 关闭防火墙&selinux

systemctl stop firewalld
systemctl disable firewalld
setenforce 0 && sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config

2.3 配置互信

# 每个节点都执行
ssh-keygen -t rsa # 一路回车
# 将公钥添加到认证文件中
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
# 并设置authorized_keys的访问权限
chmod 600 ~/.ssh/authorized_keys

# 只要在一个节点执行即可
ssh db02 cat ~/.ssh/id_rsa.pub >>~/.ssh/authorized_keys
ssh db03 cat ~/.ssh/id_rsa.pub >>~/.ssh/authorized_keys

# 分发整合后的文件到其它节点
scp ~/.ssh/authorized_keys db02:~/.ssh/
scp ~/.ssh/authorized_keys db03:~/.ssh/

#各节点执行
ssh db01 date
ssh db02 date
ssh db03 date

2.4 创建用户

groupadd hadoop
useradd hadoop -g hadoop
id hadoop

2.5 安装jdk

rpm -ivh jdk-8u251-linux-x64.rpm
cat>>/etc/profile <<EOF
export JAVA_HOME=/usr/java/jdk1.8.0_251-amd64
export PATH=$PATH:$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib
EOF
source /etc/profile

三、zookeeper部署

3.1 解压安装包并配置环境变量

tar xf apache-zookeeper-3.6.1-bin.tar.gz
mv apache-zookeeper-3.6.1-bin /usr/local/zookeeper
chown -R hadoop:hadoop /usr/local/zookeeper/
cat>>/etc/profile <<EOF
PATH=/usr/local/zookeeper/bin:$PATH
export PATH
EOF
source /etc/profile

3.2 创建数据/日志目录

mkdir -pv /data/zookeeper/{data,log} 
chown -R hadoop:hadoop /data/zookeeper/

3.3 修改配置文件

su - hadoop
cd /usr/local/zookeeper/conf
cp zoo_sample.cfg zoo.cfg
cat>>zoo.cfg <<EOF
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/zookeeper/data/
clientPort=2181

dataLogDir=/data/zookeeper/log/
server.1=db01:2888:3888
server.2=db02:2888:3888
server.3=db03:2888:3888
EOF

3.4 创建myid

#各节点配置，根据server.1就是1
echo 1 > /data/zookeeper/data/myid

3.5 启动zk集群

#依次启动各节点
zkServer.sh start
#查看集群状态
[hadoop@db01 data]$ zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: follower
[hadoop@db02 data]$ zkServer.sh status
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: leader
[hadoop@db03 data]$ zkServer.sh status
/usr/bin/java
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: follower

四、ClickHouse部署

4.1 安装

yum localinstall -y clickhouse-*.rpm

4.2 修改配置文件

4.2.1 config.xml

cp /etc/clickhouse-server/config.xml /etc/clickhouse-server/config.xml.bak

全局信息配置文件

<?xml version="1.0"?>
    <yandex>
    <!--日志-->
    <logger>
    <level>error</level>
    <log>/data/clickhouse/logs/clickhouse.log</log>
    <errorlog>/data/clickhouse/logs/error.log</errorlog>
    <size>500M</size>
    <count>5</count>
    </logger>
    <!--本地节点信息-->
    <http_port>8123</http_port>
    <tcp_port>9000</tcp_port>
    <interserver_http_port>9009</interserver_http_port>
    <interserver_http_host>db01</interserver_http_host>  <!--本机域名或IP-->
    <!--本地配置-->
    <listen_host>0.0.0.0</listen_host>
    <timezone>Asia/Shanghai</timezone>
    <max_connections>4096</max_connections>
    <keep_alive_timeout>300</keep_alive_timeout>
    <max_concurrent_queries>1000</max_concurrent_queries>
     <uncompressed_cache_size>8589934592</uncompressed_cache_size>
    <mark_cache_size>5368709120</mark_cache_size>
    <users_config>users.xml</users_config>
    <default_profile>default</default_profile>
    <default_database>default</default_database>
	<path>/data/clickhouse/data/</path>
	<user_files_path>/data/clickhouse/user_files/</user_files_path>
	<access_control_path>/data/clickhouse/access/</access_control_path>
	 <format_schema_path>/data/clickhouse/format_schemas/</format_schema_path>
	
    <tmp_path>/data/clickhouse/tmp/</tmp_path>
	<storage_configuration>
	 <default> <!-- disk name -->
            <path>/data/clickhouse/data/</path>
            <keep_free_space_bytes>1073741824</keep_free_space_bytes>
        </default>
    <disks>
        <disk_name_1> <!-- disk name -->
            <path>/data/clickhouse/data/</path>
            <keep_free_space_bytes>1073741824</keep_free_space_bytes>
        </disk_name_1>
        <disk_name_2>
            <path>/data2/clickhouse/data/</path>
            <keep_free_space_bytes>1073741824</keep_free_space_bytes>
        </disk_name_2>
    </disks>
	  <policies>
      <data_data2> <!-- name for new storage policy -->
        <volumes>  
          <data_data2_volume> <!-- name of volume -->
            <disk>disk_name_1</disk>
            <disk>disk_name_2</disk>
          </data_data2_volume>
        </volumes>
      </data_data2>
    </policies>

</storage_configuration>
    <!--集群相关配置-->
    <remote_servers incl="clickhouse_remote_servers"  />
    <zookeeper incl="zookeeper-servers"  optional="true"   />
    <macros incl="macros"  optional="true" />
    <builtin_dictionaries_reload_interval>3600</builtin_dictionaries_reload_interval>
    <max_session_timeout>3600</max_session_timeout>
    <default_session_timeout>300</default_session_timeout>
    <max_table_size_to_drop>0</max_table_size_to_drop>
    <merge_tree>
    <parts_to_delay_insert>300</parts_to_delay_insert>
    <parts_to_throw_insert>600</parts_to_throw_insert>
    <max_delay_to_insert>2</max_delay_to_insert>
    </merge_tree>
    <max_table_size_to_drop>0</max_table_size_to_drop>
     <max_partition_size_to_drop>0</max_partition_size_to_drop>
    <distributed_ddl>
    <!-- Path in ZooKeeper to queue with DDL queries -->
    <path>/clickhouse/task_queue/ddl</path>
    </distributed_ddl>
     <include_from>/etc/clickhouse-server/metrika.xml</include_from>
    </yandex>

4.2.2 users.xml

cp /etc/clickhouse-server/users.xml /etc/clickhouse-server/users.xml.bak

用户信息配置文件

<?xml version="1.0"?>
    <yandex>
    <profiles>
    <default>
    <max_memory_usage>1000000000</max_memory_usage>
    <use_uncompressed_cache>0</use_uncompressed_cache>
    <load_balancing>random</load_balancing>
    </default>
    <readonly>
    <max_memory_usage>1000000000</max_memory_usage>
    <use_uncompressed_cache>0</use_uncompressed_cache>
    <load_balancing>random</load_balancing>
    <readonly>1</readonly>
    </readonly>
    </profiles>
    <quotas>
    <!-- Name of quota. -->
    <default>
    <interval>
    <queries>0</queries>
    <errors>0</errors>
    <result_rows>0</result_rows>
    <read_rows>0</read_rows>
    <execution_time>0</execution_time>
    </interval>
    </default>
    </quotas>
    <users>
    <default>
    <!-- PASSWORD=$(base64 < /dev/urandom | head -c8); echo  "$PASSWORD"; echo -n  "$PASSWORD" | sha256sum | tr -d '-'   -->
    <!-- password UHXQQmhb -->
     <password_sha256_hex>5e884898da28047151d0e56f8dc6292773603d0d6aabbdd62a11ef721d1542d8</password_sha256_hex>
    <access_management>1</access_management>
    <networks>
    <ip>::/0</ip>
    </networks>
    <profile>default</profile>
    <quota>default</quota>
    </default>
    <ck>
    <password_sha256_hex>5e884898da28047151d0e56f8dc6292773603d0d6aabbdd62a11ef721d1542d8</password_sha256_hex>
    <networks>
    <ip>::/0</ip>
    </networks>
    <profile>readonly</profile>
    <quota>default</quota>
    </ck>
    </users>
    </yandex>

4.2.3 metrika.xml

集群信息配置文件

<yandex>
    <!--ck集群节点-->
    <clickhouse_remote_servers>
    <clickhouse_cluster_name>
    <!--分片1-->
    <shard>
    <internal_replication>true</internal_replication>
    <replica>
    <host>192.168.137.129</host>
    <port>9000</port>
    <user>default</user>
    <password>password</password>
    </replica>
    <!--复制集1-->
    <replica>
    <host>192.168.137.130</host>
    <port>9000</port>
    <user>default</user>
    <password>password</password>
    </replica>
    </shard>
    <!--分片2-->
    <shard>
    <internal_replication>true</internal_replication>
    <replica>
    <host>192.168.137.130</host>
    <port>9000</port>
    <user>default</user>
    <password>password</password>
    </replica>
    <!--复制集2-->
    <replica>
    <host>192.168.137.131</host>
    <port>9000</port>
    <user>default</user>
    <password>password</password>
    </replica>
    </shard>
    </clickhouse_cluster_name>
    </clickhouse_remote_servers>
    <!--zookeeper相关配置-->
    <zookeeper-servers>
    <node index="1">
    <host>192.168.137.129</host>
    <port>2181</port>
    </node>
    <node index="2">
    <host>192.168.137.130</host>
    <port>2181</port>
    </node>
    <node index="3">
    <host>192.168.137.131</host>
    <port>2181</port>
    </node>
    </zookeeper-servers>
    <macros>
    <layer>01</layer>
    <shard>01</shard> <!--分片号-->
    <replica>192.168.137.129</replica> <!--当前节点IP-->
    </macros>
    <networks>
    <ip>::/0</ip>
    </networks>
    <!--压缩相关配置-->
    <clickhouse_compression>
    <case>
    <min_part_size>1000000000</min_part_size>
    <min_part_size_ratio>0.01</min_part_size_ratio>
    <method>lz4</method> <!--压缩算法lz4压缩比zstd快, 更占磁盘-->
    </case>
    </clickhouse_compression>
    </yandex>

4.3 创建目录

mkdir -pv /{data,data2}/clickhouse/{data,tmp,logs,lock,access}
chown -R clickhouse.clickhouse /data/clickhouse/
chown -R clickhouse.clickhouse /data2/clickhouse/
chown -R clickhouse.clickhouse /etc/clickhouse-server/
chown -R clickhouse.clickhouse /etc/clickhouse-client/

4.4 启动服务

clickhouse-server --daemon --config-file=/etc/clickhouse-server/config.xml
systemctl start clickhouse

4.5 部署验证

# 1 未设置密码时
clickhouse-client

# 2 指定用户名和密码 
clickhouse-client -h 127.0.0.1 -u default --password password
clickhouse-client -h 127.0.0.1 --port 9000 -u default --password password  --multiline
# 指定sql命令方式
clickhouse-client -h 127.0.0.1 --port 9000 -u default --password password  --multiline -q "SELECT now()"
[root@db01 clickhouse-server]# clickhouse-client -h 127.0.0.1  -m -u default --password password
ClickHouse client version 20.10.3.30 (official build).
Connecting to 127.0.0.1:9000 as user default.
Connected to ClickHouse server version 20.10.3 revision 54441.

db01 :) SELECT * FROM system.clusters;

SELECT *
FROM system.clusters

┌─cluster─────────────────┬─shard_num─┬─shard_weight─┬─replica_num─┬─host_name───────┬─host_address────┬─port─┬─is_local─┬─user────┬─default_database─┬─errors_count─┬─estimated_recovery_time─┐
│ clickhouse_cluster_name │         1 │            1 │           1 │ 192.168.137.129 │ 192.168.137.129 │ 9000 │        1 │ default │                  │            0 │                       0 │
│ clickhouse_cluster_name │         1 │            1 │           2 │ 192.168.137.130 │ 192.168.137.130 │ 9000 │        0 │ default │                  │            0 │                       0 │
│ clickhouse_cluster_name │         2 │            1 │           1 │ 192.168.137.130 │ 192.168.137.130 │ 9000 │        0 │ default │                  │            0 │                       0 │
│ clickhouse_cluster_name │         2 │            1 │           2 │ 192.168.137.131 │ 192.168.137.131 │ 9000 │        0 │ default │                  │            0 │                       0 │
└─────────────────────────┴───────────┴──────────────┴─────────────┴─────────────────┴─────────────────┴──────┴──────────┴─────────┴──────────────────┴──────────────┴─────────────────────────┘

4 rows in set. Elapsed: 0.010 sec. 

db01 :) SELECT policy_name, volume_name, disks FROM system.storage_policies;

SELECT
    policy_name,
    volume_name,
    disks
FROM system.storage_policies

┌─policy_name─┬─volume_name───────┬─disks─────────────────────────┐
│ data_data2  │ data_data2_volume │ ['disk_name_1','disk_name_2'] │
│ default     │ default           │ ['default']                   │
└─────────────┴───────────────────┴───────────────────────────────┘

2 rows in set. Elapsed: 0.005 sec. 

db01 :) SELECT name,path,formatReadableSize(free_space) AS free,formatReadableSize(total_space) AS total,formatReadableSize(keep_free_space) AS reserved FROM system.disks;

SELECT
    name,
    path,
    formatReadableSize(free_space) AS free,
    formatReadableSize(total_space) AS total,
    formatReadableSize(keep_free_space) AS reserved
FROM system.disks

┌─name────────┬─path────────────────────┬─free──────┬─total─────┬─reserved─┐
│ default     │ /data/clickhouse/data/  │ 93.23 GiB │ 98.30 GiB │ 0.00 B   │
│ disk_name_1 │ /data/clickhouse/data/  │ 92.23 GiB │ 97.30 GiB │ 1.00 GiB │
│ disk_name_2 │ /data2/clickhouse/data/ │ 30.67 GiB │ 44.10 GiB │ 1.00 GiB │
└─────────────┴─────────────────────────┴───────────┴───────────┴──────────┘

3 rows in set. Elapsed: 0.007 sec.

五、权限资源管控

ClickHouse 的访问控制包括以下元素：

- 用户帐户

- 角色(Role)

- 行策略(Row Policy)

- 资源控制(Profile)

- 配额(Quota)

5.1 用户账户

DBA用户：默认使用default用户

BU用户：按照BU分配

#只读用户
phk_sac_read_opr
select
#读写用户
phk_sac_data
select
insert
update
delete
#测试、生产DDL权限暂收归DBA管理

5.2 角色

暂不通过角色进行权限管控

5.3 行策略

相当于过滤器，定义用户或角色可以使用哪些行

CREATE ROW POLICY filter ON mydb.mytable FOR SELECT USING id<1000 TO accountant, john@localhost
CREATE ROW POLICY filter ON mydb.mytable FOR SELECT USING id<1000 TO ALL EXCEPT test  # 对所有用户生效，除了test

5.4 资源控制

可以设置某些限制并分配给用户或角色

CREATE SETTINGS PROFILE low_mem SETTINGS max_threads = 4, max_memory_usage = 4000000000

5.5 配额

各种限制：

queries –请求总数。
errors –异常查询数。
result_rows –结果返回总行数。
read_rows –从表中读取的源行总数，以在所有远程服务器上运行查询。
execution_time –查询总时间，单位秒。

CREATE QUOTA qA FOR INTERVAL 60 MINUTE MAX QUERIES 100 TO CURRENT_USER