Clickhouse cluster installation record

1. Cluster machine planning

machine	Fragment number	Copy number
192.168.1.172	1	1
192.168.1.173	1	2
192.168.1.174	2	1
192.168.1.175	2	2

The operating system version of all installed machines is Centos7. The following installation operations are divided into two parts: local installation and cluster installation

2. Stand-alone installation

2.1 Check whether the SSE 4.2 instruction set is supported

grep -q sse4_2 /proc/cpuinfo && echo "SSE 4.2 supported" || echo "SSE 4.2 not supported"

2.2 Installation user preparation

Create a clickhouse user on all machines and set the sudo operation authority. All subsequent installations are based on this user.

2.3 Stand-alone RPM installation

Execute the following operation commands on each machine:

sudo yum install yum-utils
sudo rpm --import https://repo.clickhouse.tech/CLICKHOUSE-KEY.GPG
sudo yum-config-manager --add-repo https://repo.clickhouse.tech/rpm/stable/x86_64
sudo yum install clickhouse-server clickhouse-client

3. Cluster installation

3.1 Modify the /etc/clickhouse-server/config.xml configuration file

<?xml version="1.0"?>
<yandex>
    <logger>
        <!--日志输出级别-->
        <level>warning</level>
        <log>/var/log/clickhouse-server/clickhouse-server.log</log>
        <errorlog>/var/log/clickhouse-server/clickhouse-server.err.log</errorlog>
        <size>1000M</size>
        <count>10</count>
    </logger>

    <!--通过HTTP连接到服务器的端口-->
    <http_port>8123</http_port>

    <!--通过TCP协议与客户端进行通信的端口，即ClickHouse端口-->
    <tcp_port>9000</tcp_port>

    <!--通过MySQL协议与客户端通信的端口。-->
    <mysql_port>9004</mysql_port>

    <!--集群副本之间数据同步的端口-->
    <interserver_http_port>9009</interserver_http_port>
    
    <!--集群副本之间相互识别的主机名, 注意此处配置需要配置成每台机器各自的主机名/ip地址-->
    <interserver_http_host>192.168.1.172</interserver_http_host>
    
    <!--限制来源主机的请求, 如果要服务器回答所有请求-->
    <listen_host>0.0.0.0</listen_host>

    <max_connections>4096</max_connections>

    <!-- 使用http连接的时候的 -->
    <keep_alive_timeout>3</keep_alive_timeout>

    <!-- 最大的并发执行查询的数量 -->
    <max_concurrent_queries>100</max_concurrent_queries>

    <!--clickhouse-server使用的最大内存, 0表示不限制-->
    <max_server_memory_usage>0</max_server_memory_usage>

    <!--最大的线程池大小-->
    <max_thread_pool_size>10000</max_thread_pool_size>

    <max_server_memory_usage_to_ram_ratio>0.9</max_server_memory_usage_to_ram_ratio>


    <total_memory_profiler_step>4194304</total_memory_profiler_step>


    <total_memory_tracker_sample_probability>0</total_memory_tracker_sample_probability>

    <!--打开最大的文件数，默认最大值-->
    <max_open_files>262144</max_open_files>

    <!--表引擎从MergeTree使用的未压缩数据的缓存大小（以字节为单位，8G）。服务器有一个共享缓存，内存是按需分配的。
    如果启用，则使用高速缓存。在个别情况下，未压缩的缓存对于非常短的查询是有利的。
    -->
    <uncompressed_cache_size>8589934592</uncompressed_cache_size>

    <!--标记缓存的大小，用于MergeTree系列的表中。  以字节为单位，共享服务器的缓存，并根据需要分配内存。缓存大小必须至少为5368709120（5G）。-->
    <mark_cache_size>5368709120</mark_cache_size>

    <!-- 数据存放的位置, 这里的文件own必须是clickhouse -->
    <path>/var/lib/clickhouse/</path>

    <!--用于处理大型查询的临时数据的路径。-->
    <tmp_path>/var/lib/clickhouse/tmp/</tmp_path>

    <!--包含用户文件的目录，在表函数file()中使用-->
    <user_files_path>/var/lib/clickhouse/user_files/</user_files_path>


    <user_directories>
        <users_xml>
            <path>users.xml</path>
        </users_xml>
        <local_directory>
            <path>/var/lib/clickhouse/access/</path>
        </local_directory>
    </user_directories>

    <!--默认设置配置文件，在参数user_config中指定。-->
    <default_profile>default</default_profile>

    <default_database>default</default_database>

    <mlock_executable>true</mlock_executable>

    <remap_executable>false</remap_executable>

    <!--带替换文件的路径, 该路径中的文件配置数据会替换当前配置文件中带incl标记的属性-->
    <include_from>/etc/clickhouse-server/metrika.xml</include_from>

    <!--远程服务器，分布式表引擎和集群表功能使用的集群的配置-->
    <remote_servers incl="clickhouse_remote_servers" optional="true"/>
    
    <!--配置的集群需要zookeeper的配置-->
    <zookeeper incl="zookeeper_servers" optional="true"/>
    
    <!--这里定义的创建复制时用到的宏定义常量-->
    <macros incl="macros" optional="true"/>

    <!--重新加载内置词典的时间间隔（以秒为单位），默认3600。可以在不重新启动服务器的情况下“即时”修改词典-->
    <builtin_dictionaries_reload_interval>3600</builtin_dictionaries_reload_interval>

    <!--最大的客户端连接session超时时间-->
    <max_session_timeout>3600</max_session_timeout>

    <!--默认的客户端连接session超时时间-->
    <default_session_timeout>60</default_session_timeout>

    <!--查询记录在system.query_log表中-->
    <query_log>
        <database>system</database>
        <table>query_log</table>
        <partition_by>toYYYYMM(event_date)</partition_by>
        <!--将数据从内存中的缓冲区刷新到表的时间间隔-->
        <flush_interval_milliseconds>7500</flush_interval_milliseconds>
    </query_log>


    <!--trace_log系统表操作的设置。-->
    <trace_log>
        <database>system</database>
        <table>trace_log</table>
        <partition_by>toYYYYMM(event_date)</partition_by>
        <flush_interval_milliseconds>7500</flush_interval_milliseconds>
    </trace_log>

    <!--设置运行的查询的线程将根据query_thread_log服务器配置参数中的规则记录-->
    <query_thread_log>
        <database>system</database>
        <table>query_thread_log</table>
        <partition_by>toYYYYMM(event_date)</partition_by>
        <flush_interval_milliseconds>7500</flush_interval_milliseconds>
    </query_thread_log>

    <metric_log>
        <database>system</database>
        <table>metric_log</table>
        <flush_interval_milliseconds>7500</flush_interval_milliseconds>
        <collect_interval_milliseconds>1000</collect_interval_milliseconds>
    </metric_log>


    <asynchronous_metric_log>
        <database>system</database>
        <table>asynchronous_metric_log</table>
        <flush_interval_milliseconds>60000</flush_interval_milliseconds>
    </asynchronous_metric_log>


    <opentelemetry_span_log>
        <engine>
            engine MergeTree
            partition by toYYYYMM(finish_date)
            order by (finish_date, finish_time_us, trace_id)
        </engine>
        <database>system</database>
        <table>opentelemetry_span_log</table>
        <flush_interval_milliseconds>7500</flush_interval_milliseconds>
    </opentelemetry_span_log>


    <!--系统崩溃日志存储在system.crash_log表中-->
    <crash_log>
        <database>system</database>
        <table>crash_log</table>
        <partition_by />
        <flush_interval_milliseconds>1000</flush_interval_milliseconds>
    </crash_log>

    <!--外部词典的配置文件的路径，在config配置文件中指定-->
    <dictionaries_config>*_dictionary.xml</dictionaries_config>

    <!--MergeTree引擎表的数据压缩设置-->
    <compression incl="clickhouse_compression"></compression>

    <!--存储在zookeeper路径中的任务队列-->
    <distributed_ddl>
        <path>/clickhouse/task_queue/ddl</path>
    </distributed_ddl>

    <!--数据汇总设置-->
    <graphite_rollup_example>
        <pattern>
            <regexp>click_cost</regexp>
            <function>any</function>
            <retention>
                <age>0</age>
                <precision>3600</precision>
            </retention>
            <retention>
                <age>86400</age>
                <precision>60</precision>
            </retention>
        </pattern>
        <default>
            <function>max</function>
            <retention>
                <age>0</age>
                <precision>60</precision>
            </retention>
            <retention>
                <age>3600</age>
                <precision>300</precision>
            </retention>
            <retention>
                <age>86400</age>
                <precision>3600</precision>
            </retention>
        </default>
    </graphite_rollup_example>

    <!--包含输入格式文件（例如CapnProto格式的方案）的目录路径-->
    <format_schema_path>/var/lib/clickhouse/format_schemas/</format_schema_path>

    <!--基于Regexp的规则，应用于查询以及所有日志消息。再其存储在服务器日志中，system.query_log，system.text_log，system.processes表以及以日志形式发送给客户端。
    这样可以防止SQL查询中的敏感数据泄漏记录到日志中。
    -->
    <query_masking_rules>
        <rule>
            <name>hide encrypt/decrypt arguments</name>
            <regexp>((?:aes_)?(?:encrypt|decrypt)(?:_mysql)?)\s*\(\s*(?:'(?:\\'|.)+'|.*?)\s*\)</regexp>
            <replace>\1(???)</replace>
        </rule>
    </query_masking_rules>

</yandex>

Note that: path, interserver_http_host must be modified, and other attribute modifications should be adjusted according to the machine condition.

3.2 Create the /etc/clickhouse-server/metrika.xml file and fill in the following configuration information

<?xml  version="1.0"?>
<yandex>
    <!--这里配置集群的分布方案-->
    <clickhouse_remote_servers>
      <!--这里时2分片2副本的配置方案-->
      <cluster_2shards_2replicas>
        <!--这里配置一个分片-->
        <shard>
            <!--表示是否只将数据写入其中一个副本,然后集群间进行数据同步。默认为flase,表示写入所有副本-->
            <internal_replication>true</internal_replication>
            <!--这里配置分片在分布式表中的写权重->
            <weight>1</weight>
            <!--这里配置副本-->
            <replica>
                <host>192.168.1.172</host>
                <port>9000</port>
            </replica>
            <replica>
                <host>192.168.1.173</host>
                <port>9000</port>
            </replica>
        </shard>
        <shard>
            <internal_replication>true</internal_replication>
            <weight>1</weight>
            <replica>
                <host>192.168.1.174</host>
                <port>9000</port>
            </replica>
            <replica>
                <host>192.168.1.175</host>
                <port>9000</port>
            </replica>
        </shard>
      </cluster_2shards_2replicas>
    </clickhouse_remote_servers>

    <!--配置复制表的分片名和副本名字宏定义-->
    <macros>
        <shard>01</shard>
        <replica>192.168.1.172</replica>
    </macros>

    <!--配置zookeeper集群地址-->
    <zookeeper_servers>
        <node index="1">
            <host>192.168.1.173</host>
            <port>2181</port>
        </node>
        <node index="2">
            <host>192.168.1.174</host>
            <port>2181</port>
        </node>
        <node index="3">
            <host>192.168.1.175</host>
            <port>2181</port>
        </node>
    </zookeeper_servers>

   <networks>
     <ip>::/0</ip>
   </networks>

  <!--MergeTree引擎表的数据压缩设置-->
  <clickhouse_compression>
    <case>
      <!--数据部分的最小大小-->
      <min_part_size>10000000000</min_part_size>
      <!--数据部分大小与表大小的比率-->
      <min_part_size_ratio>0.01</min_part_size_ratio>
      <!--压缩算法，zstd和lz4-->
      <method>lz4</method>
    </case>
  </clickhouse_compression>
</yandex>

3.3 After the above operations are completed, start the server and client connection for testing

1. Start the configuration of clickhouse at each node respectively, observe whether there are errors reported, if any, check the configuration file and try again

clickhouse clickhouse-server --config-file=/etc/clickhouse-server/config.xml // 前台启动的方式，可以实时把日志输出到控制台方便进行排错
sudo clickhouse /etc/init.d/clickhouse-server start // 后台启动方式，日志目录在/var/log/clickhouse-server
sudo service clickhouse-server start // 后台启动方式, 日志目录在/var/log/clickhouse-server

2. Start the client

clickhouse-client -h 192.168.1.173 --port 9000 // 启动所有结点的客户端连接clickhouse当前所在结点的数据库

Execute the following sql statement on all clients to check whether the cluster's shards and replicas are installed correctly. After the execution, the information of the shards and replicas that we configured will be displayed.

select * from system.clusters;

3. Build a table and insert data to test

create database test on cluster cluster_2shards_2replicas;

CREATE TABLE test.test_data_local ON CLUSTER cluster_2shards_2replicas (id UInt64) \
ENGINE = ReplicatedMergeTree('/clickhouse/tables/{shard}/test_data', '{replica}') \
ORDER BY id;

CREATE TABLE test.test_data_all ON CLUSTER cluster_2shards_2replicas (id UInt64) \
ENGINE= Distributed(cluster_2shards_2replicas, test, test_data_local, rand());

insert into test.test_data_all (id)values(1231110);
insert into test.test_data_all (id)values(1231113);
insert into test.test_data_all (id)values(1231116);
insert into test.test_data_all (id)values(1231222);
 
在客户端上执行以上SQL然后再在每个节点上启动客户端连接当前的结点机器执行以下查询，然后观察各个结点客户端是否返回正确的一样的数据：
select * from test.test_data_all;