Clickhouse 集群搭建

版权声明:本文为博主原创文章,转载请注明出处 https://blog.csdn.net/vkingnew/article/details/84565100
规划:
IP地址  角色   端口
192.168.0.151  clickhouse 9000  zookeeper2181
192.168.0.152  clickhouse 9000  zookeeper2181
192.168.0.153  clickhouse 9000  zookeeper2181

下载软件:
# curl -s https://packagecloud.io/install/repositories/Altinity/clickhouse/script.rpm.sh | sudo bash
--安装软件依赖:依次提供python3和mysql-community-devel(libmysqlclient)
sudo yum install -y epel-release
sudo yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
sudo yum install -y https://dev.mysql.com/get/mysql57-community-release-el7-11.noarch.rpm

--软件搜索:
# yum search clickhouse
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: centos.ustc.edu.cn
 * epel: mirrors.aliyun.com
 * extras: mirrors.aliyun.com
 * updates: mirrors.cn99.com
 =================== N/S matched: clickhouse ======================= 
clickhouse-client.x86_64 : clickhouse client binary
clickhouse-common-static.x86_64 : clickhouse common static binaries
clickhouse-compressor.x86_64 : clickhouse compressor binary
clickhouse-debuginfo.x86_64 : Debug information for package clickhouse
clickhouse-mysql.noarch : MySQL to ClickHouse data migrator
clickhouse-odbc.x86_64 : Yandex ClickHouse DBMS ODBC driver
clickhouse-server.x86_64 : Server files for clickhouse
clickhouse-server-common.x86_64 : Common configuration files for clickhouse
clickhouse-test.x86_64 : clickhouse test suite

  Name and summary matches only, use "search all" for everything.
--软件安装和保存:
# cat /etc/yum.conf 
[main]
cachedir=/var/cache/yum/$basearch/$releasever
keepcache=1
debuglevel=2
logfile=/var/log/yum.log
exactarch=1
obsoletes=1
gpgcheck=1
plugins=1
installonly_limit=5
注意:将keepcache=0 修改为keepcache=1即可保存安装的RPM软件。

注释:若只是需要安装clickhouse-mysql则至少需要安装如下软件依赖包:
  clickhouse-client
  mysql-community-devel
  mariadb-devel
  gcc
  python34-devel 
  python34-pip
安装clickhouse-mysql需要安装如下python软件依赖包:
#sudo pip3 install clickhouse-mysql
pip3 install mysqlclient
pip3 install mysql-replication
pip3 install clickhouse-driver
通过RPM安装的可以自动安装软件: pycparser, cffi, six, asn1crypto, idna, cryptography, pymysql, mysql-replication
软件安装:
# yum -y install clickhouse-client clickhouse-common-static clickhouse-compressor clickhouse-mysql clickhouse-odbc clickhouse-server clickhouse-server-common

---配置文件说明:
默认配置文件位置:
# rpm -ql clickhouse-server-common
/etc/clickhouse-server/config.xml
/etc/clickhouse-server/users.xml
ClickHouse有几核心的配置文件:
config.xml 端口配置、本地机器名配置、内存设置等
metrika.xml 集群配置、ZK配置、分片配置等
users.xml 权限、配额设置
默认的数据目录、临时目录、日志目录:
/var/lib/clickhouse
/var/lib/clickhouse/tmp/
/var/log/clickhouse-server

# rpm -ql clickhouse-server
/etc/cron.d/clickhouse-server            定时任务脚本
/etc/init.d/clickhouse-server            默认启动脚本
/etc/security/limits.d/clickhouse.conf   安全配置
/usr/bin/clickhouse-clang                提供的二进制命令
/usr/bin/clickhouse-copier
/usr/bin/clickhouse-format
/usr/bin/clickhouse-lld
/usr/bin/clickhouse-obfuscator
/usr/bin/clickhouse-odbc-bridge
/usr/bin/clickhouse-report
/usr/bin/clickhouse-server
/usr/share/clickhouse
/usr/share/clickhouse/bin
/usr/share/clickhouse/headers
/usr/share/clickhouse/headers/18.14.15

# cat /etc/security/limits.d/clickhouse.conf 
clickhouse      soft    nofile  262144
clickhouse      hard    nofile  262144
说明:由于默认的数据目录不便于管理,需要修改下。
--clickhouse的启动和关闭:
clickhouse-server --config-file=/etc/clickhouse-server/config.xml
或:
systemctl stop clickhouse-server
systemctl start clickhouse-server
--登录:
# clickhouse-client --host=197.255.20.64 --user=default --password=oracle --port=9900
ClickHouse client version 18.14.15.
Connecting to 197.255.20.64:9900 as user default.
Connected to ClickHouse server version 18.14.15 revision 54409.
--登录进入之后clickhouse默认有两个数据库:
production :) show databases;

SHOW DATABASES

┌─name────┐
│ default │
│ system  │
└─────────┘

2 rows in set. Elapsed: 0.004 sec. 
production :) select * from system.clusters;

SELECT *
FROM system.clusters 

┌─cluster─────────────────────┬─shard_num─┬─shard_weight─┬─replica_num─┬─host_name───────────────────────┬─host_address──┬─port─┬─is_local─┬─user────┬─default_database─┐
│ bip_ck_cluster              │         1 │            1 │           1 │ 197.255.20.62                   │ 197.255.20.62 │ 9900 │        0 │ default │                  │
│ bip_ck_cluster              │         2 │            1 │           1 │ 197.155.20.64                   │ 197.155.20.64 │ 9900 │        0 │ default │                  │
│ bip_ck_cluster              │         3 │            1 │           1 │ 197.255.20.65                   │ 197.255.20.65 │ 9900 │        0 │ default │                  │
                │
└─────────────────────────────┴───────────┴──────────────┴─────────────┴─────────────────────────────────┴───────────────┴──────┴──────────┴─────────┴──────────────────┘
--创建数据库:
roduction :) create database if not exists wuhan;

CREATE DATABASE IF NOT EXISTS wuhan

Ok.

0 rows in set. Elapsed: 0.006 sec. 

production :) show databases;

SHOW DATABASES

┌─name────┐
│ default │
│ system  │
│ wuhan   │
└─────────┘

3 rows in set. Elapsed: 0.005 sec. 

--将MySQL的数据实时同步到clickhouse:
curl -s https://packagecloud.io/install/repositories/altinity/clickhouse/script.rpm.sh | sudo bash

安装EPEL和python3 以及mysql的客户端工具:
sudo yum install -y epel-release 
sudo yum install -y https://dev.mysql.com/get/mysql57-community-release-el7-11.noarch.rpm
或者:
sudo yum install -y https://download.fedoraproject.org/pub/epel/7/x86_64/Packages/e/epel-release-7-11.noarch.rpm
sudo yum install -y clickhouse-mysql
--配置mysql:
sudo cp /etc/clickhouse-mysql/clickhouse-mysql-example.conf /etc/clickhouse-mysql/clickhouse-mysql.conf 
sudo vim /etc/clickhouse-mysql/clickhouse-mysql.conf
--使用clickhouse-mysql直接同步单个表的命令:
 clickhouse-mysql \
--src-server-id=1 \
--src-resume \
--src-wait \
--nice-pause=1 \
--src-host=192.168.3.191 \
--src-user=root \
--src-password=abcd@1234 \
--src-tables=ZHIXIAOQIYE.zhixiao_data \
--dst-host=192.168.3.194 \
--dst-schema=default \  # 指定clickhouse数据库
--dst-table=zhixiao \   # 指定clickhouse数据表
--csvpool \
--csvpool-file-path-prefix=qwe_ \
--mempool-max-flush-interval=60 \
--mempool-max-events-num=10000 
--------------------- 
--可以实时同步MySQL的多个库的数据:
--通过csv文件导入:

--参考:
https://github.com/Altinity/clickhouse-mysql-data-reader#rpm-installation

猜你喜欢

转载自blog.csdn.net/vkingnew/article/details/84565100