Due to the frequent release of clickhouse, the current version is 20.7 and does not support synchronization with mysql, so the latest version 20200909 on git is compiled-the version number is 20.9.
Environmental information
系统版本：Red Hat Enterprise Linux Server release 7.7 (Maipo)
uname -a:3.10.0-1062.el7.x86_64服务器配置：32c 250g 1T（使用8c16g虚拟机编译失败，原因为内存不足导致编译进程OOM）
mysql版本：8.0.19
gcc版本：9.3.0
cmake版本：3.14.5
ninja版本：1.9.0
1. Check before installation
grep -q sse4_2 /proc/cpuinfo && echo "SSE 4.2 supported" || echo "SSE 4.2 not supported"
2. Install dependent packages
yum install git cmake ninja-build libicu-devel clang libicu-devel readline-devel mysql-devel openssl-devel unixODBC_devel bzip2 -y
3. Source code acquisition
git clone --recursive https://github.com/ClickHouse/ClickHouse
Four. Install the high version of gcc
Download the installation package
ftp://gnu.mirror.iweb.com/gcc/gcc-9.3.0/gcc-9.3.0.tar.xz
Unzip
tar xvf gcc-9.3.0.tar.xz
installation
cd gcc-9.3.0
./contrib/download_prerequisites
This step will download the dependency package, if you cannot connect to the external network, you can download it through the link
链接：https://pan.baidu.com/s/1JHv1WAS43S89LXA6PkevTg 提取码：nb35
The package name is as follows
gmp-6.1.0.tar.bz2mpfr-3.1.4.tar.bz2
mpc-1.0.3.tar.gzisl-0.18.tar.bz2
Usage: just put the package cp to the gcc-9.3.0 directory
Then execute
./contrib/download_prerequisites ---与上一步相同无需重复执行
mkdir build
cd build../configure --prefix=/opt/gcc9 --enable-languages=c,c++ --disable-multilib
export THREADS=$(grep -c ^processor /proc/cpuinfo)
make -j $THREADS
make install
It takes longer to compile=. =
Subsequent operations will cause errors similar to the following due to the gcc upgrade
ninja: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by ninja)
The solution is as follows
find / -name "libstdc++.so.6*"
Find the gcc directory with the highest version in the system, such as the previous installation directory
/opt/gcc9/lib64/libstdc++.so.6.0.28
/opt/gcc9/lib64/libstdc++.so.6
/opt/gcc9/lib64/libstdc++.so.6.0.28-gdb.py
Backup libstdc++.so that reported errors
mv /lib64/libstdc++.so.6 /lib64/libstdc++.so.6_bak
Copy the higher version of libstdc++ to the /lib64 directory
cp /opt/gcc9/lib64/libstdc++.so.6 /lib64/libstdc++.so.6
ln -s /opt/gcc9/lib64/libstdc++.so.6.0.28 /lib64/libstdc++.so.6 ----(一般执行上一步就可以了)
verification
[root@mini test]# ./ninja --version
1.9.0
Five. Install cmake 3 version
Install cmake 3, keep the replacement system default preparation software
wget https://cmake.org/files/v3.14/cmake-3.14.5-Linux-x86_64.tar.gz
tar zxvf cmake-3.14.5-Linux-x86_64.tar.gz -C /opt
ln -s cmake-3.14.5-Linux-x86_64 cmake
Add environment variables
vim /etc/profile
export CMAKE_HOME=/opt/cmake
export PATH=$CMAKE_HOME/bin:$PATH
source /etc/profile
verification
[root@mini test]# cmake --version
cmake version 3.14.5
CMake suite maintained and supported by Kitware (kitware.com/cmake).
6. Install ninja-1.9.0
download
wget https://github.com/ninja-build/ninja/releases/download/v1.9.0/ninja-linux.zip
unzip ninja-linux.zip -d /usr/local/bin/
Add environment variables
test version:
# ninja --version
1.9.0
Error handling:
[root@mini test]# ./ninja
./ninja: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by ./ninja)
./ninja: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by ./ninja)
See the treatment method: gcc 9.3.0 compile and install
Seven. Source installation Clickhouse
cd ClickHouse
mkdir build
cd build
export CC=gcc-9
export CXX=g++-9
cmake ..ninja
The process takes about 3 hours
After the installation is successful, copy the clickhouse command and the parameter file to the specified directory
mkdir -p /usr/local/clickhouse/etccd /tools/ClickHouse/programs/server ---其中/tools/ClickHouse 为clickhouse git文件根目录
cp config.xml /usr/local/clickhouse/etc
cp users.xml /usr/local/clickhouse/etc
Copy the clickhouse execution file to /usr/local/clickhouse/bin
mkdir -p /usr/local/clickhouse/bin
cp /tools/ClickHouse/build/programs/clickhouse /usr/local/clickhouse/bin
Configure environment variables
vim /etc/profile添加:/usr/local/clickhouse/bin
source /etc/profiel
Start clickhouse-server, pay attention to set the PATH directory in the configuration file before starting (clickhouse data directory and assign the rights to the corresponding directory)
nohup clickhouse server --config-file=/usr/local/clickhouse/etc/config.xml > /tmp/clickhouse.log 2>&1 &
Start the client
clickhouse client
Output:
ClickHouse client version 20.9.1.1.
Connecting to localhost:9000 as user default.
to ClickHouse server version 20.9.1 revision 54439.
XXXXXXXX :)
At this point clickhouse compilation and installation are complete, the following is the process of building mysql replication
Prepare a set of mysql that has a smooth network with the clickhouse server. The MySQL version used in this article is 8.0.19
1. Create a copy user (the minimum permissions required are not tested)
mysql> create user clickhouse@'%' identified WITH mysql_native_password by 'rpl_user';
mysql> grant ALL PRIVILEGES on *.* to clickhouse@'%';
2. Adjust the clickhouse parameters
clickhouse :) SET allow_experimental_database_materialize_mysql=1;
Clickhouse creates replication (currently replicates with database as the unit, different databases can come from different MySQL masters, so that multiple MySQL source data can be synchronized to one ClickHouse for OLAP analysis.)
CREATE DATABASE sbtest ENGINE = MaterializeMySQL(
'MYSQL_IP:MYSQL_PORT', 'DATABASE_NAME', 'USER_NAME', 'PASSWORD');
CREATE DATABASE sbtest ENGINE = MaterializeMySQL(
'XXXXXXXX:3306', 'clickhouse', 'clickhous', 'rpl_user');
If an error is reported
Received exception from server (version 20.9.1):Code: 537. DB::Exception: Received from localhost:9000. DB::Exception: Illegal MySQL variables, the MaterializeMySQL engine requires default_authentication_plugin='mysql_native_password'.
Adjust the parameters on the MYSQL side and restart (read_only parameter) default_authentication_plugin='mysql_native_password'
Three. Create a database and table, and write data
mysql> create database clickhouse;
mysql> use clickhouse;mysql> CREATE TABLE `sbtest1` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`k` int(10) unsigned NOT NULL DEFAULT '0',
`c` char(120) NOT NULL DEFAULT '',
`pad` char(60) NOT NULL DEFAULT '',
PRIMARY KEY (`id`),
KEY `k_1` (`k`))
ENGINE = InnoDB AUTO_INCREMENT = 5000001 DEFAULT CHARSET = utf8mb4 MAX_ROWS = 1000000;
mysql> INSERT INTO `sbtest1` VALUES (1,49929,...; ---sysbench数据
clickhouse> use sbtest;
clickhouse> show tables;SHOW TABLES
┌─name────┐
│ sbtest1 │
└─────────┘
1 rows in set. Elapsed: 0.002 sec.
clickhouse>select * from sbtest1;
SELECT *
FROM sbtest1
┌─id─┬─────k─┬─c───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬─pad─────────────────────────────────────────────────────────┐
│ 12 │ 48776 │ 06636928111-91412549319-87017689961-79500497523-29051692073-64380774874-78643769852-73373361096-34215043106-34370178281 │ 89292458800-98111883088-45119613990-18776802947-72334127545 │
.......略
20 rows in set. Elapsed: 0.004 sec.
View synchronization sites
Look at the synchronization site of ClickHouse
cat /var/lib/clickhouse/metadata/clickhouse/.metadata
Version: 2
Binlog File: bin.000004
Executed GTID: 45b22def-f329-11ea-acf1-3497f600d5bb:1-69:1000068
Binlog Position: 10006
So far, the slave is successfully built.
| About the author
Zhao Zizhou| Currently working at Sweet Orange Finance (China Telecom Yipay), a database engineer.
Mainly responsible for the operation and maintenance of MySQL, Redis, MongoDB, TIDB and other related open source databases.
Like to learn and share technology, and improve with everyone!
The full text is over.
Enjoy ClickHouse ：）
Zhishutang boutique courses, MySQL, SQL optimization, Python, Elastic, etc.
The MySQL course has been fully upgraded to MySQL 8.0, and now it’s just right to get on the train. Let’s start the journey of MySQL 8.0.
Join Zhishutang
Challenge 500,000 + annual salary!
Zhishutang "MySQL Practical Optimization Class" has officially launched Tencent Classroom, scan the code to join the group to learn about the details of the course, and there are discounts for teaching assistants.