Article Directory

一、MaterializeMySQL
- 1.1 Simple comparison between MySQL and CK
- 1.2 MaterializeMySQL principle
2. Real-time replication of MySQL->CK
- 1.1 Environmental preparation
- 1.2 Basic function test
3. Follow-up questions and thoughts

MySQL and Clickhouse are two completely different databases, both of which have their own advantages and disadvantages, and the business scenarios they are suitable for are also different. In actual business, we need to choose the right one according to the characteristics and advantages of the database itself. Business scene. Although the traditional MySQL database supports OLTP business very well, there are still large certain performance bottlenecks in business scenarios such as OLAP that require statistical analysis of large quantities of data. If we want to do better BI statistical analysis and query on the core online business data, we must face how to synchronize the online data to the corresponding statistical analysis database. Starting in 2020, Clickhouse has launched the MaterializeMySQL engine, which realizes the use of Clickhouse as a slave database of MySQL, real-time synchronization of MySQL data, and a leap from OLTP to OLAP.

一、MaterializeMySQL

1.1 Simple comparison between MySQL and CK

The main similarities and differences between MySQL and Clickhouse are as follows:

MySQL	Clickhouse
Relational database, supporting things	Distributed column database, does not support transactions
Row storage mode, suitable for reading as little row data as possible	Column storage mode, and high data compression ratio, has a natural advantage for mass data reading
Single-process multi-threaded service, a single business request query cannot effectively utilize multiple CPU resources	Multi-core parallel
For OLTP business	OLAP business oriented to online analytical processing

1.2 MaterializeMySQL principle

In the second half of 2020, Yandex Company released the MaterializeMySQL engine in the ClickHouse community, which is mainly used to support full and incremental real-time data synchronization from MySQL. Currently supports MySQL 5.6/5.7/8.0 version, compatible with Delete/Update statements, and most commonly used DDL operations.

As we all know, MySQL's own replication mainly relies on the binlog transaction log, and the MaterializeMySQL engine is no exception. However, the difference between the binlog replication of native MySQL is that due to the grammatical differences between the two, MaterializeMySQL does not convert the SQL statements in the event into specific statements in the CK for execution, but directly converts the Binlog Event into the underlying Block structure. Then it is written directly to the underlying storage engine, which is close to physical replication.

MaterializeMySQL implementation process:

MaterializeMySQL supports database-level replication.
After creating a database-level copy in Clickhouse, clickhouse connects to the data via TCP/IP through our designated database account, executes Flush table with read lock on the database, and obtains related binlog and table structure metadata information; after the metadata is copied Release the global read-only lock, and start copying table data information through select * from table_name.
For subsequent incremental data synchronization, MaterializeMySQL realizes real-time synchronization through the analysis of binlog event (MYSQL_QUERY_EVENT(DDL), MYSQL_WRITE_ROWS_EVENT(insert), MYSQL_UPDATE_ROWS_EVENT(update), MYSQL_DELETE_ROWS_EVENT(delete)
For DDL operations, MaterializeMySQL uses the primary key of the MySQL table data as the sort key and partition key of the CK table by default, but due to the difference between the data definitions of Clickhouse and MySQL, the DDL statement will also be converted accordingly
For Update/Delete operations, MaterializeMySQL introduces the hidden field of _version, which is used for version control and combines the _sign field to mark the validity of data

When MaterializeMySQL creates a replication channel, during the full initialization synchronization phase, you can view the specific execution details of MySQL through general_log. The specific execution logs are as follows:

2021-03-14T15:40:02.016351+08:00	   26 Connect	[email protected] on ck_test using TCP/IP
2021-03-14T15:40:02.017402+08:00	   26 Query	SET NAMES utf8
2021-03-14T15:40:02.018822+08:00	   26 Query	SHOW VARIABLES WHERE (Variable_name = 'log_bin' AND upper(Value) = 'ON') OR (Variable_name = 'binlog_format' AND upper(Value) = 'ROW') OR (Variable_name = 'binlog_row_image' AND upper(Value) = 'FULL') OR (Variable_name = 'default_authentication_plugin' AND upper(Value) = 'MYSQL_NATIVE_PASSWORD')
2021-03-14T15:40:02.032549+08:00	   26 Query	SELECT version() AS version
2021-03-14T15:40:02.033620+08:00	   26 Query	FLUSH TABLES
2021-03-14T15:40:02.051444+08:00	   26 Query	FLUSH TABLES WITH READ LOCK
2021-03-14T15:40:02.052364+08:00	   26 Query	SHOW MASTER STATUS
2021-03-14T15:40:02.053295+08:00	   26 Query	SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ
2021-03-14T15:40:02.054027+08:00	   26 Query	START TRANSACTION /*!40100 WITH CONSISTENT SNAPSHOT */
2021-03-14T15:40:02.055520+08:00	   26 Query	SELECT TABLE_NAME AS table_name FROM INFORMATION_SCHEMA.TABLES  WHERE TABLE_SCHEMA = 'ck_test'
2021-03-14T15:40:02.057527+08:00	   26 Query	SHOW CREATE TABLE ck_test.t1
2021-03-14T15:40:02.060780+08:00	   26 Query	SHOW CREATE TABLE ck_test.t2
2021-03-14T15:40:02.062275+08:00	   26 Query	UNLOCK TABLES
2021-03-14T15:40:02.075348+08:00	   26 Query	SELECT * FROM ck_test.t2
2021-03-14T15:40:02.101797+08:00	   26 Query	SELECT * FROM ck_test.t1
2021-03-14T15:40:02.106329+08:00	   26 Query	COMMIT
2021-03-14T15:40:02.109026+08:00	   27 Connect	[email protected] on  using TCP/IP
2021-03-14T15:40:02.109637+08:00	   27 Query	SET @master_binlog_checksum = 'CRC32'
2021-03-14T15:40:02.110123+08:00	   27 Query	SET @master_heartbeat_period = 1000000000
2021-03-14T15:40:02.111290+08:00	   27 Binlog Dump GTID	Log: '' Pos: 4 GTIDs: '4a2dfc1c-1f50-11eb-a38b-fa057042bc00:1-53,
a4ec8037-1a70-11eb-91ff-fa9f1ef63700:1-1741042'

2. Real-time replication of MySQL->CK

1.1 Environmental preparation

1、MySQL

Enable binlog log, and row_format=row
Copy using gtid mode

gtid_mode=ON
enforce_gtid_consistency=1
binlog_format=ROW

2、Clickhouse

1) Environmental parameters

-- 该参数默认关闭，若需要使用MaterializeMySQL引擎，必须打开该参数
mdw :) set allow_experimental_database_materialize_mysql=1;

2) Create a replication channel

-- 语法
CREATE DATABASE ${dbname} ENGINE = MaterializeMySQL('${mysql_ip}:${mysql_port}', '${mysql_dbname}', '${mysql_user}', '${mysql_passoword}');

-- 执行SQL
mdw :) CREATE DATABASE ck_test ENGINE = MaterializeMySQL('172.16.104.13:3306', 'ck_test', 'root', '123');

3) Copy information

For MySQL real-time replication information, it is stored in the metadata directory under datadir.

-- MySQL的binlog位点信息
root@mysql 15:05:  [ck_test]> show master status\G
*************************** 1. row ***************************
             File: mysql-bin.000005
         Position: 4048
     Binlog_Do_DB:
 Binlog_Ignore_DB:
Executed_Gtid_Set: 4a2dfc1c-1f50-11eb-a38b-fa057042bc00:1-37,
a4ec8037-1a70-11eb-91ff-fa9f1ef63700:1-1741042
1 row in set (0.00 sec)

-- Clickhouse复制位点信息
[root@mdw ck_test]# pwd
/data/clickhouse-server/data/metadata/ck_test
[root@mdw ck_test]# cat .metadata
Version:	2
Binlog File:	mysql-bin.000005                                            //binlog文件
Executed GTID:	4a2dfc1c-1f50-11eb-a38b-fa057042bc00:1-37,a4ec8037-1a70-11eb-91ff-fa9f1ef63700:1-1741042        //GTID信息
Binlog Position:	4048                                                    //binlog位点
Data Version:	9                                                           //数据版本信息，全局递增

1.2 Basic function test

1. Synchronization of data writing

For all MySQL data, there will be corresponding _sign and _version hidden fields in CK, which are used to mark and query version control. ,

-- MySQL
root@mysql 14:44:  [ck_test]> create table t2(id int primary key not null auto_increment,name varchar(2));
Query OK, 0 rows affected (0.03 sec)

root@mysql 14:45:  [ck_test]> insert into t2 values(null,'aa'),(null,'bb');
Query OK, 2 rows affected (0.01 sec)
Records: 2  Duplicates: 0  Warnings: 0

root@mysql 14:45:  [ck_test]> select * from t2;
+----+------+
| id | name |
+----+------+
|  1 | aa   |
|  2 | bb   |
+----+------+
2 rows in set (0.01 sec)

-- Clickhouse
mdw :) select * from ck_test.t2 order by id ;

SELECT *
FROM ck_test.t2
ORDER BY id ASC

┌─id─┬─name─┐
│  1 │ aa   │
└────┴──────┘
┌─id─┬─name─┐
│  2 │ bb   │
└────┴──────┘

2 rows in set. Elapsed: 0.009 sec.

-- Clickhouse隐藏字段查询
mdw :) select *,_sign,_version from ck_test.t2 order by id;

SELECT
    *,
    _sign,
    _version
FROM ck_test.t2
ORDER BY id ASC

┌─id─┬─name─┬─_sign─┬─_version─┐
│  1 │ aa   │     1 │        7 │                //一次性写入的_version=7一致，由于是insert操作，_sign=1。
│  2 │ bb   │     1 │        7 │
└────┴──────┴───────┴──────────┘

2 rows in set. Elapsed: 0.003 sec.

2. Data update

For the MySQL Update operation, when we directly query the CK table data, we can see that the table data has been "updated" normally, but when we additionally query the _sign, _version information, we can find that the data is not physically deleted before the update . Therefore, the method of Clickhouse Update operation is actually to write the changed record and mark _sign=1,_version=${current version}+1. When we execute the query, CK will help us finally return as Normal result set information.

-- MySQL
root@mysql 14:45:  [ck_test]> select * from t2;
+----+------+
| id | name |
+----+------+
|  1 | aa   |
|  2 | bb   |
+----+------+
2 rows in set (0.01 sec)

root@mysql 14:47:  [ck_test]> update t2 set name='aaa' where id=1;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

root@mysql 14:47:  [ck_test]> select * from t2;
+----+------+
| id | name |
+----+------+
|  1 | aaa  |
|  2 | bb   |
+----+------+
2 rows in set (0.01 sec)

-- Clickhouse
mdw :) select * from ck_test.t2 order by id ;                   //可以看到，正常的查询，CK已经帮我们做了处理

SELECT *
FROM ck_test.t2
ORDER BY id ASC

┌─id─┬─name─┐
│  1 │ aaa  │
└────┴──────┘
┌─id─┬─name─┐
│  2 │ bb   │
└────┴──────┘

2 rows in set. Elapsed: 0.014 sec.

mdw :) select *,_sign,_version from ck_test.t2 order by id;

SELECT
    *,
    _sign,
    _version
FROM ck_test.t2
ORDER BY id ASC

┌─id─┬─name─┬─_sign─┬─_version─┐
│  1 │ aaa  │     1 │        8 │                                //update更新后的数据被写入ck表，_version+1
└────┴──────┴───────┴──────────┘
┌─id─┬─name─┬─_sign─┬─_version─┐
│  1 │ aa   │     1 │        7 │
│  2 │ bb   │     1 │        7 │
└────┴──────┴───────┴──────────┘

3 rows in set. Elapsed: 0.008 sec.

3. Data deletion

For the Delete operation, when we directly query the CK table data, we can see that the table data has been "deleted" normally, but when we additionally query the _sign, _version information, we can find that the data is not physically deleted before the update, and It is to add a new row of row record data that needs to be deleted, and mark _sign=-1, _version=current version +1, so for the delete operation of ck, the record will not be physically deleted directly, but it will be deleted by sign mark. When we execute the query query, CK helps us finally return to normal result set information according to version control.

-- MySQL
root@mysql 14:47:  [ck_test]> select * from t2;
+----+------+
| id | name |
+----+------+
|  1 | aaa  |
|  2 | bb   |
+----+------+
2 rows in set (0.00 sec)

root@mysql 14:48:  [ck_test]> delete from t2 where id=2;
Query OK, 1 row affected (0.01 sec)

root@mysql 14:48:  [ck_test]> select * from t2;
+----+------+
| id | name |
+----+------+
|  1 | aaa  |
+----+------+
1 row in set (0.01 sec)

-- Clickhouse
mdw :) select * from ck_test.t2 order by id ;                           //query操作正常显示结果集信息

SELECT *
FROM ck_test.t2
ORDER BY id ASC

┌─id─┬─name─┐
│  1 │ aaa  │
└────┴──────┘

1 rows in set. Elapsed: 0.009 sec.

mdw :) select *,_sign,_version from ck_test.t2 order by id;

SELECT
    *,
    _sign,
    _version
FROM ck_test.t2
ORDER BY id ASC

┌─id─┬─name─┬─_sign─┬─_version─┐
│  1 │ aaa  │     1 │        8 │
└────┴──────┴───────┴──────────┘
┌─id─┬─name─┬─_sign─┬─_version─┐
│  1 │ aa   │     1 │        7 │
│  2 │ bb   │    -1 │        9 │                                            //新增删除记录行，并标记_sign=-1表示删除操作
│  2 │ bb   │     1 │        7 │
└────┴──────┴───────┴──────────┘

4 rows in set. Elapsed: 0.008 sec.

4 、 DDL

1) For the CK table data structure information, since MaterializeMySQL does not support the show create xx syntax for the time being, we can view the table structure information created by CK through the corresponding physical file.

[root@mdw metadata]# cat ck_test
cat: ck_test: 是一个目录
[root@mdw metadata]# cat ck_test.sql
ATTACH DATABASE ck_test
ENGINE = MaterializeMySQL('172.16.104.13:3306', 'ck_test', 'root', '123')
[root@mdw metadata]# cat ck_test/t2.sql
ATTACH TABLE t2
(
    `id` Int32,
    `name` Nullable(String),
    `_sign` Int8 MATERIALIZED 1,
    `_version` UInt64 MATERIALIZED 1
)
ENGINE = ReplacingMergeTree(_version)
PARTITION BY intDiv(id, 4294967)                                    //分区键、排序键均由MySQL表数据主键继承
ORDER BY tuple(id)
SETTINGS index_granularity = 8192

2) New field DDL operation

-- MySQL新增表字段
root@mysql 14:48:  [ck_test]> alter table t2 add age int;
Query OK, 0 rows affected (0.07 sec)
Records: 0  Duplicates: 0  Warnings: 0

root@mysql 14:52:  [ck_test]> select * from t2;
+----+------+------+
| id | name | age  |
+----+------+------+
|  1 | aaa  | NULL |
+----+------+------+
1 row in set (0.00 sec)


-- Clickhouse表字段信息
[root@mdw metadata]# cat ck_test/t2.sql
ATTACH TABLE t2
(
    `id` Int32,
    `name` Nullable(String),
    `age` Nullable(Int32),                                          //CK表同步MySQL新增字段，对于ck这种列存储来讲，新增字段的操作还是比较简单的
    `_sign` Int8 MATERIALIZED 1,
    `_version` UInt64 MATERIALIZED 1
)
ENGINE = ReplacingMergeTree(_version)
PARTITION BY intDiv(id, 4294967)
ORDER BY tuple(id)

mdw :) select * from ck_test.t2 order by id ;

SELECT *
FROM ck_test.t2
ORDER BY id ASC

┌─id─┬─name─┬──age─┐
│  1 │ aaa  │ ᴺᵁᴸᴸ │
└────┴──────┴──────┘

1 rows in set. Elapsed: 0.010 sec.

mdw :) select *,_sign,_version from ck_test.t2 order by id;

SELECT
    *,
    _sign,
    _version
FROM ck_test.t2
ORDER BY id ASC

┌─id─┬─name─┬──age─┬─_sign─┬─_version─┐
│  1 │ aa   │ ᴺᵁᴸᴸ │     1 │        7 │
│  1 │ aaa  │ ᴺᵁᴸᴸ │     1 │        8 │
│  2 │ bb   │ ᴺᵁᴸᴸ │    -1 │        9 │
│  2 │ bb   │ ᴺᵁᴸᴸ │     1 │        7 │
└────┴──────┴──────┴───────┴──────────┘

4 rows in set. Elapsed: 0.023 sec.

3. Follow-up questions and thoughts

For Clickhouse to create a replication, it is necessary to obtain a global read lock and perform a full table scan query of the growth table, which are some resource-intensive operations. Therefore, it is still recommended to copy from the library with binlog and log_slave_updates as the source for ck replication.
When ck creates a replication channel, the minimum database user permissions required are: SELECT, REPLICATION SLAVE ON .
For normal OLTP business, our table data changes still have frequent update operations, and ck all do a redundant insert operation. The first point we are concerned about is performance. Will frequent data updates cause ck to consume a lot of resources, and the second point is disk space consumption. Will ck quickly delete historical invalid data in some merge operations in the background?

Article reference:

The highway from MySQL to ClickHouse-MaterializeMySQL engine :
ClickHouse and his friends (9) MySQL real-time replication and implementation
CK official document
Clickhouse for Github

Real-time replication from MySQL to Clickhouse