ClickHouse and his friends (1) compile, develop, test

Original source: https://bohutang.me/2020/06/05/clickhouse-and-friends-development/

By chance, I had an offline communication with the ClickHouse team. Alexey mentioned the design philosophy of ClickHouse:

The product must solve actual problem
And do it better than others

A model for solving business problems with engineering thinking!

For users, what they care about is not the high-tech that is in the sky or the sky. They just need a solution that can solve their own problems. This is very rare in the open source community and grows by strength "barbaric".

So, I was full of curiosity about this weapon that exudes the flavor of vodka, and I participated in the ClickHouse community to find out. The first feeling is that it is open, friendly, and powerful (AK47 vs CK16, ClickHouse 2016 open source).

This article starts with compilation and testing, and then how to contribute Patch to the community. I hope it will be helpful to those students who want to participate in the CK community.

How to compile and test ClickHouse locally?

Source code acquisition

`1`	`git clone --recursive https://github.com/ClickHouse/ClickHouse`

Preparation for compilation

sudo apt install build-essential
sudo apt-get install software-properties-common
sudo apt-add-repository ppa:ubuntu-toolchain-r/test
sudo apt-get update

sudo apt-get install gcc-9 g++-9 git python ninja-build
sudo snap install cmake

Start compiling

cd ClickHouse
mkdir build
cd build
export CC=gcc-9
export CXX=g++-9
cmake ..
ninja

testing method

ClickHouse tests are detailed in the official development/tests (https://github.com/ClickHouse/ClickHouse/blob/master/docs/en/development/tests.md) document. Here are 3 commonly used test modes :

1. Functional Tests

Function test, mainly used for ClickHouse internal function test, method: input a sql file, output a result, similar to mtr in MySQL, test set (https://github.com/ClickHouse/ClickHouse/tree/master/tests/queries )

1
2

cd tests
./clickhouse-test -c "../build/programs/clickhouse-client" 00001_select_1

2. Integration Tests

Integration testing, mainly used for testing involving third-party services, such as MySQL/Postgres/MongoDB, etc., orchestrating and scheduling (pytest) operation in a containerized manner, testing collection

Since there are many modules involved, it is difficult to build an integrated test environment. It is recommended to use the official docker image. For example, to run the integration test set under test_mysql_protocol:

1
2
3

cd tests/integration
docker pull yandex/clickhouse-integration-tests-runner
./runner --binary /your/ClickHouse/build/programs/clickhouse  --bridge-binary /your/ClickHouse/build/programs/clickhouse-odbc-bridge --configs-dir /your/ClickHouse/programs/server/ 'test_mysql_protocol/test.py::test_java_client -ss -vv'

3. Unit Tests

Unit testing is mainly used for code module testing. The test set is in the tests directory of each module, such as: Core/tests(https://github.com/ClickHouse/ClickHouse/tree/master/src/Core/tests)

If you want to understand how a module works, it is strongly recommended to look through the tests directory of the module. For example, if you want to understand the working mechanism of the processor, track and debug Processors/tests/(https://github.com/ClickHouse/ClickHouse/ tree/master/src/Core/tests).

How to provide Patch to ClickHouse community?

1. fork

First fork a ClickHouse code on your github, such as https://github.com/BohuTANG/ClickHouse

2. Clone to local

1
2

git clone --recursive https://github.com/BohuTANG/ClickHouse
git checkout -B mysql_replica(branch名字)

3. Create a new branch

`1`	`git checkout -B mysql_replica(branch名字)`

4. Function development

Developers can submit a Draft Pull Request to the official, github will show that the Pull Request is in Draft state, and the official cannot merge

5. can be testd tags

Wait for Upstream to hit the [can be tested] label. Once the CI madmen are marked, they will run strongly. It takes about tens of hours to run a round.
Assist developers to find some errors in code style, compilation and testing, so that developers can iterate and correct in their own branches.

If you just modify typo, this tag Upstream is usually not added.

6. Completed

The development is completed, the test is OK, the Draft is promoted to a formal Pull Request, and the Upstraem Review is awaited.

7. Merge to Master

If Upstream passes, your code will be merged to Master, congratulations on becoming a ClickHouse contributor

8. Matters needing attention

ClickHouse Upstream iteration is very fast. Be sure to pay more attention to the progress of the master branch and try to keep your branch code synchronized with the master. Otherwise, if Upstream Docker is updated, your own test may fail.

It is recommended to read doc/development(https://github.com/ClickHouse/ClickHouse/tree/master/docs/en/development).

The full text is over.

Enjoy ClickHouse :)

Teacher Ye's "MySQL Core Optimization" class has been upgraded to MySQL 8.0, scan the code to start the journey of MySQL 8.0 practice