ClickHouse notes: installation, configuration and user management under Ubuntu/Centos

ClickHouse

ClickHouse is an OLAP database

OLTP and OLAP

• OLTP (On-Line Transaction Processing), focusing on transaction processing, data recording performance and security
• OLAP (On-Line Analytical Processing), focusing on data analysis, focusing on query performance

Generally, OLTP database is used for business data storage, and OLAP database is used for query analysis.

ClickHouse performance

• The writing performance is very high, basically reaching the bottleneck of disk reading and writing
• Suitable for wide table query, when JOIN query, the associated table needs to be controlled within tens of millions
• In distributed scenarios, capacity needs to be planned in advance, and the operation and maintenance costs for scenarios with continuous expansion requirements are relatively high
• Support full-text search (inverted index, by n-gram or token), for specific discussion, please refer to this discussion , currently still in experimental
• Only supports limited transactions, guarantees the atomicity of the INSERT process, including writing and reading
• Does not support Windows. Although it can run on Win10 through WSL, Docker, etc., but this way is only "running", the performance has been greatly reduced, and it has no practical value.

In distributed scenarios that require complex queries, you can consider Apache Doris.

Install

hardware requirements

• Hard disk installation requires 2.5G space
• The memory is not less than 4G, 16G or more is recommended, the bigger the better
• SSD + RAID, file format Ext4, XFS
• For cluster deployment, it is recommended to use 10G (10 Gigabit) network

Ubuntu installation

sudo apt-get install -y apt-transport-https ca-certificates dirmngr
# 在 /tmp 下创建临时目录
GNUPGHOME=$(mktemp -d)
echo $GNUPGHOME
# 生成 clickhouse-keyring.gpg
sudo GNUPGHOME="$GNUPGHOME" gpg --no-default-keyring --keyring /usr/share/keyrings/clickhouse-keyring.gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 8919F6BD2B48D754
sudo rm -r "$GNUPGHOME"
sudo chmod +r /usr/share/keyrings/clickhouse-keyring.gpg
# 创建 ck 的 apt list
echo "deb [signed-by=/usr/share/keyrings/clickhouse-keyring.gpg] https://packages.clickhouse.com/deb stable main" | sudo tee /etc/apt/sources.list.d/clickhouse.list
# 更新软件包
sudo apt-get update
# 安装
sudo apt install -y clickhouse-server clickhouse-client

CentOS7 installation

sudo yum-config-manager --add-repo https://packages.clickhouse.com/rpm/clickhouse.repo
sudo yum install -y clickhouse-server clickhouse-client

CK file structure

• Configuration file /etc/clickhouse-server, config.xml global configuration, users.xml user configuration
• Storage directory /var/lib/clickhouse
  • Path definition: /etc/clickhouse-server/config.xml <path>/var/lib/clickhouse/</path>, <tmp_path>/var/lib/clickhouse/tmp/</tmp_path>,<user_files_path>/var/lib/clickhouse/user_files/</user_files_path>
• Log path /var/log/clickhouse-server/, clickhouse-server.err.log clickhouse-server.log

service management

start service

sudo systemctl start clickhouse-server
sudo systemctl status clickhouse-server

Service IP and port

By default, it only listens to local requests, open the service port, and edit /etc/clickhouse-server/config.xml

sudo chmod 600 /etc/clickhouse-server/config.xml
sudo vi /etc/clickhouse-server/config.xml

Uncomment, serve both IPv6 and IPv4

<listen_host>::</listen_host>

If you only need to provide IPv4, you can uncomment this line

<listen_host>0.0.0.0</listen_host>

These two lines cannot be uncommented at the same time, and an error will be reported when starting

User Management

ClickHouse users are divided into two types

• Users directly configured in /etc/clickhouse-server/user.xml, such as default
• User created in SQL

The login method for both users is the same

Configure User Password

Open /etc/clickhouse-server/user.xml to see instructions on setting user passwords. The default password for user default is empty

simply put

• <password>qwerty</password>The plaintext password is set directly with
• SHA256 password for<password_sha256_hex>65e84be33532fb784c48129675f9eff3a682b27168c0ea744b2cf58ee02337c5</password_sha256_hex>
• Double SHA1 password for<password_double_sha1_hex>e395796d6546b1b65db9d665cd43f0e858dd4303</password_double_sha1_hex>
• For LDAP authentication<ldap><server>my_ldap_server</server></ldap>
• For Kerberos authentication<kerberos><realm>EXAMPLE.COM</realm></kerberos>

Corresponding password generation command

# SHA256
PASSWORD=$(base64 < /dev/urandom | head -c8); echo "$PASSWORD"; echo -n "$PASSWORD" | sha256sum | tr -d '-'
cY19OvVH                                                           <-- 口令
e17cd697e0845d75d2068ae1e1479d3fd10d76e5afa89724fbc6fe27554526e4   <-- SHA256结果

# Double SHA1
PASSWORD=$(base64 < /dev/urandom | head -c8); echo "$PASSWORD"; echo -n "$PASSWORD" | sha1sum | tr -d '-' | xxd -r -p | sha1sum | tr -d '-'
1gQO8XpM                                       <-- 口令
e9fdf3480016dfae8ad0170e846edd031180a3f4       <-- Double SHA1结果

If there is no xxd command under Centos7, it needs to be installed by the following command

sudo yum install vim-common

If necessary to increase the number of digits

PASSWORD=$(base64 < /dev/urandom | head -c16); echo "$PASSWORD"; echo -n "$PASSWORD" | sha256sum | tr -d '-'
vlIlWHFqY0BbSy2f
e08ebd515246f1b5f3bfdb24b967a797b7218289b263ed0fbb3ff47fcc121f1b

If you need to customize

PASSWORD=asdf1234; echo "$PASSWORD"; echo -n "$PASSWORD" | sha1sum | tr -d '-' | xxd -r -p | sha1sum | tr -d '-'
asdf1234
dda7b4594264195da8bb303516d7ec5509b7b942 

Add users through xml under user.d

For example, to add a new user dbowner with administrative authority, you can create a file dbowner.xml under /etc/clickhouse-server/user.d, the content is

<clickhouse>
    <users>
        <dbowner>
            <password>abcd1234</password>
            <networks>
                <ip>::/0</ip>
            </networks>

            <profile>default</profile>

            <quota>default</quota>

            <access_management>1</access_management>
        </dbowner>
    </users>
</clickhouse>

The above configuration

• User name is dbowner
• The password is abcd1234 in clear text
• Network Sources: All IPv6, IPv4 addresses
• Allow use of administrative privileges (pass access_management = 1)

When clickhouse starts, it will merge the configuration file under user.d with user.xml, and overwrite the repeated part in user.xml. Pay attention to the permissions of this file, you need to set the owner of this file to clickhouse, otherwise clickhouse will read after startup fetch will fail.

sudo chown clickhouse:clickhouse dbowner.xml

Add users through SQL and configure permissions

For specific user permission configuration, refer to ClickHouse official instructions

https://clickhouse.com/docs/en/operations/access-rights

The dbowner user added above only enables access_management. By default, both SQL-driven access control and account management are turned off. If you want to enable the complete SQL user mode, the configuration to be added <dbowner>below

<access_management>1</access_management>
<named_collection_control>1</named_collection_control>
<show_named_collections>1</show_named_collections>
<show_named_collections_secrets>1</show_named_collections_secrets>

On the machine where ClickHouse is installed, use the dbowner created above to log in to the client, create a new user dbroot, and grant full permissions

$ clickhouse-client --user dbowner --password abcd1234

:) CREATE USER dbroot IDENTIFIED BY 'root1234';
CREATE USER dbroot IDENTIFIED WITH sha256_hash BY '21AC41BC256B35A32EC2021D359AE5F297AD7ED2F8ED8F7A2A1A7B9F1F94F898' SALT '56DC39142C0AEB19BC2C61AACBD9F27DD040E25489CC29B76D07D65F6D2A3AA1'
Query id: d0099db9-b292-4905-84bd-a91da19f8edc
Ok.
0 rows in set. Elapsed: 0.005 sec. 

The user was created successfully, but an error was reported when granting permissions

:) GRANT ALL ON *.* TO dbroot WITH GRANT OPTION;
GRANT ALL ON *.* TO dbroot WITH GRANT OPTION
Query id: 92a50da8-d847-4f4f-a74c-95b9f1207a67
0 rows in set. Elapsed: 0.007 sec. 

Received exception from server (version 23.4.2):
Code: 497. DB::Exception: Received from localhost:9000. DB::Exception: dbowner: Not enough privileges. To execute this query it's necessary to have grant SHOW, SELECT, INSERT, ALTER, CREATE, DROP, UNDROP TABLE, TRUNCATE, OPTIMIZE, BACKUP, KILL QUERY, KILL TRANSACTION, MOVE PARTITION BETWEEN SHARDS, ACCESS MANAGEMENT, NAMED COLLECTION CONTROL, SYSTEM, dictGet, INTROSPECTION, SOURCES ON *.* WITH GRANT OPTION. (ACCESS_DENIED)

Add the following permissions to dbowner

<named_collection_control>1</named_collection_control>
<show_named_collections>1</show_named_collections>
<show_named_collections_secrets>1</show_named_collections_secrets>

Restart ClickHouse and execute again, the authorization will be successful

:) GRANT ALL ON *.* TO dbroot WITH GRANT OPTION;
GRANT ALL ON *.* TO dbroot WITH GRANT OPTION
Query id: f4eaa3ce-8182-4717-9270-ce2e95eb2b88
Ok.
0 rows in set. Elapsed: 0.004 sec. 

At this time, you can log in to ClickHouse with dbroot / root1234

connect

Use clickhouse-client

clickhouse-client --user [user] --password [password]

Use Tabix

Use Firefox to access http://dash.tabix.io, Chrome seems to fail, and will report a CORS error

Fill in the server address, the default is http://[server_ip]:8123, user default, password is empty

Use DBeaver

Server address, default is http://[server_ip]:8123, user default, password is empty

Common Management Commands

create database

CREATE DATABASE my_db;

create table

CREATE TABLE my_db.my_table (id UInt64, column1 String) ENGINE = MergeTree() ORDER BY id;

create user

CREATE USER my_user IDENTIFIED BY 'password';

Granted permission

# my_db下所有表的 ALTER 权限
GRANT ALTER ON my_db.* WITH GRANT OPTION;

# my_db下my_table表的 ALTER 权限
GRANT ALTER ON my_db.my_table TO my_user;

# 多个权限
GRANT SELECT, ALTER COLUMN ON my_db.my_table TO my_user WITH GRANT OPTION;

permission tree

├── ALTER (only for table and view)/
│   ├── ALTER TABLE/
│   │   ├── ALTER UPDATE
│   │   ├── ALTER DELETE
│   │   ├── ALTER COLUMN/
│   │   │   ├── ALTER ADD COLUMN
│   │   │   ├── ALTER DROP COLUMN
│   │   │   ├── ALTER MODIFY COLUMN
│   │   │   ├── ALTER COMMENT COLUMN
│   │   │   ├── ALTER CLEAR COLUMN
│   │   │   └── ALTER RENAME COLUMN
│   │   ├── ALTER INDEX/
│   │   │   ├── ALTER ORDER BY
│   │   │   ├── ALTER SAMPLE BY
│   │   │   ├── ALTER ADD INDEX
│   │   │   ├── ALTER DROP INDEX
│   │   │   ├── ALTER MATERIALIZE INDEX
│   │   │   └── ALTER CLEAR INDEX
│   │   ├── ALTER CONSTRAINT/
│   │   │   ├── ALTER ADD CONSTRAINT
│   │   │   └── ALTER DROP CONSTRAINT
│   │   ├── ALTER TTL/
│   │   │   └── ALTER MATERIALIZE TTL
│   │   ├── ALTER SETTINGS
│   │   ├── ALTER MOVE PARTITION
│   │   ├── ALTER FETCH PARTITION
│   │   └── ALTER FREEZE PARTITION
│   └── ALTER LIVE VIEW/
│       ├── ALTER LIVE VIEW REFRESH
│       └── ALTER LIVE VIEW MODIFY QUERY
├── ALTER DATABASE
├── ALTER USER
├── ALTER ROLE
├── ALTER QUOTA
├── ALTER [ROW] POLICY
└── ALTER [SETTINGS] PROFILE

View permissions

SHOW GRANTS FOR  my_user;

You can see the difference in permissions between the default with access_management enabled and the dbroot with full permissions

:) show grants for dbroot;
┌─GRANTS FOR dbroot────────────────────────────┐
│ GRANT ALL ON *.* TO dbroot WITH GRANT OPTION │
└──────────────────────────────────────────────┘

:) show grants for default;
┌─GRANTS FOR default─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ GRANT SHOW, SELECT, INSERT, ALTER, CREATE, DROP, UNDROP TABLE, TRUNCATE, OPTIMIZE, BACKUP, KILL QUERY, KILL TRANSACTION, MOVE PARTITION BETWEEN SHARDS, SYSTEM, dictGet, INTROSPECTION, SOURCES, CLUSTER ON *.* TO default │
└────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

cancel permission

REVOKE ALTER COLUMN ON my_db.my_table FROM my_user;

Related Links

• https://jishuin.proginn.com/p/763bfbd59c4b
• https://zhuanlan.zhihu.com/p/421469439
• https://www.tinybird.co/blog-posts/text-search-at-scale-with-clickhouse
• https://clickhouse.com/blog/clickhouse-search-with-inverted-indices