Are there any operation and maintenance SQL statements in ClickHouse that can "settle your life". I think many of my friends will be interested in this issue, so I will share it here.
In ClickHouse default system database (databse), there are many system tables. Our various information about ClickHouse's operating status mainly comes from these system tables.
Next, I will list some common operation and maintenance SQL statements.
-
Current connections
As we all know, the native interfaces exposed by CH are divided into two types: TCP and HTTP. The current number of connections between TCP, HTTP and internal copies can be queried through system.metrics.
ch7.nauu.com :) SELECT * FROM system.metrics WHERE metric LIKE '%Connection';SELECT *FROM system.metricsWHERE metric LIKE '%Connection'┌─metric────────────────┬─value─┬─description─────────────────────────────────────────────────────────┐│ TCPConnection │ 2 │ Number of connections to TCP server (clients with native interface) ││ HTTPConnection │ 1 │ Number of connections to HTTP server ││ InterserverConnection │ 0 │ Number of connections from other replicas to fetch parts │└───────────────────────┴───────┴─────────────────────────────────────────────────────────────────────┘
-
The query currently being executed
The query currently being executed can be queried through system.processes, for example:
ch7.nauu.com :) SELECT query_id, user, address, query FROM system.processes ORDER BY query_id;SELECTquery_id,user,address,queryFROM system.processesORDER BY query_id ASC┌─query_id─────────────────────────────┬─user────┬─address────────────┬─query─────────────────────────────────────────────────────────────────────────────┐│ 203f1d0e-944e-472d-8d8f-bae548ff9899 │ default │ ::ffff:10.37.129.4 │ SELECT query_id, user, address, query FROM system.processes ORDER BY query_id ASC ││ fb7fba85-b2a0-4271-87ff-22da97ae511b │ default │ ::ffff:10.37.129.4 │ INSERT INTO hits_v1 FORMAT TSV │└──────────────────────────────────────┴─────────┴────────────────────┴───────────────────────────────────────────────────────────────────────────────────┘
As you can see, CH is currently executing two statements, of which the second is that the INSERT query is writing data.
-
Terminate query
Through the KILL QUERY statement, you can terminate the query being executed:
KILL QUERY WHERE query_id = 'query_id'For example, to terminate the INSERT query just now:
ch7.nauu.com :) KILL QUERY WHERE query_id='ff695827-dbf5-45ad-9858-a853946ea140';KILL QUERY WHERE query_id = 'ff695827-dbf5-45ad-9858-a853946ea140' ASYNCOk.0 rows in set. Elapsed: 0.024 sec.
As we all know, in addition to the regular SELECT and INSERT, there is also a type of operation called Mutation in ClickHouse, that is, ALTER DELETE and ALTER UPDATE .
For mutation operations, ClickHouse provides system.mutations for query, for example:
ch7.nauu.com :) SELECT database, table, mutation_id, command, create_time, is_done FROM system.mutations;SELECTdatabase,table,mutation_id,command,create_time,is_doneFROM system.mutations┌─database─┬─table──────┬─mutation_id────┬─command──────────────────┬─────────create_time─┬─is_done─┐│ default │ testcol_v9 │ mutation_2.txt │ DELETE WHERE ID = 'A003' │ 2020-06-29 01:15:04 │ 1 │└──────────┴────────────┴────────────────┴──────────────────────────┴─────────────────────┴─────────┘1 rows in set. Elapsed: 0.002 sec.
Similarly, you can use KILL MUTATION to terminate the Mutation operation being performed:
KILL MUTATION WHERE mutation_id = 'mutation_id';
-
Storage space statistics
Query the space of each storage path of CH:
ch5.nauu.com :) SELECT name,path,formatReadableSize(free_space) AS free,formatReadableSize(total_space) AS total,formatReadableSize(keep_free_space) AS reserved FROM system.disksSELECTname,path,formatReadableSize(free_space) AS free,formatReadableSize(total_space) AS total,formatReadableSize(keep_free_space) AS reservedFROM system.disks┌─name──────┬─path──────────────┬─free──────┬─total─────┬─reserved─┐│ default │ /chbase/data/ │ 36.35 GiB │ 49.09 GiB │ 0.00 B ││ disk_cold │ /chbase/cloddata/ │ 35.35 GiB │ 48.09 GiB │ 1.00 GiB ││ disk_hot1 │ /chbase/data/ │ 36.35 GiB │ 49.09 GiB │ 0.00 B ││ disk_hot2 │ /chbase/hotdata1/ │ 36.35 GiB │ 49.09 GiB │ 0.00 B │└───────────┴───────────────────┴───────────┴───────────┴──────────┘4 rows in set. Elapsed: 0.001 sec.
-
Statistics of the space occupied by each database
ch7.nauu.com :) SELECT database, formatReadableSize(sum(bytes_on_disk)) on_disk FROM system.parts GROUP BY database;SELECTdatabase,formatReadableSize(sum(bytes_on_disk)) AS on_diskFROM system.partsGROUP BY database┌─database─┬─on_disk──┐│ system │ 1.59 MiB ││ default │ 3.60 GiB │└──────────┴──────────┘
-
Statistics of the space occupied by each column field
The compression size and compression ratio of each column field and the proportion of the data size of each row of the column
SELECTdatabase,table,column,any(type),sum(column_data_compressed_bytes) AS compressed,sum(column_data_uncompressed_bytes) AS uncompressed,round(uncompressed / compressed, 2) AS ratio,compressed / sum(rows) AS bpr,sum(rows)FROM system.parts_columnsWHERE active AND database != 'system'GROUP BYdatabase,table,columnORDER BYdatabase ASC,table ASC,column ASC┌─database─┬─table────────┬─column─────────────────────┬─any(type)──────────────────────────────┬─compressed─┬─uncompressed─┬──ratio─┬───────────────────bpr─┬─sum(rows)─┐│ default │ hits_v1 │ AdvEngineID │ UInt8 │ 351534 │ 26621706 │ 75.73 │ 0.013204788603705563 │ 26621706 ││ default │ hits_v1 │ Age │ UInt8 │ 7543552 │ 26621706 │ 3.53 │ 0.2833609536518809 │ 26621706 ││ default │ hits_v1 │ BrowserCountry │ FixedString(2) │ 6549379 │ 53243412 │ 8.13 │ 0.24601650247358303 │ 26621706 ││ default │ hits_v1 │ BrowserLanguage │ FixedString(2) │ 2819085 │ 53243412 │ 18.89 │ 0.10589422781545255 │ 26621706 ││ default │ hits_v1 │ CLID │ UInt32 │ 2311006 │ 106486824 │ 46.08 │ 0.08680908729140048 │ 26621706 ││ default │ hits_v1 │ ClientEventTime │ DateTime │ 98518704 │ 106486824 │ 1.08 │ 3.7006908573026838 │ 26621706 ││ default │ hits_v1 │ ClientIP │ UInt32 │ 25120766 │ 106486824 │ 4.24 │ 0.9436196913901761 │ 26621706 ││ default │ hits_v1 │ ClientIP6 │ FixedString(16) │ 25088558 │ 425947296 │ 16.98 │ 0.9424098515699934 │ 26621706 ││ default │ hits_v1 │ ClientTimeZone │ Int16 │ 8487148 │ 53243412 │ 6.27 │ 0.3188055641512982 │ 26621706 ││ default │ hits_v1 │ CodeVersion │ UInt32 │ 11976952 │ 106486824 │ 8.89 │ 0.4498942329240658 │ 26621706 ││ default │ hits_v1 │ ConnectTiming │ Int32 │ 27937373 │ 106486824 │ 3.81 │ 1.0494208372671534 │ 26621706 ││ default │ hits_v1 │ CookieEnable │ UInt8 │ 202718 │ 26621706 │ 131.32 │ 0.007614763681936838 │ 26621706 ││ default │ hits_v1 │ CounterClass │ Int8 │ 425492 │ 26621706 │ 62.57 │ 0.015982897564866805 │ 26621706 │...
-
Slow query
SELECTuser,client_hostname AS host,client_name AS client,formatDateTime(query_start_time, '%T') AS started,query_duration_ms / 1000 AS sec,round(memory_usage / 1048576) AS MEM_MB,result_rows AS RES_CNT,result_bytes / 1048576 AS RES_MB,read_rows AS R_CNT,round(read_bytes / 1048576) AS R_MB,written_rows AS W_CNT,round(written_bytes / 1048576) AS W_MB,queryFROM system.query_logWHERE type = 2ORDER BY query_duration_ms DESCLIMIT 10┌─user────┬─host─────────┬─client────────────┬─started──┬────sec─┬─MEM_MB─┬──RES_CNT─┬────────────────RES_MB─┬────R_CNT─┬─R_MB─┬───W_CNT─┬─W_MB─┬─query───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐│ default │ ch7.nauu.com │ ClickHouse client │ 01:05:03 │ 51.434 │ 1031 │ 8873898 │ 8706.51146697998 │ 0 │ 0 │ 8873898 │ 8707 │ INSERT INTO hits_v1 FORMAT TSV ││ default │ ch7.nauu.com │ ClickHouse client │ 01:01:48 │ 43.511 │ 1031 │ 8873898 │ 8706.51146697998 │ 0 │ 0 │ 8873898 │ 8707 │ INSERT INTO hits_v1 FORMAT TSV ││ default │ ch7.nauu.com │ ClickHouse client │ 17:12:04 │ 11.12 │ 1801 │ 18874398 │ 446.8216323852539 │ 6291466 │ 351 │ 0 │ 0 │ SELECT id, arrayJoin(arrayConcat(groupArray(a), groupArray(b), groupArray(c))) AS v FROM test_y GROUP BY id ORDER BY v ASC ││ default │ ch7.nauu.com │ ClickHouse client │ 17:13:28 │ 3.992 │ 1549 │ 18874398 │ 446.8216323852539 │ 6291466 │ 351 │ 0 │ 0 │ SELECT id, arrayJoin(arrayConcat(groupArray(a), groupArray(b), groupArray(c))) AS v FROM test_y GROUP BY id ││ default │ ch7.nauu.com │ ClickHouse client │ 17:13:12 │ 3.976 │ 1549 │ 18874398 │ 446.8216323852539 │ 6291466 │ 351 │ 0 │ 0 │ SELECT id, arrayJoin(arrayConcat(groupArray(a), groupArray(b), groupArray(c))) AS v FROM test_y GROUP BY id ││ default │ ch7.nauu.com │ ClickHouse client │ 01:25:39 │ 3.962 │ 1549 │ 18874398 │ 446.8216323852539 │ 6291466 │ 351 │ 0 │ 0 │ SELECT id, arrayJoin(arrayConcat(groupArray(a), groupArray(b), groupArray(c))) AS v FROM test_y GROUP BY id ││ default │ ch7.nauu.com │ ClickHouse client │ 04:32:29 │ 3.114 │ 1542 │ 10000000 │ 219.82192993164062 │ 10500000 │ 231 │ 0 │ 0 │ SELECT user_id, argMax(score, create_time) AS score, argMax(deleted, create_time) AS deleted, max(create_time) AS ctime FROM test_a GROUP BY user_id HAVING deleted = 0 ││ default │ ch7.nauu.com │ ClickHouse client │ 02:59:56 │ 3.03 │ 1544 │ 10000000 │ 219.75380992889404 │ 10500000 │ 231 │ 0 │ 0 │ SELECT user_id, argMax(score, create_time) AS score, argMax(is_update, create_time) AS is_update, max(create_time) AS ctime FROM test_a GROUP BY user_id ││ default │ ch7.nauu.com │ ClickHouse client │ 02:54:01 │ 3.019 │ 1543 │ 10000000 │ 219.3450927734375 │ 10500000 │ 230 │ 0 │ 0 │ SELECT user_id, argMax(score, create_time) AS score, argMax(delete, create_time) AS delete, max(create_time) AS ctime FROM test_a GROUP BY user_id ││ default │ │ │ 03:03:12 │ 2.857 │ 1543 │ 10 │ 0.0002269744873046875 │ 10500000 │ 231 │ 0 │ 0 │ SELECT * FROM view_test_a limit 10 │└─────────┴──────────────┴───────────────────┴──────────┴────────┴────────┴──────────┴───────────────────────┴──────────┴──────┴─────────┴──────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘10 rows in set. Elapsed: 0.017 sec. Processed 1.44 thousand rows, 200.81 KB (83.78 thousand rows/s., 11.68 MB/s.)
-
Copy warning monitoring
Use the following SQL statement to perform early warning monitoring on the copy, and the variables of each early warning can be adjusted according to your own situation.
SELECT database, table, is_leader, total_replicas, active_replicasFROM system.replicasWHERE is_readonlyOR is_session_expiredOR future_parts > 30OR parts_to_check > 20OR queue_size > 30OR inserts_in_queue > 20OR log_max_index - log_pointer > 20OR total_replicas < 2OR active_replicas < total_replicas┌─database─┬─table───────────────────────┬─is_leader─┬─total_replicas─┬─active_replicas─┐│ default │ replicated_sales_12 │ 0 │ 0 │ 0 ││ default │ test_fetch │ 0 │ 0 │ 0 ││ default │ test_sharding_simple2_local │ 0 │ 0 │ 0 │└──────────┴─────────────────────────────┴───────────┴────────────────┴─────────────────┘
Okay, that's all for today's sharing. For CH's daily operation and maintenance SQL is far more than that, here is just a piece of cake.
If this article is helpful to you, you are welcome to subscribe, forward, and read it. Three combos :)