Following the previous article "Transmission Encryption" of the OceanBase security audit, this article mainly practices data transparent encryption and verifies whether the encryption is effective.
Author: Zhang Qian, Alien No. 2, also serves as the shit shoveler of four cats.
Produced by the Aikeson open source community. Original content may not be used without authorization. Please contact the editor and indicate the source for reprinting.
This article is about 1,200 words and is expected to take 4 minutes to read.
environment
Version: OceanBase 4.1.0.0 Enterprise Edition
Encryption configuration
The detailed encryption steps are skipped, and the MySQL tenant is used this time.
Turn on transparent encryption and create a table space
The administrator user logs in to the cluster's MySQL tenant.
# 开启 internal 方式的透明加密
# tde_method 默认值为 none，表示关闭透明表空间加密
obclient [oceanbase]> ALTER SYSTEM SET tde_method='internal';
Query OK, 0 rows affected (0.022 sec)
obclient [oceanbase]> SHOW PARAMETERS LIKE 'tde_method';
+-------+----------+-------------+----------+------------+-----------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+--------+---------+-------------------+
| zone | svr_type | svr_ip | svr_port | name | data_type | value | info | section | scope | source | edit_level |
+-------+----------+-------------+----------+------------+-----------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+--------+---------+-------------------+
| zone1 | observer | 172.17.0.13 | 2882 | tde_method | NULL | internal | none : transparent encryption is none, none means cannot use tde, internal : transparent encryption is in the form of internal tables, bkmi : transparent encryption is in the form of external bkmi | OBSERVER | TENANT | DEFAULT | DYNAMIC_EFFECTIVE |
+-------+----------+-------------+----------+------------+-----------+----------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+--------+---------+-------------------+
1 row in set (0.017 sec)
# 执行该语句，生成主密钥
obclient [oceanbase]> ALTER INSTANCE ROTATE INNODB MASTER KEY;
Query OK, 0 rows affected (0.028 sec)
# 创建表空间并指定加密算法，其中 'y' 表示默认使用 aes-256 算法
obclient [oceanbase]> CREATE TABLESPACE sectest_ts1 encryption = 'y';
Query OK, 0 rows affected (0.021 sec)Create a new table within an encrypted tablespace
An ordinary user logs in to the MySQL tenant of the database and creates a new table t1.
# 创建表并指定表空间
obclient [sysbenchdb]> CREATE TABLE t1 (id1 int, id2 int) TABLESPACE sectest_ts1;
Query OK, 0 rows affected (0.076 sec)
# 确认表空间内的表是否标记为加密
# encryptionalg 为 aes-256，且 encrypted 为 YES 则表示表加密配置成功
obclient [oceanbase]> SELECT table_name,encryptionalg,encrypted FROM oceanbase.V$OB_ENCRYPTED_TABLES;
+------------+---------------+-----------+
| table_name | encryptionalg | encrypted |
+------------+---------------+-----------+
| t1 | aes-256 | YES |
+------------+---------------+-----------+
1 row in set (0.048 sec)Insert a value into the table and perform a large merge so that the value is placed in SSTable.
# 插入值
obclient [sysbenchdb]> insert into t1 values (147852369,999999991);
Query OK, 1 row affected (0.005 sec)
# 做大合并
ALTER SYSTEM MAJOR FREEZE TENANT=ALL;
# 查看合并进度
SELECT * FROM oceanbase.CDB_OB_ZONE_MAJOR_COMPACTION\GCreate an unencrypted table for comparison
An ordinary user logs in to the MySQL tenant of the database and creates a new table without specifying the encryption space ttttttt2.
Also insert a piece of data and perform a large merge.
obclient [sysbenchdb]> CREATE TABLE ttttttt2 (id1 int, id2 int);
Query OK, 0 rows affected (0.076 sec)
obclient [sysbenchdb]> insert into ttttttt2 values (147852369,999999991);
Query OK, 1 row affected (0.005 sec)
# 做大合并
ALTER SYSTEM MAJOR FREEZE TENANT=ALL;
# 查看合并进度
SELECT * FROM oceanbase.CDB_OB_ZONE_MAJOR_COMPACTION\GEncrypted verification
The verification method is to use the tool ob_admin , whose dumpsst function can display the contents of the block_file file.
Use dumpsst to view the contents of the encrypted table and verify whether it is encrypted.
If you need to know the target data before use macro block id, first find the corresponding data above macro block id.
Find macro block id
First, oceanbase.DBA_OB_TABLE_LOCATIONSfind the two tables according to TABLET_ID, among which the encrypted table t1is TABLET_ID200001, and the unencrypted table ttttttt2is TABLET_ID200002.
obclient [oceanbase]> select * from oceanbase.DBA_OB_TABLE_LOCATIONS where TABLE_NAME='t1';
+---------------+------------+----------+------------+----------------+-------------------+------------+---------------+-----------+-------+-------+-------------+----------+--------+--------------+
| DATABASE_NAME | TABLE_NAME | TABLE_ID | TABLE_TYPE | PARTITION_NAME | SUBPARTITION_NAME | INDEX_NAME | DATA_TABLE_ID | TABLET_ID | LS_ID | ZONE | SVR_IP | SVR_PORT | ROLE | REPLICA_TYPE |
+---------------+------------+----------+------------+----------------+-------------------+------------+---------------+-----------+-------+-------+-------------+----------+--------+--------------+
| sysbenchdb | t1 | 500006 | USER TABLE | NULL | NULL | NULL | NULL | 200001 | 1001 | zone1 | 172.17.0.13 | 2882 | LEADER | FULL |
+---------------+------------+----------+------------+----------------+-------------------+------------+---------------+-----------+-------+-------+-------------+----------+--------+--------------+
1 row in set (0.005 sec)
obclient [oceanbase]> select * from oceanbase.DBA_OB_TABLE_LOCATIONS where TABLE_NAME='ttttttt2';
+---------------+------------+----------+------------+----------------+-------------------+------------+---------------+-----------+-------+-------+-------------+----------+--------+--------------+
| DATABASE_NAME | TABLE_NAME | TABLE_ID | TABLE_TYPE | PARTITION_NAME | SUBPARTITION_NAME | INDEX_NAME | DATA_TABLE_ID | TABLET_ID | LS_ID | ZONE | SVR_IP | SVR_PORT | ROLE | REPLICA_TYPE |
+---------------+------------+----------+------------+----------------+-------------------+------------+---------------+-----------+-------+-------+-------------+----------+--------+--------------+
| sysbenchdb | ttttttt2 | 500007 | USER TABLE | NULL | NULL | NULL | NULL | 200002 | 1001 | zone1 | 172.17.0.13 | 2882 | LEADER | FULL |
+---------------+------------+----------+------------+----------------+-------------------+------------+---------------+-----------+-------+-------+-------------+----------+--------+--------------+
1 row in set (0.005 sec)Hold TABLET_ID, according to the merger time, inGV$OB_TABLET_COMPACTION_HISTORYFound in MACRO_ID_LIST, the recorded in it IDis what we need macro block id.
From the output, we can see that the encrypted table t1corresponds macro block idto 387 and the unencrypted table ttttttt2corresponds macro block idto 718.
obclient [oceanbase]> select * from GV$OB_TABLET_COMPACTION_HISTORY where TABLET_ID=200001 and TYPE='MAJOR_MERGE' order by START_TIME \G
*************************** 1. row ***************************
SVR_IP: 172.17.0.13
SVR_PORT: 2882
TENANT_ID: 1004
LS_ID: 1001
TABLET_ID: 200001
TYPE: MAJOR_MERGE
COMPACTION_SCN: 1685093467526445446
START_TIME: 2023-05-26 17:31:22.478149
FINISH_TIME: 2023-05-26 17:31:22.482045
TASK_ID: YB42AC11000D-0005FC95091493EB-0-0
OCCUPY_SIZE: 432
MACRO_BLOCK_COUNT: 1
MULTIPLEXED_MACRO_BLOCK_COUNT: 0
NEW_MICRO_COUNT_IN_NEW_MACRO: 1
MULTIPLEXED_MICRO_COUNT_IN_NEW_MACRO: 0
TOTAL_ROW_COUNT: 1
INCREMENTAL_ROW_COUNT: 1
COMPRESSION_RATIO: 0.67
NEW_FLUSH_DATA_RATE: 100
PROGRESSIVE_COMPACTION_ROUND: 1
PROGRESSIVE_COMPACTION_NUM: 0
PARALLEL_DEGREE: 1
PARALLEL_INFO: -
PARTICIPANT_TABLE: table_cnt=4,[MAJOR]scn=1;[MINI]start_scn=1,end_scn=1685093478867382402;
MACRO_ID_LIST: 387
COMMENTS: serialize_medium_list:{cnt=1;1685093467526445446}|time_guard=EXECUTE=4.20ms|(0.79)|CREATE_SSTABLE=648us|(0.12)|total=5.32ms;
*************************** 2. row ***************************
SVR_IP: 172.17.0.13
SVR_PORT: 2882
TENANT_ID: 1004
LS_ID: 1001
TABLET_ID: 200001
TYPE: MAJOR_MERGE
COMPACTION_SCN: 1685094492266634220
START_TIME: 2023-05-26 17:48:27.276906
FINISH_TIME: 2023-05-26 17:48:27.282468
TASK_ID: YB42AC11000D-0005FC9509149878-0-0
OCCUPY_SIZE: 432
MACRO_BLOCK_COUNT: 1
MULTIPLEXED_MACRO_BLOCK_COUNT: 0
NEW_MICRO_COUNT_IN_NEW_MACRO: 1
MULTIPLEXED_MICRO_COUNT_IN_NEW_MACRO: 0
TOTAL_ROW_COUNT: 1
INCREMENTAL_ROW_COUNT: 1
COMPRESSION_RATIO: 0.67
NEW_FLUSH_DATA_RATE: 71
PROGRESSIVE_COMPACTION_ROUND: 1
PROGRESSIVE_COMPACTION_NUM: 0
PARALLEL_DEGREE: 1
PARALLEL_INFO: -
PARTICIPANT_TABLE: table_cnt=3,[MAJOR]scn=1685093467526445446;[MINI]start_scn=1685093467530410154,end_scn=1685094504683817069;
MACRO_ID_LIST: 718
COMMENTS: serialize_medium_list:{cnt=1;1685094492266634220}|time_guard=EXECUTE=5.92ms|(0.45)|CREATE_SSTABLE=5.94ms|(0.45)|total=13.10ms;
obclient [oceanbase]> select * from GV$OB_TABLET_COMPACTION_HISTORY where TABLET_ID=200002 and TYPE='MAJOR_MERGE' order by START_TIME \G
*************************** 1. row ***************************
SVR_IP: 172.17.0.13
SVR_PORT: 2882
TENANT_ID: 1004
LS_ID: 1001
TABLET_ID: 200002
TYPE: MAJOR_MERGE
COMPACTION_SCN: 1685094492266634220
START_TIME: 2023-05-26 17:48:27.277801
FINISH_TIME: 2023-05-26 17:48:27.284542
TASK_ID: YB42AC11000D-0005FC9509149879-0-0
OCCUPY_SIZE: 424
MACRO_BLOCK_COUNT: 1
MULTIPLEXED_MACRO_BLOCK_COUNT: 0
NEW_MICRO_COUNT_IN_NEW_MACRO: 1
MULTIPLEXED_MICRO_COUNT_IN_NEW_MACRO: 0
TOTAL_ROW_COUNT: 1
INCREMENTAL_ROW_COUNT: 1
COMPRESSION_RATIO: 0.61
NEW_FLUSH_DATA_RATE: 40
PROGRESSIVE_COMPACTION_ROUND: 1
PROGRESSIVE_COMPACTION_NUM: 0
PARALLEL_DEGREE: 1
PARALLEL_INFO: -
PARTICIPANT_TABLE: table_cnt=4,[MAJOR]scn=1685093467526445446;[MINI]start_scn=1,end_scn=1685094504683817070;
MACRO_ID_LIST: 718
COMMENTS: serialize_medium_list:{cnt=1;1685094492266634220}|time_guard=EXECUTE=10.20ms|(0.86)|total=11.87ms;
Parse block_file file
After installing ob_admin, use dumpsst to parse what you got in the previous step macro block id.
Notice:
- ob_admin dumpsst must be run at the ${path_to_oceanbase}/oceanbase level because reads
etc/observer.config.binuse relative paths.- After current testing, it must be specified
--macro-id, otherwise an error will be reported (the error content needs to be viewed in ob_admin.log).
Several parameters used this time are introduced as follows:
-fSpecify the data directory.-dMacroblock type, currently only macro_block is supported.-aThat ismacro-id, fill in the values obtained in the above steps.-tSpecifytablet_id, further precise scope.-iThat ismicro block id, -1 means all micro blocks.
Parse the t1 table, which is the encrypted table
You can see that the output tablet_idis 200001 row_countand 1, which corresponds to the piece of data we inserted.
The content of this line of data is not displayed, and the verification data is successfully encrypted.
[admin@ob_4 oceanbase]$ ob_admin dumpsst -f /home/admin/oceanbase/store/obdemo/ -d macro_block -a 387 -t 200001 -i -1
succ to open, filename=ob_admin.log, fd=3, wf_fd=2
old log_file need close, old = ob_admin.log new = ob_admin.log
succ to open, filename=ob_admin.log, fd=3, wf_fd=2
succ to open, filename=ob_admin_rs.log, fd=4, wf_fd=2
------------------------------{Common Header}------------------------------
| header_size|24
| version|1
| magic|1001
| attr|1
| payload_size|952
| payload_checksum|-1027413104
--------------------------------------------------------------------------------
------------------------------{SSTable Macro Block Header}------------------------------
| header_size|208
| version|1
| magic|1007
| tablet_id|200001
| logical_version|1685093467526445446
| data_seq|0
| column_count|5
| rowkey_column_count|3
| row_store_type|1
| row_count|1
| occupy_size|432
| micro_block_count|1
| micro_block_data_offset|232
| data_checksum|2617981320
| compressor_type|6
| master_key_id|500004
--------------------------------------------------------------------------------
--------{column_index column_type column_order column_checksum collation_type}----------
| [0 ObUInt64Type ASC 3344869974 63]
| [1 ObIntType ASC 313654433 63]
| [2 ObIntType ASC 2388842353 63]
| [3 ObInt32Type ASC 2776795072 63]
| [4 ObInt32Type ASC 82537422 63]
--------------------------------------------------------------------------------Parse the ttttttt2 table, which is the unencrypted table
Replace " tablet_idand " in the command macro block idwith ttttttt2"table" idfor parsing.
Compared with the encrypted table t1, the output information of the unencrypted table is richer, and the specific data content can be seen.
Here is a simplified display, you can see that the previously inserted data [{"INT":147852369}][{"INT":999999991}] is displayed in Total Rows.
[admin@ob_4 oceanbase]$ ob_admin dumpsst -f /home/admin/oceanbase/store/obdemo/ -d macro_block -a 718 -t 200002 -i -1
succ to open, filename=ob_admin.log, fd=3, wf_fd=2
old log_file need close, old = ob_admin.log new = ob_admin.log
succ to open, filename=ob_admin.log, fd=3, wf_fd=2
succ to open, filename=ob_admin_rs.log, fd=4, wf_fd=2
------------------------------{Common Header}------------------------------
| header_size|24
| version|1
| magic|1001
| attr|1
| payload_size|892
| payload_checksum|-1696352947
--------------------------------------------------------------------------------
------------------------------{SSTable Macro Block Header}------------------------------
| header_size|208
| version|1
| magic|1007
| tablet_id|200002
| logical_version|1685094492266634220
| data_seq|0
| column_count|5
| rowkey_column_count|3
| row_store_type|1
| row_count|1
| occupy_size|424
| micro_block_count|1
| micro_block_data_offset|232
| data_checksum|725485397
| compressor_type|6
| master_key_id|0
--------------------------------------------------------------------------------
……
------------------------------{Total Rows[1]}------------------------------
|ROW[0]:trans_id=[{txid:0}],dml_flag=[N|INSERT],mvcc_flag=[]|[{"BIGINT UNSIGNED":1}][{"BIGINT":-1685094482154160502}][{"BIGINT":0}][{"INT":147852369}][{"INT":999999991}]
……
------------------------------{Encoding Column Header[4]}------------------------------
| type|0
| attribute|0
| is fix length|0
| has extend value|0
| is bit packing|0
| is last var field|0
| extend value index|65542
| store object type|0
| offset|0
| length|0
--------------------------------------------------------------------------------
------------------------------{Index Micro Block[0]}------------------------------
------------------------------{Total Rows[1]}------------------------------
|ROW[0]:trans_id=[{txid:0}],dml_flag=[N|INSERT],mvcc_flag=[]|[{"BIGINT UNSIGNED":1}][{"BIGINT":-1685094482154160502}][{"BIGINT":0}][{"VARCHAR":"
", collation:"binary", coercibility:"NUMERIC"}]
|Index Block Row Header|[{version:1, row_store_type:1, compressor_type:6, is_data_index:1, is_data_block:1, is_leaf_block:0, is_major_node:1, is_pre_aggregated:0, is_deleted:0, contain_uncommitted_row:0, is_macro_node:0, has_string_out_row:0, all_lob_in_row:1, macro_id:[-1](ver=0,mode=0,seq=0), block_offset:232, block_size:192, master_key_id:0, encrypt_id:0, encrypt_key:"data_size:16, data:00000000000000000000000000000000", row_count:1, schema_version:1685094464567160, macro_block_count:0, micro_block_count:1}]
------------------------------{Macro Meta Micro Block}------------------------------
------------------------------{Encoding Micro Header}------------------------------
| header_size|96
| version|2
| magic|1005
| column_count|4
| rowkey_column_count|3
| row_count|1
| row_store_type|2
| row_index_byte|0
| var_column_count|0
| row_data_offset|357
|column_chksum[ 0]|3344869974
|column_chksum[ 1]|1868627082
|column_chksum[ 2]|2388842353
|column_chksum[ 3]|1583982749
--------------------------------------------------------------------------------
……summary
This article mainly uses the dumpsst function of the ob_admin tool to parse block_file and verify the OceanBase data transparent encryption function.
When you encounter problems when using dumpsst, it is recommended to pay more attention to them ob_admin.log, which will be more helpful for troubleshooting.
For more technical articles, please visit: https://opensource.actionsky.com/
About SQLE
SQLE from the Axon open source community is a SQL audit tool for database users and managers that supports multi-scenario audits, standardized online processes, native support for MySQL audits and scalable database types.
SQLE get
| type | address |
|---|---|
| Repository | https://github.com/actiontech/sqle |
| document | https://actiontech.github.io/sqle-docs/ |
| release news | https://github.com/actiontech/sqle/releases |
| Data audit plug-in development documentation | https://actiontech.github.io/sqle-docs/docs/dev-manual/plugins/howtouse |