Flow data lake platform Apache Paimon (4) integrates Hive engine

Chapter 3 Integrating the Hive Engine

When integrating with Flink, by using the paimon Hive Catalog, you can create, delete, query and insert into the paimon table from Flink. These operations directly affect the corresponding Hive metastore. Tables created this way can also be accessed directly from Hive.

To further integrate with Hive, you can use Hive SQL to create and query Paimon tables.

3.1 Environment preparation

Paimon currently supports Hive 3.1, 2.3, 2.2, 2.1, and 2.1-cdh-6.3. Supports MR and Tez execution engines for Hive Read, and MR execution engines for Hive Write (beeline also does not support hive write).

Create the auxlib folder in the Hive root directory, and copy paimon-hive-connector-0.5-SNAPSHOT.jar to auxlib (add jar is not recommended, and the MR engine will report an exception when running the join statement):

Download address: https://repository.apache.org/snapshots/org/apache/paimon/paimon-hive-connector-3.1/0.5-SNAPSHOT/

mkdir /opt/module/hive/auxlib

cp paimon-hive-connector-3.1-0.5-20230703.002437-65.jar /opt/module/hive/auxlib

3.2 Access the existing Paimon table

USE test;

SHOW TABLES;

 

SELECT * FROM ws_t;

 

INSERT INTO test_table VALUES (9,9,9);

3.3 Create Paimon table

SET hive.metastore.warehouse.dir=hdfs://hadoop102:8020/paimon/hive;

 

CREATE TABLE test_h(

  a INT COMMENT 'The a field',

  b STRING COMMENT 'The b field'

)

STORED BY 'org.apache.paimon.hive.PaimonStorageHandler'

3.4 Access Paimon table through external table

To access existing paimon tables, you can also register them as external tables in Hive, without specifying any column or table properties, just the path.

CREATE EXTERNAL TABLE test.hive_ex

STORED BY 'org.apache.paimon.hive.PaimonStorageHandler'

LOCATION 'hdfs://hadoop102:8020/paimon/hive/test.db/ws_t';

– or write the path in a table property:

CREATE EXTERNAL TABLE hive_ex

STORED BY 'org.apache.paimon.hive.PaimonStorageHandler'

TBLPROPERTIES (

 'paimon_location' ='hdfs://hadoop102:8020/paimon/hive/test.db/ws_t'

);

Operate external tables:

SELECT * FROM hive_ex;

 

INSERT INTO hive_ex VALUES (8,8,8);