1. Scene Description
After greenplum cluster deployed, software Pharaoh encounter some problems in practical use, simple record, hope to help a friend in need.
2. Solution
2.1 gpcc monitoring address
Description: Very important, greenplum cluster monitoring, database cluster can monitor usage, including cpu, memory, hard disk, performance and so on.
http://192.168.85.158:28080
gpmon
gpmon
2.2 typical to create the table
Software Description Pharaoh:
(1) WITH (appendonly = true, compresslevel = 5) shows that only the new data back, the compression ratio is 5 (1-10);
(2) DISTRIBUTED BY (laowang), key distribution, the distribution server according to each segement this field;
(3) PARTITION, by the time partition.
CREATE TABLE "public"."t_ruanjianlaowang" (
"laowang" text,
"timestamp" text,
)
WITH (appendonly=true, compresslevel=5)
DISTRIBUTED BY (laowang)
PARTITION BY RANGE(timestamp)
(
PARTITION t_ruanjianlaowang_20170701 START ('2017-06-01 00:00:00'::timestamp without time zone) END ('2017-07-01 00:00:00'::timestamp without time zone),
PARTITION t_ruanjianlaowang_20170801 START ('2017-07-01 00:00:00'::timestamp without time zone) END ('2017-08-01 00:00:00'::timestamp without time zone)
)
2.3 Create the test table, whether the test page data duplication
Description: Create a test table, test data is repeated Page
--创建测试表
create table test_ischongfu2 as select * from t_ruanjianlawoang limit 1 offset 0;
--插入数据
insert into test_ischongfu2 select * from t_ruanjianlawoang limit 1000 offset 0;
insert into test_ischongfu2 select * from t_ruanjianlawoang limit 1000 offset 1001;
insert into test_ischongfu2 select * from t_ruanjianlawoang limit 1000 offset 2001;
insert into test_ischongfu2 select * from t_ruanjianlawoang limit 1000 offset 3001;
insert into test_ischongfu2 select * from t_ruanjianlawoang limit 1000 offset 4001;
2.4 Create an external table
Software Description Pharaoh:
Create an external table, super fast speed, 100,000 data, a few seconds to complete.
CREATE EXTERNAL TABLE t_ex_ruanjianlaowang (
laowang1 double precision,
laowang2 double precision,
laowang3 varchar(100)
) location ('gpfdist://192.168.85.158:8081/d.csv') format 'csv' (DELIMITER ',');
2.5 Delete external table
Description: After the spark import data, generate a lot of garbage external table data, you can also delete the next batch.
drop external table spark_3f59d3ff03a55cd3_3d9d854163f8f07a_driver_73 ;
drop external table spark_3f59d3ff03a55cd3_3d9d854163f8f07a_driver_72 ;
drop external table spark_3f59d3ff03a55cd3_3d9d854163f8f07a_driver_60
2.6 normal construction of the table
Description:
No partition, level 5 compression, distribution key for the laowang1.
CREATE TABLE t_ruanjianlaowang(
laowang1 double precision,
laowang2 double precision
)
WITH (appendonly=true,orientation=column,compresstype=zlib,COMPRESSLEVEL=5)
distributed by (laowang1);
2.7 greenplum create a resource queue
Description: When a cluster share, for greenplum cluster resource allocation.
----新增插入role
create role gpinsertrole with login password 'gpinsertrole';
--权限控制感觉有问题，直接在navcat上更改为超级管理员可能会快点
GRANT ALL PRIVILEGES ON DATABASE "db_ruanjianlawoang" TO gpinsertrole;
create resource queue q_pginsert with (ACTIVE_STATEMENTS=10,MEMORY_LIMIT='2000MB',PRIORITY=HIGH,COST_OVERCOMMIT=true,MIN_COST=100,MAX_COST=10000);
alter role gpinsertrole resource queue q_pginsert;
2.8 Index on whether to go
After insertion of large quantities of data, sometimes not taking the index, we need to operate under.
Greenplum中的vacuum和analyze
vacuum t_ruanjianlaowang;
I'm "Software Pharaoh," I felt that if I may, to focus the next chant, subsequent updates seconds to know! Welcome forum, No. namesake public message exchange!