Postgresql统计信息分析（Oracle统计信息迁移到PG）

本篇主要分析PG的统计信息。

ORACLE统计信息迁移到PG（不接触用户数据），应用场景比较特殊，不再赘述。

准备

drop table mapping;
create table mapping(id0 int, id1 int, id2 int, info varchar(32));
insert into mapping values(1, 23, 413, 'b');
insert into mapping values(2, 23, 325, 'a');
insert into mapping values(3, 23, 652, 'c');
insert into mapping values(4, 17, 584, 'b');
insert into mapping values(5, 43, 325, 'd');
insert into mapping values(6, 39, 149, 'e');
insert into mapping values(7, 93, 999, 'ffff');
insert into mapping values(8, 24, 999, 'gggg');
insert into mapping values(9, 24, NULL, 'hhh');
insert into mapping values(0, 19, NULL, 'zzz');

-- 便于直观看到统计信息，先查一下pg_stats这个视图

postgres=# select * from pg_stats where tablename='mapping';
-[ RECORD 1 ]----------+----------------------------
schemaname             | public
tablename              | mapping
attname                | id0
inherited              | f
null_frac              | 0
avg_width              | 4
n_distinct             | -1
most_common_vals       |
most_common_freqs      |
histogram_bounds       | {0,1,2,3,4,5,6,7,8,9}
correlation            | 0.454545
most_common_elems      |
most_common_elem_freqs |
elem_count_histogram   |
-[ RECORD 2 ]----------+----------------------------
schemaname             | public
tablename              | mapping
attname                | id1
inherited              | f
null_frac              | 0
avg_width              | 4
n_distinct             | -0.7
most_common_vals       | {23,24}
most_common_freqs      | {0.3,0.2}
histogram_bounds       | {17,19,39,43,93}
correlation            | 0.260606
most_common_elems      |
most_common_elem_freqs |
elem_count_histogram   |
-[ RECORD 3 ]----------+----------------------------
schemaname             | public
tablename              | mapping
attname                | id2
inherited              | f
null_frac              | 0.2
avg_width              | 4
n_distinct             | -0.6
most_common_vals       | {325,999}
most_common_freqs      | {0.2,0.2}
histogram_bounds       | {149,413,584,652}
correlation            | 0.428571
most_common_elems      |
most_common_elem_freqs |
elem_count_histogram   |
-[ RECORD 4 ]----------+----------------------------
schemaname             | public
tablename              | mapping
attname                | info
inherited              | f
null_frac              | 0
avg_width              | 3
n_distinct             | -0.9
most_common_vals       | {b}
most_common_freqs      | {0.2}
histogram_bounds       | {a,c,d,e,ffff,gggg,hhh,zzz}
correlation            | 0.975758
most_common_elems      |
most_common_elem_freqs |
elem_count_histogram   |

ORACLE统计信息

方式一：stat_table

drop table st_t1_all;
exec dbms_stats.gather_table_stats('SCOTT','MAPPING',cascade=>true,degree => 8,method_opt => 'for all columns');
exec dbms_stats.create_stat_table (ownname => 'SCOTT', stattab => 'ST_T1_ALL', tblspace => 'users');
exec dbms_stats.export_table_stats(ownname =>'SCOTT',tabname=>'MAPPING',stattab=>'ST_T1_ALL',statid => 'A');
select * from ST_T1_ALL;

含义：

STATTAB	T	I	C	C’histogram
STATID
TYPE
VERSION
FLAGS
C1	TABLE_NAME	INDEX_NAME	TABLE_NAME	TABLE_NAME
C2	PARTITION_NAME	PARTITION_NAME	PARTITION_NAME	PARTITION_NAME
C3	SUBPARTITION_NAME	SUBPARTITION_NAME	SUBPARTITION_NAME	SUBPARTITION_NAME
C4			COLUMN_NAME	COLUMN_NAME
C5	OWNER	OWNER	OWNER	OWNER
N1	NUM_ROWS	NUM_ROWS	NUM_DISTINCT
N2	BLOCKS	LEAF_BLOCKS	DENSITY
N3	AVG_ROW_LEN	DISTINCT_KEYS
N4	SAMPLE_SIZE	LEAF_BLOCKS_PER_KEY	SAMPLE_SIZE
N5		DATA_BLOCKS_PER_KEY	NUM_NULLS
N6		CLUSTERING_FACTOR	LO_VALUE
N7		BLEVEL	HI_VALUE
N8		SAMPLE_SIZE	AVG_COL_LEN
N9
N10				ENDPOINT_NUMBER
N11				ENDPOINT_VALUE
N12
D1
R1
R2
CH1
CL1

方式二：视图

select * from all_tab_columns where table_name='MAPPING';
select * from all_tab_histograms where table_name='MAPPING';
select * from ALL_TAB_STATISTICS where table_name='MAPPING';

POSTGRESQL统计信息

统计信息通过SQL命令vacuum和analyze（人工触发或autovacuum触发）写入pg_class和pg_statistic中

pg_class

表级统计信息

• relpages：占了多少磁盘页
• reltuples：行数估计值

postgres=# \d pg_class
      Table "pg_catalog.pg_class"
     Column     |   Type    | Modifiers
----------------+-----------+-----------
 ...
 relpages       | integer   | not null  -- 表的页数统计
 reltuples      | real      | not null  -- 表的行数统计
 ...

pg_statistic

列级统计信息，除了前面的几个基本信息，其他5组SLOT数据分别关注不同侧面

每一组由四列组成：[stakindN, staopN, stanumbersN, stavaluesN]

每一组的位置不固定，需要遍历SLOT找到可用的值

• stakindN：表示该组数据的关注点
  • 1 - MCV
  • 2 - histogram直方图
  • 3 - correlation相关性
• staopN：统计值的操作符
• stanumbersN
• stavaluesN

postgres=# \d pg_statistic
             Table "pg_catalog.pg_statistic"
   Column    |   Type   | Collation | Nullable | Default
-------------+----------+-----------+----------+---------
 starelid    | oid      |           | not null |  -- 关联pg_class.oid，表名
 staattnum   | smallint |           | not null |  -- 关联pg_attribute.attnum，列名
 stainherit  | boolean  |           | not null |  -- 是否为继承表的统计信息
 stanullfrac | real     |           | not null |  -- 该列空值比例
 stawidth    | integer  |           | not null |  -- 该列平均长度
 stadistinct | real     |           | not null |  -- 该列唯一值个数
 stakind1    | smallint |           | not null |  -- 下面都是统计信息SLOT
 stakind2    | smallint |           | not null |  
 stakind3    | smallint |           | not null |
 stakind4    | smallint |           | not null |
 stakind5    | smallint |           | not null |
 staop1      | oid      |           | not null |
 staop2      | oid      |           | not null |
 staop3      | oid      |           | not null |
 staop4      | oid      |           | not null |
 staop5      | oid      |           | not null |
 stanumbers1 | real[]   |           |          |
 stanumbers2 | real[]   |           |          |
 stanumbers3 | real[]   |           |          |
 stanumbers4 | real[]   |           |          |
 stanumbers5 | real[]   |           |          |
 stavalues1  | anyarray |           |          |
 stavalues2  | anyarray |           |          |
 stavalues3  | anyarray |           |          |
 stavalues4  | anyarray |           |          |
 stavalues5  | anyarray |           |          |

下面分别对每一列做说明

stanullfrac(float4)

该列空值比例

（ORACLE记录的是空值个数需要转换一下）

stawidth(int4)

非 NULL 值平均长度（字节）

• 定长类型的平均长度即为数据类型的长度

• 变长类型的平均长度为存储宽度的平均值

  （比如text类型的字符串$stawidth= \cfrac {\sum (stringlength+1)}{rows} $）

（ORACLE记录的数据宽度需要转换）

stadistinct(float4)

该列唯一值个数

• -1表示唯一值个数等于行数
• 小于1表示唯一值占比
• 大于等于1表示实际的唯一值估计个数
• 0表示未知

（ORACLE似乎总是记录唯一值的个数，这里不需要转换可以直接导入）

stakind\staopN\stanumbersN\stavaluesN

列名	类型	填值方法
stakindN	int2	直接填值（1到5）
staopN	oid	使用::regoperator填值
stanumbersN	float4[]	使用::real[]填值
stavaluesN	anyarray	使用array_in(…)::anyarray填值

四个一组共同表达某一个统计信息，下面分开讲。（其中stakindN表明该组的含义）

注意：这一组统计信息可以随意放在五个槽位的任意位置，例如上面字段的stakind5、staop5、stanumbers5、stavalues5就表示第五槽位。

STATISTIC_KIND_MCV

stakindN=1时，该组表示MCV（most common values），使用=操作符确定值是不是相等。

只有出现超过一次的值会被记录下来，注意：unique约束的列是没有这个slot的。

ps. 并不是所有值都会记录进来，如果表的所有值都作为采样则全部统计，否则选出现频率超过平均值25%的值

• staopN:数据比较实用的操作符，这里可以使用regoperator将字符串转换成操作符的OID
• stavaluesN：变长数组，记录值
• stanumbersN：变长数组，记录上面值的出现概率，注意这个是按频率从高到低排序的！

（ORACLE没有MCV的统计）

例如ID1列：

stakind1表示该组统计信息为MVC，staop1说明数据比较实用的操作符是等号。

stavalues1数组里面就记录了出现次数超过两次的数据23和24。

stanumbers1数据分别记录了23和24的出现频率0.3和0.2。

id0 | id1 | id2 | info
-----+-----+-----+------
   1 |  23 | 413 | b
   2 |  23 | 325 | a
   3 |  23 | 652 | c
   4 |  17 | 584 | b
   5 |  43 | 325 | d
   6 |  39 | 149 | e
   7 |  93 | 999 | ffff
   8 |  24 | 999 | gggg
   9 |  24 |     | hhh
   0 |  19 |     | zzz
(10 rows)


stakind1 = 1
staop1 = 'pg_catalog.=(pg_catalog.int4, pg_catalog.int4)'::regoperator
stanumbers1 = '{0.3,0.2}'::real[], 
stavalues1 = array_in('{23,24}', 'pg_catalog.int4'::regtype, -1)::anyarray, 

STATISTIC_KIND_HISTOGRAM

stakindN=2时，代表这个SLOT描述的是列数据直方图信息，直方图的构造必须将数据排序分段，排序使用的操作符在staop中指定。

桶数量控制：

-- 针对单表
alter table tenk1 alter unique1 set statistics 100;

-- 针对全库修改参数（默认100）
default_statistics_target (integer)

• stavaluesN包含M（>=2）个非空值，它将非空列数据值划分为数据个数大致相等的几个区间。第一个stavalues最小，最后一个stavalues最大，如果同时MCV有值，这里需要忽略MCV的值，这样可以更精确的表示一些非MCV值的分布性。两种情况直方图为空：如果列的值都在MCV中表示了、数据类型没有<操作符
• stanumbersN为空
• staopN为对应数据类型的操作符

（ORACLE中字符型数据的直方图是加密过的值，需要还原成明文的直方图）

例如ID1列：MCV中出现的23和24会排除出去，剩下的17、43、39、93、19来做直方图

 id0 | id1 | id2 | info
-----+-----+-----+------
   1 |  23 | 413 | b
   2 |  23 | 325 | a
   3 |  23 | 652 | c
   4 |  17 | 584 | b
   5 |  43 | 325 | d
   6 |  39 | 149 | e
   7 |  93 | 999 | ffff
   8 |  24 | 999 | gggg
   9 |  24 |     | hhh
   0 |  19 |     | zzz
(10 rows)


stakind2 = 2
staop2 = 'pg_catalog.<(pg_catalog.int4, pg_catalog.int4)'::regoperator
stanumbers2 = NULL::real[], 
stavalues2 = array_in('{17,19,39,43,93}', 'pg_catalog.int4'::regtype, -1)::anyarray, 

STATISTIC_KIND_CORRELATION

stakindN=3时，这个SLOT表示该列物理行序和逻辑行序的相关性，范围从-1到+1。

• stanumbersN数据中的第一个元素保存相关性的数值。绝对值为1时，表示物理顺序和逻辑顺序是相关的，正1代表正相关（即物理顺序和逻辑顺序一样），负1代表负相关。这个值接近一的时候执行器会认为这个使用使用索引扫描的代价会更小，因为这时候对磁盘的随机访问会少，random_cost会更低
• stavaluesN为空
• staopN为对应数据类型的操作符

（PG是堆表迁移后的物理顺序未知，这项的意义不大）

STATISTIC_KIND_MCELEM

stakindN=4时，该SLOT中有两个信息，在*<@, &&, and @>*时会用到：

• stavaluesN表示该列为多值类型时，多值元素的高频词
• stanumbers1表示该列为多值类型时，多值元素的高频词的出现频率

（这里计算使用Lossy Counting算法，论文链接）

STATISTIC_KIND_DECHIST

stakindN=5时，该SLOT代表多值元素的柱状图中，每个区间的非空唯一元素个数

STATISTIC_KIND_RANGE_LENGTH_HISTOGRAM

stakindN=6

range-type column类型的列的distribution of range lengths

STATISTIC_KIND_BOUNDS_HISTOGRAM

stakindN=7

range-type column类型列的直方图

SQL构造方法

导入pg_class（按需）

-- relpages用法比较特殊，规划器会自己再取一个估计值和这个值对比
-- 如果不相等规划器会对reltuples基数进行放缩
update pg_catalog.pg_class c
set reltuples=?, relpages=?
where to_schema_qualified_relation(oid)='public.mapping';

导入pg_statistic（必须）

模版

WITH upsert as (    
  UPDATE pg_catalog.pg_statistic SET column_name = expression [, ...]    
  WHERE to_schema_qualified_relation(starelid) = t_relname    
    AND to_attname(t_relname, staattnum) = t_attname    
    AND to_atttype(t_relname, staattnum) = t_atttype    
    AND stainherit = t_stainherit    
  RETURNING *)    
ins as (    
  SELECT expression [, ...]    
  WHERE NOT EXISTS (SELECT * FROM upsert)    
    AND to_attnum(t_relname, t_attname) IS NOT NULL    
    AND to_atttype(t_relname, t_attname) = t_atttype)    
INSERT INTO pg_catalog.pg_statistic SELECT * FROM ins;    

以测试表为例，针对表的每一列构造一条SQL

ID0列

WITH upsert AS
  (UPDATE pg_catalog.pg_statistic
   SET stanullfrac = 0,
       stawidth = 4,
       stadistinct = -1,
       stakind1 = 2,
       stakind2 = 3,
       stakind3 = 0,
       stakind4 = 0,
       stakind5 = 0,
       staop1 = 'pg_catalog.<(pg_catalog.int4, pg_catalog.int4)'::regoperator,
       staop2 = 'pg_catalog.<(pg_catalog.int4, pg_catalog.int4)'::regoperator,
       staop3 = '0'::regoperator,
       staop4 = '0'::regoperator,
       staop5 = '0'::regoperator,
       stanumbers1 = NULL::real[], 
       stanumbers2 = '{0.454545}'::real[], 
       stanumbers3 = NULL::real[], 
       stanumbers4 = NULL::real[], 
       stanumbers5 = NULL::real[], 
       stavalues1 = array_in('{0,1,2,3,4,5,6,7,8,9}', 'pg_catalog.int4'::regtype, -1)::anyarray, 
       stavalues2 = NULL::anyarray, 
       stavalues3 = NULL::anyarray, 
       stavalues4 = NULL::anyarray, 
       stavalues5 = NULL::anyarray 
       WHERE to_schema_qualified_relation(starelid) = 'public.mapping' 
         AND to_attname('public.mapping', staattnum) = 'id0' 
         AND to_atttype('public.mapping', staattnum) = 'pg_catalog.int4' 
         AND stainherit = false RETURNING *), 
  ins AS 
  ( SELECT 'public.mapping'::regclass, 
  	    to_attnum('public.mapping', 'id0'), 
  	    'false'::boolean, 
  	    0::real, 
  	    4::integer, 
  	    -1::real, 
  	    2, 
  	    3, 
  	    0, 
  	    0, 
  	    0, 
  	    'pg_catalog.<(pg_catalog.int4, pg_catalog.int4)'::regoperator, 
  	    'pg_catalog.<(pg_catalog.int4, pg_catalog.int4)'::regoperator, 
  	    '0'::regoperator, 
  	    '0'::regoperator, 
  	    '0'::regoperator, 
  	    NULL::real[], 
  	    '{0.454545}'::real[], 
  	    NULL::real[], 
  	    NULL::real[], 
  	    NULL::real[],
        array_in('{0,1,2,3,4,5,6,7,8,9}', 
        'pg_catalog.int4'::regtype, -1)::anyarray,
        NULL::anyarray,
        NULL::anyarray,
        NULL::anyarray,
        NULL::anyarray
    WHERE NOT EXISTS (SELECT * FROM upsert)
      AND to_attnum('public.mapping', 'id0') IS NOT NULL
      AND to_atttype('public.mapping', 'id0') = 'pg_catalog.int4')
INSERT INTO pg_catalog.pg_statistic SELECT * FROM ins;

ID1列

WITH upsert AS
  (UPDATE pg_catalog.pg_statistic
   SET stanullfrac = 0,
       stawidth = 4,
       stadistinct = -0.7, 
       stakind1 = 1,
       stakind2 = 2,
       stakind3 = 3,
       stakind4 = 0,
       stakind5 = 0,
       staop1 = 'pg_catalog.=(pg_catalog.int4, pg_catalog.int4)'::regoperator,
       staop2 = 'pg_catalog.<(pg_catalog.int4, pg_catalog.int4)'::regoperator,
       staop3 = 'pg_catalog.<(pg_catalog.int4, pg_catalog.int4)'::regoperator,
       staop4 = '0'::regoperator,
       staop5 = '0'::regoperator,
       stanumbers1 = '{0.3,0.2}'::real[], 
       stanumbers2 = NULL::real[], 
       stanumbers3 = '{0.260606}'::real[], 
       stanumbers4 = NULL::real[], 
       stanumbers5 = NULL::real[], 
       stavalues1 = array_in('{23,24}', 'pg_catalog.int4'::regtype, -1)::anyarray, 
       stavalues2 = array_in('{17,19,39,43,93}', 'pg_catalog.int4'::regtype, -1)::anyarray, 
       stavalues3 = NULL::anyarray, 
       stavalues4 = NULL::anyarray, 
       stavalues5 = NULL::anyarray 
       WHERE to_schema_qualified_relation(starelid) = 'public.mapping' 
         AND to_attname('public.mapping', staattnum) = 'id1' 
         AND to_atttype('public.mapping', staattnum) = 'pg_catalog.int4' 
         AND stainherit = false RETURNING *), 
  ins AS 
  ( SELECT 'public.mapping'::regclass, 
  	    to_attnum('public.mapping', 'id1'), 
  	    'false'::boolean, 
  	    0::real, 
  	    4::integer, 
  	    -0.7::real, 
  	    1, 
  	    2, 
  	    3, 
  	    0, 
  	    0, 
  	    'pg_catalog.=(pg_catalog.int4, pg_catalog.int4)'::regoperator, 
  	    'pg_catalog.<(pg_catalog.int4, pg_catalog.int4)'::regoperator, 
  	    'pg_catalog.<(pg_catalog.int4, pg_catalog.int4)'::regoperator, 
  	    '0'::regoperator, 
  	    '0'::regoperator, 
  	    '{0.3,0.2}'::real[], 
  	    NULL::real[], 
  	    '{0.260606}'::real[], 
  	    NULL::real[], 
  	    NULL::real[],
        array_in('{23,24}', 'pg_catalog.int4'::regtype, -1)::anyarray,
        array_in('{17,19,39,43,93}', 'pg_catalog.int4'::regtype, -1)::anyarray,
        NULL::anyarray,
        NULL::anyarray,
        NULL::anyarray
    WHERE NOT EXISTS (SELECT * FROM upsert)
      AND to_attnum('public.mapping', 'id1') IS NOT NULL
      AND to_atttype('public.mapping', 'id1') = 'pg_catalog.int4')
INSERT INTO pg_catalog.pg_statistic SELECT * FROM ins;

ID2列

WITH upsert AS
  (UPDATE pg_catalog.pg_statistic
   SET stanullfrac = 0.2, 
       stawidth = 4,
       stadistinct = -0.6, 
       stakind1 = 1,
       stakind2 = 2,
       stakind3 = 3,
       stakind4 = 0,
       stakind5 = 0,
       staop1 = 'pg_catalog.=(pg_catalog.int4, pg_catalog.int4)'::regoperator,
       staop2 = 'pg_catalog.<(pg_catalog.int4, pg_catalog.int4)'::regoperator,
       staop3 = 'pg_catalog.<(pg_catalog.int4, pg_catalog.int4)'::regoperator,
       staop4 = '0'::regoperator,
       staop5 = '0'::regoperator,
       stanumbers1 = '{0.2,0.2}'::real[], 
       stanumbers2 = NULL::real[], 
       stanumbers3 = '{0.428571}'::real[], 
       stanumbers4 = NULL::real[], 
       stanumbers5 = NULL::real[], 
       stavalues1 = array_in('{325,999}', 'pg_catalog.int4'::regtype, -1)::anyarray, 
       stavalues2 = array_in('{149,413,584,652}', 'pg_catalog.int4'::regtype, -1)::anyarray, 
       stavalues3 = NULL::anyarray, 
       tavalues4 = NULL::anyarray, 
       stavalues5 = NULL::anyarray 
       WHERE to_schema_qualified_relation(starelid) = 'public.mapping' 
         AND to_attname('public.mapping', staattnum) = 'id2' 
         AND to_atttype('public.mapping', staattnum) = 'pg_catalog.int4' 
         AND stainherit = false RETURNING *), 
  ins AS 
  ( SELECT 'public.mapping'::regclass, 
  	    to_attnum('public.mapping', 'id2'), 
  	    'false'::boolean, 
  	    0.2::real, 
  	    4::integer, 
  	    -0.6::real, 
  	    1, 
  	    2, 
  	    3, 
  	    0, 
  	    0, 
  	    'pg_catalog.=(pg_catalog.int4, pg_catalog.int4)'::regoperator, 
  	    'pg_catalog.<(pg_catalog.int4, pg_catalog.int4)'::regoperator, 
  	    'pg_catalog.<(pg_catalog.int4, pg_catalog.int4)'::regoperator, 
  	    '0'::regoperator, 
  	    '0'::regoperator, 
  	    '{0.2,0.2}'::real[], 
  	    NULL::real[], 
  	    '{0.428571}'::real[], 
  	    NULL::real[], 
  	    NULL::real[],
        array_in('{325,999}', 'pg_catalog.int4'::regtype, -1)::anyarray,
        array_in('{149,413,584,652}', 'pg_catalog.int4'::regtype, -1)::anyarray,
        NULL::anyarray,
        NULL::anyarray,
        NULL::anyarray
    WHERE NOT EXISTS (SELECT * FROM upsert)
      AND to_attnum('public.mapping', 'id2') IS NOT NULL
      AND to_atttype('public.mapping', 'id2') = 'pg_catalog.int4')
INSERT INTO pg_catalog.pg_statistic SELECT * FROM ins;

INFO列

WITH upsert AS
  (UPDATE pg_catalog.pg_statistic
   SET stanullfrac = 0,
       stawidth = 3,
       stadistinct = -0.9, 
       stakind1 = 1,
       stakind2 = 2,
       stakind3 = 3,
       stakind4 = 0,
       stakind5 = 0,
       staop1 = 'pg_catalog.=(pg_catalog.text, pg_catalog.text)'::regoperator,
       staop2 = 'pg_catalog.<(pg_catalog.text, pg_catalog.text)'::regoperator,
       staop3 = 'pg_catalog.<(pg_catalog.text, pg_catalog.text)'::regoperator,
       staop4 = '0'::regoperator,
       staop5 = '0'::regoperator,
       stanumbers1 = '{0.2}'::real[], 
       stanumbers2 = NULL::real[], 
       stanumbers3 = '{0.975758}'::real[], 
       stanumbers4 = NULL::real[], 
       stanumbers5 = NULL::real[], 
       stavalues1 = array_in('{b}', 'pg_catalog."varchar"'::regtype, -1)::anyarray, 
       stavalues2 = array_in('{a,c,d,e,ffff,gggg,hhh,zzz}', 'pg_catalog."varchar"'::regtype, -1)::anyarray, 
       stavalues3 = NULL::anyarray, 
       stavalues4 = NULL::anyarray, 
       stavalues5 = NULL::anyarray
       WHERE to_schema_qualified_relation(starelid) = 'public.mapping' 
         AND to_attname('public.mapping', staattnum) = 'info' 
         AND to_atttype('public.mapping', staattnum) = 'pg_catalog."varchar"' 
         AND stainherit = false RETURNING *), 
  ins AS 
  ( SELECT 'public.mapping'::regclass, 
  	    to_attnum('public.mapping', 'info'), 
  	    'false'::boolean, 
  	    0::real, 
  	    3::integer, 
  	    -0.9::real, 
  	    1, 
  	    2, 
  	    3, 
  	    0, 
  	    0, 
  	    'pg_catalog.=(pg_catalog.text, pg_catalog.text)'::regoperator, 
  	    'pg_catalog.<(pg_catalog.text, pg_catalog.text)'::regoperator, 
  	    'pg_catalog.<(pg_catalog.text, pg_catalog.text)'::regoperator, 
  	    '0'::regoperator, 
  	    '0'::regoperator, 
  	    '{0.2}'::real[], 
  	    NULL::real[], 
  	    '{0.975758}'::real[], 
  	    NULL::real[], 
  	    NULL::real[],
        array_in('{b}', 'pg_catalog."varchar"'::regtype, -1)::anyarray,
        array_in('{a,c,d,e,ffff,gggg,hhh,zzz}', 'pg_catalog."varchar"'::regtype, -1)::anyarray,
        NULL::anyarray,
        NULL::anyarray,
        NULL::anyarray
    WHERE NOT EXISTS (SELECT * FROM upsert)
      AND to_attnum('public.mapping', 'info') IS NOT NULL
      AND to_atttype('public.mapping', 'info') = 'pg_catalog."varchar"')
INSERT INTO pg_catalog.pg_statistic SELECT * FROM ins;

PG统计信息导入方案

方案一（old）

1. DDL迁移
2. 数据不迁移
3. 统计信息迁移

具体导入流程

1、PG关闭autovacuum，避免统计信息自动更新

alter system set autovacuum = off;
select pg_reload_conf();

show autovacuum;
 autovacuum
------------
 off
(1 row)

2、迁移ORACLE的表和索引，在PG中插入一行fake数据然后删除，这一步是保证pg_class.relpages不为0，否责PG不会使用系统表中记录的统计信息。注意这步完成后需要analyze，pg_statistic才会自动构造列的信息tuple。

这个过程在pg_class系统表中增加一行记录。

postgres=# select * from pg_class where relname='mapping';
-[ RECORD 1 ]-------+--------
relname             | mapping
relnamespace        | 2200
...
relpages            | 0     -- 当前表的页数统计
reltuples           | 0     -- 当前表的行数统计
...

pg_statistic中保存列信息，但是只建表的话PG不会在pg_statistic更新统计信息，需要插入至少一行数据，使relpages不等于0，PG才会开始构造这个表的统计信息。

create table mapping(id0 int, id1 int, id2 int, info varchar(32));
analyze mapping ;

select * from pg_statistic where starelid='mapping'::regclass;
(0 rows)

insert into mapping values(1, 23, 413, 'b');
analyze mapping ;

select * from pg_statistic where starelid='mapping'::regclass;
...
...
...
...
(4 rows)

3、ORACLE导出统计信息并构造符合PG统计信息格式的SQL，更新pg_class和pg_statistic的对应列。

4、PG中执行explain user_sql;得到pg中的执行计划和cost，根据cost可以评估该SQL的执行成本。

验证

使用PG REGRESS用例中的tenk1表作测试（10000行）

初始化

postgres库：真实数据导入

CREATE TABLE tenk1 (
        unique1         int4,
        unique2         int4,
        two                     int4,
        four            int4,
        ten                     int4,
        twenty          int4,
        hundred         int4,
        thousand        int4,
        twothousand     int4,
        fivethous       int4,
        tenthous        int4,
        odd                     int4,
        even            int4,
        stringu1        name,
        stringu2        name,
        string4         name
) WITH OIDS;
CREATE INDEX tenk1_unique1 ON tenk1 USING btree(unique1 int4_ops);
CREATE INDEX tenk1_unique2 ON tenk1 USING btree(unique2 int4_ops);
CREATE INDEX tenk1_hundred ON tenk1 USING btree(hundred int4_ops);
CREATE INDEX tenk1_thous_tenthous ON tenk1 (thousand, tenthous);

COPY tenk1 FROM '/home/mingjie.gmj/projects/postgresql-10.1/src/test/regress/data/tenk.data';
analyze tenk1;

SELECT relpages, reltuples FROM pg_class WHERE relname = 'tenk1';
 relpages | reltuples
----------+-----------
      358 |     10000
(1 row)

faketank1库：无数据，执行DDL后倒入统计信息

CREATE TABLE tenk1 (
        unique1         int4,
        unique2         int4,
        two                     int4,
        four            int4,
        ten                     int4,
        twenty          int4,
        hundred         int4,
        thousand        int4,
        twothousand     int4,
        fivethous       int4,
        tenthous        int4,
        odd                     int4,
        even            int4,
        stringu1        name,
        stringu2        name,
        string4         name
) WITH OIDS;
CREATE INDEX tenk1_unique1 ON tenk1 USING btree(unique1 int4_ops);
CREATE INDEX tenk1_unique2 ON tenk1 USING btree(unique2 int4_ops);
CREATE INDEX tenk1_hundred ON tenk1 USING btree(hundred int4_ops);
CREATE INDEX tenk1_thous_tenthous ON tenk1 (thousand, tenthous);

explain select * from tenk1;
                        QUERY PLAN
----------------------------------------------------------
 Seq Scan on tenk1  (cost=0.00..13.00 rows=300 width=244)
(1 row)

-- 现在表中无数据，直接写入页数和行数
update pg_catalog.pg_class c
set reltuples=10000, relpages=358
where to_schema_qualified_relation(oid)='public.tenk1';

explain select * from tenk1;
                      QUERY PLAN
-------------------------------------------------------
 Seq Scan on tenk1  (cost=0.00..0.00 rows=1 width=244)
(1 row)

-- 插入fake数据后analyze
insert into tenk1 values(1,2,3,4,5,6,7,8,9,1,2,3,4,'a','b','c');

analyze tenk1;

select relpages,reltuples 
from pg_class where to_schema_qualified_relation(oid)='public.tenk1';
 relpages | reltuples
----------+-----------
        1 |         1

explain select * from tenk1;
                      QUERY PLAN
-------------------------------------------------------
 Seq Scan on tenk1  (cost=0.00..1.01 rows=1 width=244)
(1 row)

update pg_catalog.pg_class c
set reltuples=10000, relpages=1 
where to_schema_qualified_relation(oid)='public.tenk1';


select relpages,reltuples from pg_class where to_schema_qualified_relation(oid)='public.tenk1';
 relpages | reltuples
----------+-----------
        1 |     10000
(1 row)

全表扫描–>ok

目标计划

explain select * from tenk1;
                         QUERY PLAN
-------------------------------------------------------------
 Seq Scan on tenk1  (cost=0.00..458.00 rows=10000 width=244)

构造计划

explain select * from tenk1;
                         QUERY PLAN
-------------------------------------------------------------
 Seq Scan on tenk1  (cost=0.00..101.00 rows=10000 width=244)
(1 row)

-- 强制将页数转换到cost
set seq_page_cost = 358;
-- 和真是计划一致
explain select * from tenk1;
                         QUERY PLAN
-------------------------------------------------------------
 Seq Scan on tenk1  (cost=0.00..458.00 rows=10000 width=244)
(1 row)

范围扫描–>fail

目标计划

EXPLAIN SELECT * FROM tenk1 WHERE unique1 < 1000;
                                   QUERY PLAN
--------------------------------------------------------------------------------
 Bitmap Heap Scan on tenk1  (cost=24.04..394.56 rows=1001 width=244)
   Recheck Cond: (unique1 < 1000)
   ->  Bitmap Index Scan on tenk1_unique1  (cost=0.00..23.79 rows=1001 width=0)
         Index Cond: (unique1 < 1000)
(4 rows)

构造计划

EXPLAIN SELECT * FROM tenk1 WHERE unique1 < 1000;
                                   QUERY PLAN
---------------------------------------------------------------------------------
 Index Scan using tenk1_unique1 on tenk1  (cost=0.16..25.68 rows=1001 width=244)
   Index Cond: (unique1 < 1000)

修改参数也无法解决问题，参数控制粒度很粗而且值没办法计算出来

而且索引的估计与page_cost有关，这种方法无解

#seq_page_cost = 1.0                    # measured on an arbitrary scale
#random_page_cost = 4.0                 # same scale as above
#cpu_tuple_cost = 0.01                  # same scale as above
#cpu_index_tuple_cost = 0.005           # same scale as above
#cpu_operator_cost = 0.0025             # same scale as above
#parallel_tuple_cost = 0.1              # same scale as above
#parallel_setup_cost = 1000.0

方案二

1. DDL迁移
2. 构造fake数据（按照平均列宽构造）
3. 统计信息迁移覆盖现有统计信息

具体导入流程

1. PG关闭autovacuum，避免统计信息自动更新

   alter system set autovacuum = off;
   select pg_reload_conf();
   
   show autovacuum;
    autovacuum
   ------------
    off
   (1 row)
   
   

2. 迁移ORACLE的表和索引

3. 在PG中插入fake数据，保证数据列宽度按平均宽度构造，行数和统计信息中的真实行数一致。

   这一步保证pg_class中reltuples和relpages和真实数据一致

4. ORACLE导出统计信息并构造符合PG统计信息格式的SQL，更新pg_statistic的对应列

5. PG中执行explain user_sql;得到pg中的执行计划和cost，根据cost可以评估该SQL的执行成本。

验证

使用PG REGRESS用例中的tenk1表作测试（10000行）

构造fake数据注意与真实数据的行宽度一致

初始化

postgres库：真实数据导入

CREATE TABLE tenk1 (
        unique1         int4,
        unique2         int4,
        two                     int4,
        four            int4,
        ten                     int4,
        twenty          int4,
        hundred         int4,
        thousand        int4,
        twothousand     int4,
        fivethous       int4,
        tenthous        int4,
        odd                     int4,
        even            int4,
        stringu1        name,
        stringu2        name,
        string4         name
) WITH OIDS;
CREATE INDEX tenk1_unique1 ON tenk1 USING btree(unique1 int4_ops);
CREATE INDEX tenk1_unique2 ON tenk1 USING btree(unique2 int4_ops);
CREATE INDEX tenk1_hundred ON tenk1 USING btree(hundred int4_ops);
CREATE INDEX tenk1_thous_tenthous ON tenk1 (thousand, tenthous);

COPY tenk1 FROM '/home/mingjie.gmj/projects/postgresql-10.1/src/test/regress/data/tenk.data';
analyze tenk1;

select count(*) from tenk1;
 count
-------
 10000
 
SELECT relpages, reltuples FROM pg_class WHERE relname = 'tenk1';
 relpages | reltuples
----------+-----------
      358 |     10000
(1 row)

select tablename,attname,avg_width from pg_stats where tablename='tenk1';
 tablename |   attname   | avg_width
-----------+-------------+-----------
 tenk1     | unique1     |         4
 tenk1     | unique2     |         4
 tenk1     | two         |         4
 tenk1     | four        |         4
 tenk1     | ten         |         4
 tenk1     | twenty      |         4
 tenk1     | hundred     |         4
 tenk1     | thousand    |         4
 tenk1     | twothousand |         4
 tenk1     | fivethous   |         4
 tenk1     | tenthous    |         4
 tenk1     | odd         |         4
 tenk1     | even        |         4
 tenk1     | stringu1    |        64
 tenk1     | stringu2    |        64
 tenk1     | string4     |        64
(16 rows)

faketank2库：执行DDL后，构造fake数据插入，注意行数与数据宽度与真实数据保持一致（保证relpages是正确的）

CREATE TABLE tenk1 (
        unique1         int4,
        unique2         int4,
        two                     int4,
        four            int4,
        ten                     int4,
        twenty          int4,
        hundred         int4,
        thousand        int4,
        twothousand     int4,
        fivethous       int4,
        tenthous        int4,
        odd                     int4,
        even            int4,
        stringu1        name,
        stringu2        name,
        string4         name
) WITH OIDS;
CREATE INDEX tenk1_unique1 ON tenk1 USING btree(unique1 int4_ops);
CREATE INDEX tenk1_unique2 ON tenk1 USING btree(unique2 int4_ops);
CREATE INDEX tenk1_hundred ON tenk1 USING btree(hundred int4_ops);
CREATE INDEX tenk1_thous_tenthous ON tenk1 (thousand, tenthous);

insert into tenk1 
select trunc(random()*100),
       trunc(random()*100),
       trunc(random()*100),
       trunc(random()*100),
       trunc(random()*100),
       trunc(random()*100),
       trunc(random()*100),
       trunc(random()*100),
       trunc(random()*100),
       trunc(random()*100),
       trunc(random()*100),
       trunc(random()*100),
       trunc(random()*100),
       'x',
       'x',
       'x'
from generate_series(1,10000);

analyze tenk1;

-- 验证1：实际行数一致
select count(*) from tenk1;
 count
-------
 10000
-- 验证2:页数行数统计信息一致
SELECT relpages, reltuples FROM pg_class WHERE relname = 'tenk1';
 relpages | reltuples
----------+-----------
      358 |     10000
(1 row)

-- 验证3:宽度一致
select tablename,attname,avg_width from pg_stats where tablename='tenk1';
 tablename |   attname   | avg_width
-----------+-------------+-----------
 tenk1     | unique1     |         4
 tenk1     | unique2     |         4
 tenk1     | two         |         4
 tenk1     | four        |         4
 tenk1     | ten         |         4
 tenk1     | twenty      |         4
 tenk1     | hundred     |         4
 tenk1     | thousand    |         4
 tenk1     | twothousand |         4
 tenk1     | fivethous   |         4
 tenk1     | tenthous    |         4
 tenk1     | odd         |         4
 tenk1     | even        |         4
 tenk1     | stringu1    |        64
 tenk1     | stringu2    |        64
 tenk1     | string4     |        64
 
-- 开始灌入pg_statistic统计信息
psql faketenk2 -f ~/dump_tenk1.sql

全表扫描–>ok

期望结果

EXPLAIN SELECT * FROM tenk1;
                         QUERY PLAN
-------------------------------------------------------------
 Seq Scan on tenk1  (cost=0.00..458.00 rows=10000 width=244)
(1 row)

实际结果

EXPLAIN SELECT * FROM tenk1;
                         QUERY PLAN
-------------------------------------------------------------
 Seq Scan on tenk1  (cost=0.00..458.00 rows=10000 width=244)
(1 row)

范围扫描–>ok

期望结果

EXPLAIN SELECT * FROM tenk1 WHERE unique1 < 1000;
                                   QUERY PLAN
--------------------------------------------------------------------------------
 Bitmap Heap Scan on tenk1  (cost=24.04..394.56 rows=1001 width=244)
   Recheck Cond: (unique1 < 1000)
   ->  Bitmap Index Scan on tenk1_unique1  (cost=0.00..23.79 rows=1001 width=0)
         Index Cond: (unique1 < 1000)
(4 rows)

实际结果

EXPLAIN SELECT * FROM tenk1 WHERE unique1 < 1000;
                                   QUERY PLAN
--------------------------------------------------------------------------------
 Bitmap Heap Scan on tenk1  (cost=24.04..394.56 rows=1001 width=244)
   Recheck Cond: (unique1 < 1000)
   ->  Bitmap Index Scan on tenk1_unique1  (cost=0.00..23.79 rows=1001 width=0)
         Index Cond: (unique1 < 1000)
(4 rows)

选择性计算：这里直方图信息生效

SELECT histogram_bounds FROM pg_stats WHERE tablename='tenk1' AND attname='unique1';
histogram_bounds | {0,99,199,299,399,499,599,699,799,899,999,1099,1199,1299,1399,1499,1599,1699,1799,1899,1999,2099,2199,2299,2399,2499,2599,2699,2799,2899,2999,3099,3199,3299,3399,3499,3599,3699,3799,3899,3999,4099,4199,4299,4399,4499,4599,4699,4799,4899,4999,5099,5199,5299,5399,5499,5599,5699,5799,5899,5999,6099,6199,6299,6399,6499,6599,6699,6799,6899,6999,7099,7199,7299,7399,7499,7599,7699,7799,7899,7999,8099,8199,8299,8399,8499,8599,8699,8799,8899,8999,9099,9199,9299,9399,9499,9599,9699,9799,9899,9999}

直方图把数据分成等等频的桶，要做的是将数值所在的桶里找出来，计算这个桶中的部分和所有之前桶中的部分。

1000在bucket[11]中，需要选择：前面10个桶全部选择加上第十一个桶的一部分：
KaTeX parse error: No such environment: align at position 8: \begin{̲a̲l̲i̲g̲n̲}̲ selectivity&=(…
评估行数：
KaTeX parse error: No such environment: align at position 8: \begin{̲a̲l̲i̲g̲n̲}̲ rows &= rel\_c…
直方图信息有效可用。

等值查询–>ok

情况1：查询值在MCV中

期望结果

EXPLAIN SELECT * FROM tenk1 WHERE string4 = 'HHHHxx';
                         QUERY PLAN
------------------------------------------------------------
 Seq Scan on tenk1  (cost=0.00..483.00 rows=2500 width=244)
   Filter: (string4 = 'HHHHxx'::name)
(2 rows)

实际结果

EXPLAIN SELECT * FROM tenk1 WHERE string4 = 'HHHHxx';
                         QUERY PLAN
------------------------------------------------------------
 Seq Scan on tenk1  (cost=0.00..483.00 rows=2500 width=244)
   Filter: (string4 = 'HHHHxx'::name)
(2 rows)

选择性计算：依靠MVC

SELECT * FROM pg_stats WHERE tablename='tenk1' AND attname='string4';
schemaname             | public
tablename              | tenk1
attname                | string4
inherited              | f
null_frac              | 0
avg_width              | 64
n_distinct             | 4
most_common_vals       | {AAAAxx,HHHHxx,OOOOxx,VVVVxx}
most_common_freqs      | {0.25,0.25,0.25,0.25}
histogram_bounds       |
correlation            | 0.250375
most_common_elems      |
most_common_elem_freqs |
elem_count_histogram   |

评估行数
KaTeX parse error: No such environment: align at position 8: \begin{̲a̲l̲i̲g̲n̲}̲ rows &= select…

情况2：查询值不在MCV中

EXPLAIN SELECT * FROM tenk1 WHERE string4 = 'sss';
QUERY PLAN
---------------------------------------------------------
 Seq Scan on tenk1  (cost=0.00..483.00 rows=1 width=244)
   Filter: (string4 = 'sss'::name)

这一列比较特殊，四个mcv占了全部数据的100%，所以任何其他条件显然没有数据

行数按下面公式可以估算，即 非mcv出现的频率在 / 非mcv的个数=单个非mcv值的出现概率

selectivity = (1 - sum(mvc_f))/(num_distinct - num_mcv)

情况3：范围查询使用MCV

CREATE TABLE person (
        name            text,
        age             int4,
        location        point
);
COPY person FROM '/home/mingjie.gmj/projects/postgresql-10.1/src/test/regress/data/person.data'
select relpages,reltuples from pg_class where relname='person';
 relpages | reltuples
----------+-----------
        1 |        50
        
select * from pg_stats where tablename='person' and attname='age';
-[ RECORD 1 ]----------+---------------------------------------------------
schemaname             | public
tablename              | person
attname                | age
inherited              | f
null_frac              | 0
avg_width              | 4
n_distinct             | -0.36
most_common_vals       | {38,78,88,18,58,68,28,48,98,8}
most_common_freqs      | {0.12,0.12,0.12,0.1,0.08,0.08,0.06,0.06,0.06,0.04}
histogram_bounds       | {19,20,24,25,30,34,40,50}
correlation            | 0.317167
most_common_elems      |
most_common_elem_freqs |
elem_count_histogram   |


explain select * from person where age<45;
                       QUERY PLAN
--------------------------------------------------------
 Seq Scan on person  (cost=0.00..1.62 rows=23 width=25)
   Filter: (age < 45)

$rows=(mvc\_selectivity+his\_selectivity) * rel\_cardinality$

综上说明MCV的数据是正常生效的

多条件–>ok

期望结果

EXPLAIN SELECT * FROM tenk1 WHERE unique1 < 1000 AND stringu1 = 'xxx';
                                   QUERY PLAN
--------------------------------------------------------------------------------
 Bitmap Heap Scan on tenk1  (cost=23.79..396.81 rows=1 width=244)
   Recheck Cond: (unique1 < 1000)
   Filter: (stringu1 = 'xxx'::name)
   ->  Bitmap Index Scan on tenk1_uniqux`e1  (cost=0.00..23.79 rows=1001 width=0)
         Index Cond: (unique1 < 1000)

实际结果

EXPLAIN SELECT * FROM tenk1 WHERE unique1 < 1000 AND stringu1 = 'xxx';
                                   QUERY PLAN
--------------------------------------------------------------------------------
 Bitmap Heap Scan on tenk1  (cost=23.79..396.81 rows=1 width=244)
   Recheck Cond: (unique1 < 1000)
   Filter: (stringu1 = 'xxx'::name)
   ->  Bitmap Index Scan on tenk1_unique1  (cost=0.00..23.79 rows=1001 width=0)
         Index Cond: (unique1 < 1000)

连接–>ok

期望结果

EXPLAIN SELECT * FROM tenk1 t1, tenk1 t2 WHERE t1.unique1 < 50 AND t1.unique2 = t2.unique2;
                                      QUERY PLAN
--------------------------------------------------------------------------------------
 Nested Loop  (cost=4.97..464.53 rows=51 width=488)
   ->  Bitmap Heap Scan on tenk1 t1  (cost=4.68..144.59 rows=51 width=244)
         Recheck Cond: (unique1 < 50)
         ->  Bitmap Index Scan on tenk1x`_unique1  (cost=0.00..4.67 rows=51 width=0)
               Index Cond: (unique1 < 50)
   ->  Index Scan using tenk1_unique2 on tenk1 t2  (cost=0.29..6.26 rows=1 width=244)
         Index Cond: (unique2 = t1.unique2)
(7 rows)

实际结果

EXPLAIN SELECT * FROM tenk1 t1, tenk1 t2 WHERE t1.unique1 < 50 AND t1.unique2 = t2.unique2;
                                      QUERY PLAN
--------------------------------------------------------------------------------------
 Nested Loop  (cost=4.97..476.53 rows=51 width=488)
   ->  Bitmap Heap Scan on tenk1 t1  (cost=4.68..144.59 rows=51 width=244)
         Recheck Cond: (unique1 < 50)
         ->  Bitmap Index Scan on tenk1_unique1  (cost=0.00..4.67 rows=51 width=0)
               Index Cond: (unique1 < 50)
   ->  Index Scan using tenk1_unique2 on tenk1 t2  (cost=0.29..6.50 rows=1 width=244)
         Index Cond: (unique2 = t1.unique2)
(7 rows)