wild_table

wild_table

Table of Contents

• 1. problem
• 2. Solution

1 issue

• Width table storage data space, will face a problem: many, the data is stored N times. N times will take up more space.

2 Solutions

• Storage json format

  test:

  -- 建表
  drop table if exists jsontest;
  
  CREATE TABLE IF NOT EXISTS jsonTest
  (teacher_name varchar(10),
  major varchar(10),
  students_info string comment "学生信息"
  )
  comment "学生课程信息"
  row format serde 'org.apache.hive.hcatalog.data.JsonSerDe'
  LOCATION
    'hdfs://nameservice1/user/hive/warehouse/bigdata.db/jsontest';
  
  insert into jsontest values
  ('t1','语文','{"grade":1,"info":{"name":"xinzi","age":14,"sex":"M"}}|{"grade":3,"info":{"name":"lisi","age":14,"sex":"M"}}'),
  ('t2','maths','{"Grade ": 2," info ": {" name ":" zhangs "," Age ": 14," Sex ":" F. "}} ') 
  - to create a view
  select row_number () over (), . jsontest from A * A;
  - View existing data
  ;
  
  
  
  
  drop view if exists v_jsontest;
  create view if not exists v_jsontest
  as
  select teacher_name,major,a.grade,b.name,b.age,b.gender from jsontest lateral view explode(split(students_info,'\\|')) st as stin
  lateral view json_tuple(stin,'grade','info') a as grade,info
  lateral view json_tuple(a.info,'name','age','sex') b as name,age,gender;
  

- the original table data

hive> select * from jsontest; OK t1 语文 {"grade":1,"info":{"name":"xinzi","age":14,"sex":"M"}}|{"grade":3,"info":{"name":"lisi","age":14,"sex":"M"}} t2 maths {"grade":2,"info":{"name":"zhangs","age":14,"sex":"F"}} Time taken: 0.054 seconds, Fetched: 2 row(s)

- the reference results

hive> select * from v_jsontest; ………. OK t1 语文 1 xinzi 14 M t1 语文 3 lisi 14 M t2 maths 2 zhangs 14 F Time taken: 12.933 seconds, Fetched: 3 row(s)

Author: halberd.lee

Created: 2020-04-09 Thu 12:50

Validate