sparkSQL metadata cache is not synchronized The table structure of beeline connection is inconsistent with hive

For the pits encountered before, the spark thirft server is connected through beeline. When the table structure is modified in Hive, such as replace/add/change columns, the table structure does not change, and it is still the old table structure, so the data cannot be verified.

The operation steps are as follows:

-- 1.create table
-- hive/beeline execute
create table test_table(id int);

-- 2.check metadata in beeline
desc test_table;

-- 3. add columns for test_table
-- hive execute
alter table test_table add columns (name string);
alter table test_table replace columns (id int,name_new string);

-- 4. check metadata in hive
-- hive execute
desc test_table;

-- 5.check metadata in beeline
-- beeline execute
desc test_table;

-- 6. refresh metadata cache
-- beeline execute
refresh table test_table;

-- 7.check metadata in beeline
-- beeline execute
desc test_table;

After testing, there is no such problem in spark2.1.

Guess you like

Origin http://43.154.161.224:23101/article/api/json?id=325032417&siteId=291194637