For the pits encountered before, the spark thirft server is connected through beeline. When the table structure is modified in Hive, such as replace/add/change columns, the table structure does not change, and it is still the old table structure, so the data cannot be verified.
The operation steps are as follows:
-- 1.create table -- hive/beeline execute create table test_table(id int); -- 2.check metadata in beeline desc test_table; -- 3. add columns for test_table -- hive execute alter table test_table add columns (name string); alter table test_table replace columns (id int,name_new string); -- 4. check metadata in hive -- hive execute desc test_table; -- 5.check metadata in beeline -- beeline execute desc test_table; -- 6. refresh metadata cache -- beeline execute refresh table test_table; -- 7.check metadata in beeline -- beeline execute desc test_table;
After testing, there is no such problem in spark2.1.