20/09/24 14:38:37 INFO hive.HiveImport: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:An exception was thrown while adding/validating class(es) : Column length too big for column 'PARAM_VALUE' (max = 21845); use BLOB or TEXT instead
This problem occurred when I used sqoop to import database tables and data from mysql to hive database.
The problem is hive
This error also occurs when creating a table in hive
The error reported by sqoop is like this
20/09/24 14:38:37 ERROR tool.ImportTool: Encountered IOException running import job: java.io.IOException: Hive exited with status 1
at org.apache.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java:389)
at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:339)
at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:240)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:514)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)Baidu has read a lot and said to replace the mysql driver package in the sqoop/lib directory with mysql-connector-java-5.1.32.jar
I was misled by this error, the actual error is in the info above
20/09/24 14:38:11 INFO hive.HiveImport:
20/09/24 14:38:11 INFO hive.HiveImport: Logging initialized using configuration in jar:file:/export/servers/hive-1.1.0-cdh5.14.0/lib/hive-common-1.1.0-cdh5.14.0.jar!/hive-log4j.properties
20/09/24 14:38:37 INFO hive.HiveImport: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:An exception was thrown while adding/validating class(es) : Column length too big for column 'PARAM_VALUE' (max = 21845); use BLOB or TEXT instead
20/09/24 14:38:37 INFO hive.HiveImport: com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Column length too big for column 'PARAM_VALUE' (max = 21845); use BLOB or TEXT insteadthe reason:
Coding problem, just change the database code associated with hive and mysql to latin1 again (hive found online only supports latin1 and has not been tested)
Solution:
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| hive |
| mysql |
| performance_schema |
| sys |
| test1 |
+--------------------+
6 rows in set (0.00 sec)
mysql> show variables like '%char%';
+--------------------------+----------------------------------------------+
| Variable_name | Value |
+--------------------------+----------------------------------------------+
| character_set_client | utf8 |
| character_set_connection | utf8 |
| character_set_database | utf8 |
| character_set_filesystem | binary |
| character_set_results | utf8 |
| character_set_server | utf8 |
| character_set_system | utf8 |
| character_sets_dir | /export/servers/mysql-5.7.31/share/charsets/ |
+--------------------------+----------------------------------------------+
8 rows in set (0.01 sec)
mysql> alter database hive character set latin1;
Query OK, 1 row affected (0.00 sec)
mysql> flush privileges;
mysql> show variables like '%char%';
+--------------------------+----------------------------------------------+
| Variable_name | Value |
+--------------------------+----------------------------------------------+
| character_set_client | utf8 |
| character_set_connection | utf8 |
| character_set_database | latin1 |
| character_set_filesystem | binary |
| character_set_results | utf8 |
| character_set_server | utf8 |
| character_set_system | utf8 |
| character_sets_dir | /export/servers/mysql-5.7.31/share/charsets/ |
+--------------------------+----------------------------------------------+
8 rows in set (0.01 sec)