使用sqoop将mysql中的数据导入hive,遇到错误如下:
18/05/14 16:35:47 ERROR manager.SqlManager: Error executing statement: java.sql.SQLException: Access denied for user 'root'@'172.16.21.22' (using password: YES) java.sql.SQLException: Access denied for user 'root'@'172.16.21.22' (using password: YES) at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:964) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3973) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3909) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:873) at com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1710) at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1226) at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2205) at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2236) at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2035) at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:790) at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.mysql.jdbc.Util.handleNewInstance(Util.java:425) at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:400) at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:330) at java.sql.DriverManager.getConnection(DriverManager.java:664) at java.sql.DriverManager.getConnection(DriverManager.java:247) at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:904) at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52) at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:763) at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:786) at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:289) at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:260) at org.apache.sqoop.manager.SqlManager.getColumnTypesForQuery(SqlManager.java:253) at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:336) at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1858) at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1657) at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:106) at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:494) at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:621) at org.apache.sqoop.Sqoop.run(Sqoop.java:147) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243) at org.apache.sqoop.Sqoop.main(Sqoop.java:252) 18/05/14 16:35:47 ERROR tool.ImportTool: Import failed: java.io.IOException: No columns to generate for ClassWriter at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1663) at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:106) at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:494) at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:621) at org.apache.sqoop.Sqoop.run(Sqoop.java:147) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243) at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
sqoop语句如下:
/usr/bin/sqoop import --connect jdbc:mysql://172.xx.xx.174:3306/autozi_jyj --username root --password "aotuzi" --query "select ID,SCENE,PAID_AMOUNT,STATUS,REPLACE(REMARK,'\\\\',''),PAY_CATEGORY,PAY_TYPE,PAYER_ID,PAYEE_ID,REPLACE(PAYER_BANK_ACCOUNT,'\\\\',''),REPLACE(PAYEE_BANK_ACCOUNT,'\\\\',''),CREATE_TIME,PAID_TIME,CREATE_USER_ID,REPLACE(PAY_SERIAL_NUMBER,'\\\\',''),DOMAIN,REPLACE(SER_NO,'\\\\',''),REPLACE(QUERY_ID,'\\\\',''),REPLACE(REFER_NO,'\\\\',''),REPLACE(ORDER_HEADER_CODE,'\\\\',''),REPLACE(PAY_CODE,'\\\\',''),REPLACE(PAY_CHANNEL,'\\\\','') from autozi_jyj.pay_payment_sales where \$CONDITIONS" --split-by "ID" -m 1 --fields-terminated-by "" --bindir /home/hadoop/edw/etl/sqoop/classes/bdl/pay_payment_sales/ --outdir /home/hadoop/edw/etl/sqoop/src/bdl/ --target-dir /user/hive/warehouse/bdl.db/bdl_jyj_pay_payment_sales --hive-import --hive-table bdl.bdl_jyj_pay_payment_sales --null-string '\\N' --null-non-string '\\N' --hive-drop-import-delims
经过网上搜索,大概有这几种原因:
1-mysql驱动有问题
有的人是使用了mysql-connector-5.1.1X.jar报错,换成mysql-connector-5.1.31.jar成功.也可以尝试mysql-connector-5.1.41.jar
2-root用户权限问题.
登录语句用导出数据的mysql数据库,运行授权语句.()
grant all privileges on *.* to root@'%' identified by 'your password'; grant all privileges on *.* to root@'your IP' identified by 'your password'; flush privileges;
3-以上两种办法我都试了,运行还是报错.最终是将root用户换成了一个其他的只有写权限的用户,查询成功.
mysql -u root -p password 登录导出数据的库 运行下面语句创建一个只有读权限的用户: grant select on *.* to username@'yourSqoopIp' identified by 'yourPassword';
步骤3做完以后,运行sqoop成功.应该是root用户授权太乱了,我机器上有好几个root对应不同的机器和权限.
参考:https://blog.csdn.net/fanrenxiang/article/details/72597166