环境:
数据源:oracle 11.2G
hadoop:ambari 2.6.1 hdp2.6.4
sqoop :1.4.6
出现的问题:
问题1:
ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.RuntimeException: Could not load db driver class: oracle.jdbc.OracleDriver
java.lang.RuntimeException: Could not load db driver class: oracle.jdbc.OracleDriver
解决方法:将ojdbc6.jar拷贝到sqoop/lib目录下
问题2:
ERROR manager.SqlManager: Error executing statement: java.sql.SQLException: Listener refused the connection with the following error:
ORA-12505, TNS:listener does not currently know of SID given in connect descriptor
解决方法:
将--connect jdbc:oracle:thin:@<host>:1521:orcl
改为--connect jdbc:oracle:thin:@//<host>:1521/orcl
问题3:
ERROR tool.ImportTool: Error during import: No primary key could be found for table UPRO_LNT.T_LNT_CARD_INFO. Please specify one with --split-by or perform a sequential import with '-m 1'.
解决方法:
数据表没有主键,3种解决方法:
1.修改源表,建立主键。
2.将你的map个数设置为1(Sqoop默认是4)
3.使用--split-by,后面跟上表的最后一列名字。从而能够对数据进行分行 ,命令如下:
sqoop import --connect jdbc:oracle:thin:@//10.240.32.21:1521/clearing --username LNT_BIGDATA --password ClR#snowBB057 --table UPRO_LNT.T_LNT_CARD_INFO --target-dir /source/db/cardinfo/20180507 --m 5 --columns "LCN,FCN,RN,BAL,IS_BATCH,IS_TIME,IS_TYPE,IS_VERSION,IS_AREA,DEPOSTIT,VALID_TIME,CARD_TYPE,PURSE_TYPE,CARD_MARK,TRAN_TIM,IS_MD,TRAN_NUM,IS_NUMBER,ULIMIT_BAL" --split-by FCN
问题4:
ERROR tool.ImportTool: Encountered IOException running import job: org.apache.hadoop.security.AccessControlException: Permission denied: user=root, access=WRITE, inode="/user/root/.staging":hdfs:hdfs:drwxr-xr-x
解决方法:
2种方法:
1.切换到hdfs用户 su hdfs
sqoop ............
2.sudo -u hdfs sqoop .......
问题5:
ERROR tool.ImportTool: Encountered IOException running import job: java.io.IOException: Generating splits for a textual index column allowed only in case of "-Dorg.apache.sqoop.splitter.allow_text_splitter=true" property passed as a parameter
解决方法:
sudo -u hdfs sqoop import -Dorg.apache.sqoop.splitter.allow_text_splitter=true ......
数据源:oracle 11.2G
hadoop:ambari 2.6.1 hdp2.6.4
sqoop :1.4.6
出现的问题:
问题1:
ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.RuntimeException: Could not load db driver class: oracle.jdbc.OracleDriver
java.lang.RuntimeException: Could not load db driver class: oracle.jdbc.OracleDriver
解决方法:将ojdbc6.jar拷贝到sqoop/lib目录下
问题2:
ERROR manager.SqlManager: Error executing statement: java.sql.SQLException: Listener refused the connection with the following error:
ORA-12505, TNS:listener does not currently know of SID given in connect descriptor
解决方法:
将--connect jdbc:oracle:thin:@<host>:1521:orcl
改为--connect jdbc:oracle:thin:@//<host>:1521/orcl
问题3:
ERROR tool.ImportTool: Error during import: No primary key could be found for table UPRO_LNT.T_LNT_CARD_INFO. Please specify one with --split-by or perform a sequential import with '-m 1'.
解决方法:
数据表没有主键,3种解决方法:
1.修改源表,建立主键。
2.将你的map个数设置为1(Sqoop默认是4)
3.使用--split-by,后面跟上表的最后一列名字。从而能够对数据进行分行 ,命令如下:
sqoop import --connect jdbc:oracle:thin:@//10.240.32.21:1521/clearing --username LNT_BIGDATA --password ClR#snowBB057 --table UPRO_LNT.T_LNT_CARD_INFO --target-dir /source/db/cardinfo/20180507 --m 5 --columns "LCN,FCN,RN,BAL,IS_BATCH,IS_TIME,IS_TYPE,IS_VERSION,IS_AREA,DEPOSTIT,VALID_TIME,CARD_TYPE,PURSE_TYPE,CARD_MARK,TRAN_TIM,IS_MD,TRAN_NUM,IS_NUMBER,ULIMIT_BAL" --split-by FCN
问题4:
ERROR tool.ImportTool: Encountered IOException running import job: org.apache.hadoop.security.AccessControlException: Permission denied: user=root, access=WRITE, inode="/user/root/.staging":hdfs:hdfs:drwxr-xr-x
解决方法:
2种方法:
1.切换到hdfs用户 su hdfs
sqoop ............
2.sudo -u hdfs sqoop .......
问题5:
ERROR tool.ImportTool: Encountered IOException running import job: java.io.IOException: Generating splits for a textual index column allowed only in case of "-Dorg.apache.sqoop.splitter.allow_text_splitter=true" property passed as a parameter
解决方法:
sudo -u hdfs sqoop import -Dorg.apache.sqoop.splitter.allow_text_splitter=true ......