0 phenomenon
When synchronizing data with sqoop, the log shows a failure of the sqoop task, but on the data display side, it is found that the data in the mysql table has been imported. In order to analyze the reason in a deeper level, I observed the specific log in yarn and found that the error content was: Caused by: java.sql.SQLException: null, message from server: "Host '10.9.4.41' is blocked because of many connection errors; unblock with'mysqladmin flush-hosts'". The details are shown in the screenshot below:
sqoop log:
mysql side: the actual data has been exported after the error is reported
Check the specific logs in yarn as follows:
2020-12-11 19:36:40,014 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1607676423534_0062_m_000002_0: Error: java.io.IOException: java.sql.SQLException: null, message from server: "Host '10.9.4.41' is blocked because of many connection errors; unblock with 'mysqladmin flush-hosts'"
at org.apache.sqoop.mapreduce.ExportOutputFormat.getRecordWriter(ExportOutputFormat.java:79)
at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:647)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:767)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
Caused by: java.sql.SQLException: null, message from server: "Host '10.9.4.41' is blocked because of many connection errors; unblock with 'mysqladmin flush-hosts'"
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1078)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:989)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:975)
at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1112)
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2488)
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2521)
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2306)
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:839)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:49)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:421)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:350)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:302)
at org.apache.sqoop.mapreduce.AsyncSqlRecordWriter.<init>(AsyncSqlRecordWriter.java:78)
at org.apache.sqoop.mapreduce.ExportOutputFormat$ExportRecordWriter.<init>(ExportOutputFormat.java:95)
at org.apache.sqoop.mapreduce.ExportOutputFormat.getRecordWriter(ExportOutputFormat.java:77)
... 8 more2 Cause analysis and solutions
The same ip incorrectly connects to the database multiple times in a short period of time, causing the number of incorrect connections to exceed the set maximum value (max_connection_errors) and blocking the current IP's request to the database.
After knowing the specific reason, we went to the MYSQL database to perform the following operations:
- (1) flush hosts (clear cache)
- (2) Modify the number of max_connection_errors, and set the value a bit larger. For example: 100000
If you need a permanent solution, you need to modify the corresponding attribute value in the MYSQL configuration file.
important point:
When the client connects to the server timeout (more than connect_timeout ), the server will log an error to the client. When the number of errors reaches max_connect_errors , the client will be locked. Therefore, according to the business, try to set this value as large as possible. The default value of mysql is 10 , and we can set it to a larger value according to specific needs. Here it is set to 100000. (Not the bigger the better, the bigger the security is when attacked)
According to the above, we choose the following configuration in our production environment, and you can choose according to the situation:
among them
- connect_timeout：600000
- max_connect_errors：100000
