Introduction: The first time you install and use datax, or your server Mysql version is <= 5.7, please refer to my previous essay: https://www.cnblogs.com/zifan/p/9194793.html .
Background: Mysql after upgrading from 5.6 to 8.0.19, found that the original normal running datax wrong.
My first look at the error message :( please see page zoom)
1 2020-03-18 10:00:10.038 [0-0-0-writer] ERROR StdoutPluginCollector - 2 java.sql.SQLException: Could not retrieve transation read-only status server 3 at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:996) ~[mysql-connector-java-5.1.34.jar:5.1.34] 4 at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:935) ~[mysql-connector-java-5.1.34.jar:5.1.34] 5 at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:924) ~[mysql-connector-java-5.1.34.jar:5.1.34] 6 at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:870) ~[mysql-connector-java-5.1.34.jar:5.1.34] 7 at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:902) ~[mysql-connector-java-5.1.34.jar:5.1.34] 8 at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:892) ~[mysql-connector-java-5.1.34.jar:5.1.34] 9 at com.mysql.jdbc.ConnectionImpl.isReadOnly(ConnectionImpl.java:3607) ~[mysql-connector-java-5.1.34.jar:5.1.34] 10 at com.mysql.jdbc.ConnectionImpl.isReadOnly(ConnectionImpl.java:3576) ~[mysql-connector-java-5.1.34.jar:5.1.34] 11 at com.mysql.jdbc.PreparedStatement.checkReadOnlySafeStatement(PreparedStatement.java:1114) ~[mysql-connector-java-5.1.34.jar:5.1.34] 12 at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1134) ~[mysql-connector-java-5.1.34.jar:5.1.34] 13 at com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Task.doOneInsert(CommonRdbmsWriter.java:382) [plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na] 14 at com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Task.doBatchInsert(CommonRdbmsWriter.java:362) [plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na] 15 at com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Task.startWriteWithConnection(CommonRdbmsWriter.java:297) [plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na] 16 at com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Task.startWrite(CommonRdbmsWriter.java:319) [plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na] 17 at com.alibaba.datax.plugin.writer.mysqlwriter.MysqlWriter$Task.startWrite(MysqlWriter.java:78) [mysqlwriter-0.0.1-SNAPSHOT.jar:na] 18 at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:56) [datax-core-0.0.1-SNAPSHOT.jar:na] 19 at java.lang.Thread.run(Thread.java:748) [na:1.8.0_241] 20 Caused by: java.sql.SQLException: Unknown system variable 'tx_read_only'
Just focus on the first and last lines. The first line you go to Baidu, some of the information will tell you to let you set up a database under "transaction isolation level" Reference: https://www.cnblogs.com/jeffen/p/6005410.html
SET GLOBAL transaction_isolation='READ-COMMITTED';
I also tried, in terms of Mysql8 still being given , but other mysql version you can try this solution . Reference: https://www.cnblogs.com/lewic/p/10685004.html , find the ultimate cause should still drive version is not the problem.
Let's look last line: java.sql.SQLException: Unknown system variable 'tx_read_only', most of the information indicates that the drive is inconsistent versioning issues. Summing up the above, there is only one cause of death, is the "drive version mysql wrong", so the solution is to upgrade the natural drives.
First look at the original drive:
ll {%Datax_Path}/plugin/writer/mysqlwriter/libs/ | grep mysql-con
As of 2020/03/23, GitHub MySQL driver download the latest datax tar package contains the mysql-connector-java-5.1.34.jar. It does not match ah, this time someone will think first of 8 direct download from the Internet or replace the drive throw into like,
Reference: https://blog.csdn.net/jason_9527/article/details/100995505
I thought no problem, but can not completely solve the problem, but there will be some error, warning out. such as:
1、ERROR: zeroDateTimeBehavior=convertToNull要改为CONVERT_TO_NULL
2、WARN: Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary. 3、WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
So we have to adopt "modify the source code, compile it yourself" approach to solve all the problems. Reference: http://www.manongjc.com/detail/14-iexdburliccybhe.html
The first step Download Source:
https://github.com/alibaba/DataX/archive/master.zip git clone mode or direct download.
The second step modified version mysql-connector-java-driven:
Locate {DataX_source_code_home} / mysqlreader / and {} DataX_source_code_home pom.xml file in the / mysqlwriter /, changed the value of version 8.0.X (can directly search for 'mysql-connector-java' rapid positioning, X want to write your own to the version number)
If you only use a write-mysql, you can only modify the xml file in the directory mysqlwriter.
The third step is to modify the value convertToNull zeroDateTimeBehavior CONVERT_TO_NULL:
编辑{DataX_source_code_home}/plugin-rdbms-util/src/main/java/com/alibaba/datax/plugin/rdbms/util/DataBaseType.java
The full text is replaced convertToNull CONVERT_TO_NULL
# CHRIP {DataX_source_code_home} / convertToNull all the files contained in the folder string, to locate the file and then replace
vim {DataX_source_code_home} / plugin-rdbms -util / src / main / java / com / alibaba / datax / plugin / rdbms / util / DataBaseType.java
# replace text and command mode
: % S / convertToNull / CONVERT_TO_NULL /
This, if you skip the fourth, five-step direct packaging datax, the program has run correctly, if you want the perfect solution other warnings, follow the steps to continue.
The fourth step is to modify the name of jdbc driver is com.mysql.cj.jdbc.Driver com.mysql.jdbc.Driver
vi {DataX_source_code_home}/plugin-rdbms-util/src/main/java/com/alibaba/datax/plugin/rdbms/util/DataBaseType.java vi {DataX_source_code_home}/adswriter/src/main/java/com/alibaba/datax/plugin/writer/adswriter/load/AdsHelper.java
The full text is replaced com.mysql.jdbc.Driver com.mysql.cj.jdbc.Driver.
The fifth step jdbc link additional useSSL = false setting :
For example: python datax.py file.json, the file.json configuration items: "the jdbcUrl": "JDBC: MySQL: // $ {} writer_gateway / OMS & useUnicode = characterEncoding = UTF-to true. 8? & = UseSSL to false ",
A sixth step by packing maven :( reference Git the procedure provided by the developer to a)
cd {DataX_source_code_home} mvn -U clean package assembly:assembly -Dmaven.test.skip=true
Packaging Success, log shows the following:
[INFO] BUILD SUCCESS
[INFO] -----------------------------------------------------------------
[INFO] Total time: 08:12 min
[INFO] Finished at: 2015-12-13T16:26:48+08:00
[INFO] Final Memory: 133M/960M
[INFO] -----------------------------------------------------------------
After successful packing bag located DataX {DataX_source_code_home} / target / datax / datax /, see datax / plugin / writer / mysqlwriter / libs / directory, new drivers and already contains jar package
last step:
Datax is to move to your desired location, then execute: python datax.py ./file.json, so far, OVER