background
Previously, I encountered a need SQLServerto migrate an online database from conversion to MySQL. There are more than 70 data tables, and the total amount of data is not large. From the Internet, there are many recommended uses SQLyog, and there are Oracle MySQL Serverofficial ones Workbenchfor migration, but the steps are a bit cumbersome; later, I found DB2DBthis tool from a corner of an article, which comes from Misoft Technology.
Official introduction: DB2DBIt is used for data conversion between various mainstream databases. The first version was developed in 2015. After many years of product iterations, it has passed the verification of more than 400 customers. It is a mature and handy product. In terms of software functions, the current version 1.30.107supports the mutual migration and conversion of 6 databases, such as SQLServer, MySQL, SQLite, Access, SQLServer CE, , etc., which is sufficient. PostgreSQLFor the specific migration process, refer to: Heterogeneous database conversion tool experience: SQLServer data conversion and migration to MySQL.
However, it should be noted that the tool of DB2DBthis CSarchitecture defaults to an unauthorized version (trial version) after downloading, with a limit of 50,000 records .
Let's first introduce the protagonist of this migration process: ETLCloud , which reconstructs the enterprise data fusion architecture, solves various complex, instant, and highly compliant data integration needs, and covers one-stop coverage: task development, task scheduling, blood relationship analysis, data quality management, data service development, and task monitoring. Today, the same migration work as above can be completed through the version of the data integration tool based on the microservice architecture BS: ETLCloud Community Edition .
migration practice
Next, enter our migration practice: the whole process is zero-code, visualized, drag-and-drop, and the batch migration of heterogeneous data tables can be completed with one click of the mouse. The most important thing is that the community version is free!
Note: The community version is selected here, and Dockerthe deployment method is lightweight and quick to start: docker pull ccr.ccs.tencentyun.com/restcloud/restcloud-etl:V2.2.
Data source configuration
SQLServerand MySQLare both ETLCloudbuilt-in data sources supported by the community edition, just create a new data source directly.
- Configure Source: SQL Server
Create a new data source, select it SQLServer, and fill in the IP: port and user password information.
The test connection is successful~
- Configure Sink: MySQL
Create a new data source, select it MySQL, and fill in the IP: port and user password information.
The test connection is successful~
Create applications and processes
Create an application and fill in the basic application configuration information. Then, create a data flow and fill in the information.
After creating the process, you can click the "Process Design" button to enter the process visualization configuration page.
Visual configuration process
Before configuring the process, briefly introduce the various areas of this configuration page: the left side is the component area, the middle top is the functional area, and most of the middle is the process drawing area. Double-click the component in the drawing area, and you can see the component detailed configuration item area that pops up in a drawer style.
- Database table batch input: SQLServer
In the library table synchronization component on the left, select "Batch Input Library Table", drag it to the central process drawing area, and double-click to enter the configuration stage.
SQLServerStep 1: Select the data source we have configured . Here, select the page-by-page reading method, and the user can be empty.
Step 2: You can load SQLServerexisting tables and views in , there are 74 tables and 4 views in total.
According to the needs, select the data table to be migrated, select all tables and views here, and save.
- Database table batch output: MySQL
In the library table synchronization component on the left, select "Batch Output of Library Table", drag it to the central process drawing area, and double-click to enter the configuration stage.
Step 1: Select the MySQL data source we configured.
Step 2: Configurable output options: whether to clear the table data, whether to automatically build the table, how to update the data, etc.
- Improve the process
Finally, by 流程线connecting the start , database table batch input , library table batch output , and end components respectively, the visual configuration of batch database table data migration is completed. Done~
run process
Save the process and run the process; then you can view the corresponding process log and visually monitor the migration progress.
problem record
Of course, the entire migration process is not a one-off process. Several problems were encountered in the data source configuration and migration links, which are recorded as follows.
SQLServerThe data source reported a connection error due to a clientJDKversion problem
Problem description: Connection failed (The driver could not establish a secure connection to SQL Server by using Secure Sockets Layer (SSL) encryption. Error: “The server selected protocol version TLS10 is not accepted by client preferences [TLS12]”. ClientConnectionId: aacc4edf-d ca7-4296-bd33-a4462fcf0033) Problem analysis: The protocol version used by the client is , but the protocol version of the server is , that is, the database version is too low, and the client version needs to
be TLS12lowered TLS10to
adapt . First enter the container and navigate to the file.jdk/jre/lib/security/java.securityjdk.tls.disabledAlgorithms
java.security
# 进入容器
[root@etl ~]# docker exec -it de63b29c71d0 /bin/bash
# 查看ETLCloud容器中自带的jdk版本信息
root@de63b29c71d0:/usr# java -version
java version "1.8.0_333"
Java(TM) SE Runtime Environment (build 1.8.0_333-b02)
Java HotSpot(TM) 64-Bit Server VM (build 25.333-b02, mixed mode)
# 定位到java.security文件
root@de63b29c71d0:/usr# which java
/usr/jdk/bin/java
root@de63b29c71d0:/usr# ls
bin games include jdk lib local sbin share src start.sh tomcat velocity.log
root@de63b29c71d0:/usr# cd jdk/jre/lib/security/
root@de63b29c71d0:/usr/jdk/jre/lib/security# ls
blacklist blacklisted.certs cacerts java.policy java.security javaws.policy policy public_suffix_list.dat trusted.libraries
root@de63b29c71d0:/usr/jdk/jre/lib/security# vi java.security
bash: vi: command not found
However, since the container does not provide vithe command, the file in the container is first /usr/jdk/jre/lib/security/java.securitycopied to the host, and then copied back to the container after modification.
# 从容器中复制文件到宿主机
[root@etl opt]# docker cp de63b29c71d0://usr/jdk/jre/lib/security/java.security /opt/
Successfully copied 58.9kB to /opt/
# 修改配置文件
[root@etl opt]# vi java.security
删除jdk.tls.disabledAlgorithms中的TLS1.*算法配置
# 从宿主机复制文件到容器中
[root@etl opt]# docker cp /opt/java.security de63b29c71d0://usr/jdk/jre/lib/security/
Successfully copied 58.9kB to de63b29c71d0://usr/jdk/jre/lib/security/
MySQLThe data source reports a connection error due to database table permission issues
Problem description: Access denied for user 'root'@'%' to database qwert.
Solution: Enable remote access to the target database, and configure permissions such as creating tables.
grant create,alter,drop,index,select,insert,update,delete on qwert.* to root@'%' identified by 'root';
flush PRIVILEGES;
ELTCloudWhen readingSQLServerthe table,DateTimethe field of the type will be recognized asTimeStampthe type, resulting inMySQLan error when writing date data in
Problem description: Caused by: com.mysql.cj.jdbc.exceptions.MysqlDataTruncation: Data truncation: Incorrect datetime value: '1753-01-01 12:00:00' for column 'ErrorTime' at row 1.
Solution: Modify the migration process. In the configuration of the library table batch input component, first configure only the table structure to be synchronized; after the execution process completes the table structure synchronization, manually change the type to the type on the MySQLterminal .TimeStampDateTime
Summarize
The above is based on ETLCloudthe database table synchronization component to realize batch migration of heterogeneous data tables, and complete the practice of data migration from SQLServerto MySQLdatabase table, which is simple, direct and effective; in addition, it also records several configuration-related problems and solutions during the migration process. Compared with the previously used CStype of migration tools, ETLCloudthe interface is more friendly and easier to use, and the functions of the community version are powerful enough.
Reference
If you have any questions or any bugs are found, please feel free to contact me.
Your comments and suggestions are welcome!