Kettle optimized table output

       Today, when the classroom information was organized and entered into the database, the mysql server was replaced. As a result, the data insertion speed is extremely slow, so tuning is required—increasing the data insertion speed. So I considered some ways to solve the data inserted in nearly 7 minutes in less than three seconds. The following are the optimization methods and results

Table of contents

        mysql connection number optimization

Kettle database insert operation optimization

Kettle allows multiple threads

Appropriately increase the size of the dataset

Increase Java virtual machine memory

ALTER TABLE OUTPUT COMMIT RECORD NUMBER

Optimization Results

---

mysql connection number optimization

       We all know that the default value of the maximum number of connections in MySQL is 100. This value is far from enough for the application of many databases with concurrent connections. When the connection request is greater than the default number of connections, there will be an error that the database cannot be connected, so we It needs to be scaled up appropriately. When using the MySQL database, we often encounter such a problem, which is "Can not connect to MySQL server. Too many connections"-mysql 1040 error, because the number of connections that have not been released to access MySQL has reached the upper limit of MySQL . Usually, the maximum number of mysql connections defaults to 100, and the maximum can reach 16384.

1、查看最大连接数
show variables like '%max_connections%';
2、修改最大连接数
set GLOBAL max_connections = 16384;

Kettle database insert operation optimization

The reason for the slowdown when the mysql table is output may be because the properties of the network link are set to
useServerPrepStmts=false  
rewriteBatchedStatements=true  
useCompression=true

Kettle allows multiple threads

We can use the distribution method to process data, please use this method with caution when it comes to database read and write operations.

Appropriately increase the size of the dataset

Increase Java virtual machine memory

ALTER TABLE OUTPUT COMMIT RECORD NUMBER

 

Optimization Results