Problem
#1 mysql query is too large Solution:
https://www.jianshu.com/p/629685b97030
MySQL will limit the packet size accepted by the Server according to the configuration file. Sometimes large inserts and updates are limited by the max_allowed_packet parameter, causing large data writes or updates to fail.
You can edit my.cnf and modify it in the [mysqld] section or the server configuration section of mysql.
max_allowed_packet = 200M
Modify my.cnf, the configuration must be reloaded to take effect
#2 Use jvm memory viewing and analysis tool jviusalvm
https://blog.csdn.net/yaowj2/article/details/7107818
#2.1 Check jdk configuration
vim /etc/ profile
export JAVA_HOME=/usr/java/jdk1.7.0_80
export CLASSPATH=.:${JAVA_HOME}/jre/lib/rt.jar:${JAVA_HOME}/lib/dt.jar:${JAVA_HOME}/lib/tools .jar
export PATH=$PATH:${JAVA_HOME}/bin
source /etc/profile
#2.2 JVM memory optimization and adjustment
common parameter types (configuration memory)
configuration heap area: -Xms, -Xmx, -XX:newSize, -XX:MaxnewSize, -Xmn
configuration non-heap area: -XX:PermSize, -XX:MaxPermSize
heap Area parameter configuration
1, -Xms: Indicates the initial memory allocation size of the java virtual machine heap area, which is usually 1/64 of the available memory of the operating system, but still needs to be allocated according to the actual situation. It is possible that when such a rule is actually allocated, the designed software hangs before it can run.
2. -Xmx: Indicates the maximum upper limit of the java virtual machine heap memory that can be allocated, usually 1/4 of the available memory of the operating system. However, during the development process, the two parameters -Xms and -Xmx are usually configured with the same value. The purpose is to waste resources without re-partitioning the heap area after the java garbage collection mechanism cleans up the heap area.
Generally speaking, for memory allocation in the heap area, you only need to configure the above two parameters reasonably.
Back to top
Non-heap area parameter configuration
1. -XX:PermSize: Indicates the initial memory allocation size of the non-heap area, which is abbreviated as permanent size (persistent memory)
2. -XX:MaxPermSize: Indicates the size of the memory allocated for the non-heap area Max Cap
Bet:
Before configuring, you must carefully consider the size of the non-heap memory required by your own software, because the memory here will not be processed by the java garbage collection mechanism. And it is even more important to note that the sum of the maximum heap memory and the maximum non-heap memory must not exceed the available memory of the operating system.
Reason for reference: If 98% of the time in the JVM is used for GC and is available, this exception message will be thrown when the Heap size is less than 2%.
The setting of the JVM heap refers to the setting of the memory space that the JVM can allocate and use during the running of the java program. The JVM will automatically set the value of the Heap size when it starts, and its initial space (ie -Xms) is 1/64 of the physical memory. The maximum space (-Xmx) is 1/4 of the physical memory.
Options such as -Xmn -Xms -Xmx provided by the JVM can be used for setting. The maximum Heap Size should not exceed 80% of the available physical memory. Generally, the -Xms and -Xmx options should be set to the same value, and -Xmn is 1/4 of the -Xmx value.
The -Xms -Xmn settings of Heap size should not exceed the size of physical memory. Otherwise, "Error occurred during initialization of VM Could not reserve enough space for object heap" will be prompted.
# Java heap space heap memory OOME exception log
[QuartzScheduler.java:2425] - Job (DEFAULT.J300 threw an exception.
org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: java.lang.OutOfMemoryError: Java heap space]
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Caused by: java.lang.OutOfMemoryError: Java heap space
[ERROR] 2018-07-04 15:19:58,861 [JobRunShell.java:211] - Job DEFAULT.J300 threw an unhandled Exception:
java.lang.OutOfMemoryError: Java heap space
[ERROR] 2018-07-04 15:19:58,861 [QuartzScheduler.java:2425] - Job (DEFAULT.J300 threw an exception.
org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: java.lang.OutOfMemoryError: Java heap space]
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Caused by: java.lang.OutOfMemoryError: Java heap space
[ERROR] 2018-07-04 15:27:41,400 [JobRunShell.java:211] - Job DEFAULT.J300 threw an unhandled Exception:
java.lang.OutOfMemoryError: Java heap space
[ERROR] 2018-07-04 15:27:41,400 [QuartzScheduler.java:2425] - Job (DEFAULT.J300 threw an exception.
org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: java.lang.OutOfMemoryError: Java heap space]
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Caused by: java.lang.OutOfMemoryError: Java heap space
[ERROR] 2018-07-04 15:36:03,004 [JobRunShell.java:211] - Job DEFAULT.J300 threw an unhandled Exception:
java.lang.OutOfMemoryError: Java heap space
[ERROR] 2018-07-04 15:36:03,004 [QuartzScheduler.java:2425] - Job (DEFAULT.J300 threw an exception.
org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: java.lang.OutOfMemoryError: Java heap space]
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Caused by: java.lang.OutOfMemoryError: Java heap space
[ERROR] 2018-07-04 15:45:34,912 [JobRunShell.java:215] - Job DEFAULT.J300 threw an unhandled Exception:
java.lang.OutOfMemoryError: Java heap space
[ERROR] 2018-07-04 15:45:34,912 [QuartzScheduler.java:2425] - Job (DEFAULT.J300 threw an exception.
org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: java.lang.OutOfMemoryError: Java heap space]
at org.quartz.core.JobRunShell.run(JobRunShell.java:215)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Caused by: java.lang.OutOfMemoryError: Java heap space
[ERROR] 2018-07-04 15:52:44,044 [JobRunShell.java:215] - Job DEFAULT.J300 threw an unhandled Exception:
java.lang.OutOfMemoryError: Java heap space
[ERROR] 2018-07-04 15:52:44,044 [QuartzScheduler.java:2425] - Job (DEFAULT.J300 threw an exception.
org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: java.lang.OutOfMemoryError: Java heap space]
at org.quartz.core.JobRunShell.run(JobRunShell.java:215)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Caused by: java.lang.OutOfMemoryError: Java heap space
# MYSLQ query is too large log
Packet for query is too large (114006052 > 62914560). You can change this value on the server by setting the max_allowed_packet' variable.
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3583)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2464)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2639)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2551)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1864)
at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1194)
at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172)
at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172)
at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.ibatis.logging.jdbc.PreparedStatementLogger.invoke(PreparedStatementLogger.java:60)
at com.sun.proxy.$Proxy16.execute(Unknown Source)
at org.apache.ibatis.executor.statement.PreparedStatementHandler.update(PreparedStatementHandler.java:41)
#1 mysql query is too large Solution:
https://www.jianshu.com/p/629685b97030
MySQL will limit the packet size accepted by the Server according to the configuration file. Sometimes large inserts and updates are limited by the max_allowed_packet parameter, causing large data writes or updates to fail.
You can edit my.cnf and modify it in the [mysqld] section or the server configuration section of mysql.
max_allowed_packet = 200M
Modify my.cnf, the configuration must be reloaded to take effect
#2 Use jvm memory viewing and analysis tool jviusalvm
https://blog.csdn.net/yaowj2/article/details/7107818
#2.1 Check jdk configuration
vim /etc/ profile
export JAVA_HOME=/usr/java/jdk1.7.0_80
export CLASSPATH=.:${JAVA_HOME}/jre/lib/rt.jar:${JAVA_HOME}/lib/dt.jar:${JAVA_HOME}/lib/tools .jar
export PATH=$PATH:${JAVA_HOME}/bin
source /etc/profile
#2.2 JVM memory optimization and adjustment
common parameter types (configuration memory)
configuration heap area: -Xms, -Xmx, -XX:newSize, -XX:MaxnewSize, -Xmn
configuration non-heap area: -XX:PermSize, -XX:MaxPermSize
heap Area parameter configuration
1, -Xms: Indicates the initial memory allocation size of the java virtual machine heap area, which is usually 1/64 of the available memory of the operating system, but still needs to be allocated according to the actual situation. It is possible that when such a rule is actually allocated, the designed software hangs before it can run.
2. -Xmx: Indicates the maximum upper limit of the java virtual machine heap memory that can be allocated, usually 1/4 of the available memory of the operating system. However, during the development process, the two parameters -Xms and -Xmx are usually configured with the same value. The purpose is to waste resources without re-partitioning the heap area after the java garbage collection mechanism cleans up the heap area.
Generally speaking, for memory allocation in the heap area, you only need to configure the above two parameters reasonably.
Back to top
Non-heap area parameter configuration
1. -XX:PermSize: Indicates the initial memory allocation size of the non-heap area, which is abbreviated as permanent size (persistent memory)
2. -XX:MaxPermSize: Indicates the size of the memory allocated for the non-heap area Max Cap
Bet:
Before configuring, you must carefully consider the size of the non-heap memory required by your own software, because the memory here will not be processed by the java garbage collection mechanism. And it is even more important to note that the sum of the maximum heap memory and the maximum non-heap memory must not exceed the available memory of the operating system.
Reason for reference: If 98% of the time in the JVM is used for GC and is available, this exception message will be thrown when the Heap size is less than 2%.
The setting of the JVM heap refers to the setting of the memory space that the JVM can allocate and use during the running of the java program. The JVM will automatically set the value of the Heap size when it starts, and its initial space (ie -Xms) is 1/64 of the physical memory. The maximum space (-Xmx) is 1/4 of the physical memory.
Options such as -Xmn -Xms -Xmx provided by the JVM can be used for setting. The maximum Heap Size should not exceed 80% of the available physical memory. Generally, the -Xms and -Xmx options should be set to the same value, and -Xmn is 1/4 of the -Xmx value.
The -Xms -Xmn settings of Heap size should not exceed the size of physical memory. Otherwise, "Error occurred during initialization of VM Could not reserve enough space for object heap" will be prompted.
# Java heap space heap memory OOME exception log
[QuartzScheduler.java:2425] - Job (DEFAULT.J300 threw an exception.
org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: java.lang.OutOfMemoryError: Java heap space]
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Caused by: java.lang.OutOfMemoryError: Java heap space
[ERROR] 2018-07-04 15:19:58,861 [JobRunShell.java:211] - Job DEFAULT.J300 threw an unhandled Exception:
java.lang.OutOfMemoryError: Java heap space
[ERROR] 2018-07-04 15:19:58,861 [QuartzScheduler.java:2425] - Job (DEFAULT.J300 threw an exception.
org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: java.lang.OutOfMemoryError: Java heap space]
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Caused by: java.lang.OutOfMemoryError: Java heap space
[ERROR] 2018-07-04 15:27:41,400 [JobRunShell.java:211] - Job DEFAULT.J300 threw an unhandled Exception:
java.lang.OutOfMemoryError: Java heap space
[ERROR] 2018-07-04 15:27:41,400 [QuartzScheduler.java:2425] - Job (DEFAULT.J300 threw an exception.
org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: java.lang.OutOfMemoryError: Java heap space]
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Caused by: java.lang.OutOfMemoryError: Java heap space
[ERROR] 2018-07-04 15:36:03,004 [JobRunShell.java:211] - Job DEFAULT.J300 threw an unhandled Exception:
java.lang.OutOfMemoryError: Java heap space
[ERROR] 2018-07-04 15:36:03,004 [QuartzScheduler.java:2425] - Job (DEFAULT.J300 threw an exception.
org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: java.lang.OutOfMemoryError: Java heap space]
at org.quartz.core.JobRunShell.run(JobRunShell.java:213)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Caused by: java.lang.OutOfMemoryError: Java heap space
[ERROR] 2018-07-04 15:45:34,912 [JobRunShell.java:215] - Job DEFAULT.J300 threw an unhandled Exception:
java.lang.OutOfMemoryError: Java heap space
[ERROR] 2018-07-04 15:45:34,912 [QuartzScheduler.java:2425] - Job (DEFAULT.J300 threw an exception.
org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: java.lang.OutOfMemoryError: Java heap space]
at org.quartz.core.JobRunShell.run(JobRunShell.java:215)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Caused by: java.lang.OutOfMemoryError: Java heap space
[ERROR] 2018-07-04 15:52:44,044 [JobRunShell.java:215] - Job DEFAULT.J300 threw an unhandled Exception:
java.lang.OutOfMemoryError: Java heap space
[ERROR] 2018-07-04 15:52:44,044 [QuartzScheduler.java:2425] - Job (DEFAULT.J300 threw an exception.
org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: java.lang.OutOfMemoryError: Java heap space]
at org.quartz.core.JobRunShell.run(JobRunShell.java:215)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Caused by: java.lang.OutOfMemoryError: Java heap space
# MYSLQ query is too large log
Packet for query is too large (114006052 > 62914560). You can change this value on the server by setting the max_allowed_packet' variable.
at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3583)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2464)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2639)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2551)
at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1864)
at com.mysql.jdbc.PreparedStatement.execute(PreparedStatement.java:1194)
at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172)
at org.apache.commons.dbcp.DelegatingPreparedStatement.execute(DelegatingPreparedStatement.java:172)
at sun.reflect.GeneratedMethodAccessor59.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.ibatis.logging.jdbc.PreparedStatementLogger.invoke(PreparedStatementLogger.java:60)
at com.sun.proxy.$Proxy16.execute(Unknown Source)
at org.apache.ibatis.executor.statement.PreparedStatementHandler.update(PreparedStatementHandler.java:41)