This article was first published on the personal WeChat public account "andyqian", looking forward to your attention~
introduction
Children's shoes who are familiar with Java should be familiar with OOM. Such problems are generally more difficult. Because there are many reasons for such problems. Today I will share a very interesting case. (To be honest. I only encountered this problem two days ago. It took me a lot of time to analyze it. I found it very interesting and recorded it here. Don’t spray it!)
problem scenario
When the application is restarted, it is debugged to send a message to MQ. There are the following errors:
Exception in thread "Thread-4" java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717)
at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1378)
at org.apache.activemq.thread.PooledTaskRunner.wakeup(PooledTaskRunner.java:81)
at org.apache.activemq.transport.failover.FailoverTransport.reconnect(FailoverTransport.java:757)
at org.apache.activemq.transport.failover.FailoverTransport.start(FailoverTransport.java:344)
at org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:58)
at org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:58)At first, I thought it was caused by the system disk or full memory. View system information by topcommand. are within the normal range. While trying to find other solutions. Enter the command in SSH yes. The following error occurs:
$ bash: fork: Resource temporarily unavailableAfter the above problem occurs: a very obvious prompt is caused by insufficient resources.
After analysis: it is because the number of threads used by the current system is greater than the number of max_user_processes in ulimit.
Linux ulimit parameter
As mentioned above, it is related to the ulimit parameter in the system (Linux). ulimit is used to limit system resources.
These include:
-
max memory size.
-
open files (number of open files).
-
max user processes (the maximum number of user processes)
and so on.
System performance optimizations often optimize this parameter.
(Interested children's shoes can learn more by themselves)
ulimit common commands:
1. Display open files
$ ulimit -n
65535The result displayed by ulimit -n is: The value of open files.
2. Display the maximum number of processes for the current user
$ ulimit -u
60000The result displayed by ulimit -u is: max_user_processes value.
2. Display all properties of ulimit
$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 127399
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 65535
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 60000
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimitedin:
-
open files represents the maximum number of open files.
-
max user processes The maximum number of processes that can be opened by the current user.
(1) Modify the number of open files
The open files attribute can be /etc/security/limits.confmodified in the file. added at the end of the file. Code as shown below:
* soft nofile 65535
* hard nofile 65535in:
-
nofile means open files
-
The attributes corresponding to nproc are: max_user_processes
-
where 65535 is the demo number. Please adjust according to the actual situation of the environment.
(2) Modify the number of max_user_processes
The max_user_processes attribute can /etc/security/limits.d/90-nproc.confbe modified under the path.
Modify as follows:
* soft nproc 60000
root soft nproc unlimitedhave to be aware of is:
-
The maximum value in max_user_processes: 60000.
If the setting exceeds 60000, it defaults to the maximum value. For example: set 655535, it will be 60000. -
After performing the above steps. Exit the current session. Reconnect to take effect! Before modifying the session connected before, you need to reconnect to see the new configuration.
View System Resources Methods
1. View the number of processes used by the current system.
We can use the : ps aux|wc -lor ps -ef|wc -lcommand to see the number of processes currently in use.
As follows:
$ ps aux|wc -l
309Sometimes ps aux|wc -l the data obtained by the command view is less than max_user_processes the value.
Also appears:
$ bash: fork: Resource temporarily unavailable`This is because a process can contain multiple threads.
2. View the number of threads in the specified number of processes.
When we know the process ID (PID), we
can also check how many threads there are in the current process by the following method:
command:
$ cat /proc/<pid>/statusThe case is as follows:
andy@andyqian:/java$ cat /proc/11723/status
Name: java
State: S (sleeping)
Tgid: 11723
Ngid: 0
Pid: 11723
PPid: 2434
TracerPid: 0
Uid: 1000 1000 1000 1000
Gid: 1000 1000 1000 1000
FDSize: 128
Groups: 4 24 27 30 46 108 124 1000
Threads: 28
SigQ: 0/47456
Seccomp: 0
Cpus_allowed: ff
Cpus_allowed_list: 0-7
Mems_allowed: 00000000,00000001
Mems_allowed_list: 0
voluntary_ctxt_switches: 88
nonvoluntary_ctxt_switches: 3in:
-
Threads: 28 means that in the process, one has 28 threads.
-
Due to space reasons, some of the above information has been deleted.
3. How to view the process of the application There may be children's shoes who don't know how to view the number of processes of the application. Here is a simple case:
for example, view a file namedtomcat01 tomcat. We can check it with the following command:
ps -aux|grep tomcat01As follows:
andy@andyqian:/java$ ps -aux|grep tomcat01
andy 11723 0.5 1.0 6484560 127292 pts/1Among them: 11723 corresponds to the process ID of tomcat01.
Scan the code to follow and make progress together
Personal blog: http://www.andyqian.com