Micro-channel public number: "Little sister flavor", attached to the end the two-dimensional code, welcome attention.
Server environment using the root account to run the application is very dangerous, easy to get the shell becomes broiler. So little sense of team, will establish a low-rights ordinary users to run java program.
Low-privilege, a bit like his own son, in a tense moment, especially resource constraints can see it.
phenomenon
The problem is a common occurrence in a test environment machines, no formal environment and reproduce. This server deployed dozens of services and the deployment of accounts from recently rootswitched to xjjbot.
After running for some time, the server problem occurs often. First, a large number of connections in the CLOSE_WAIT state, once thought it was a passive closed issue. But it is not.
netstat -antp | grep CLOSE | awk '{print $7}' | sort | uniq -c
The strange thing is, the use of rootaccount or other account login system, operating as usual. However, when switched to the xjjbotaccount will be reported the following error:
# sudo su - xjjbot
bash: fork: retry: no child processes
bash: fork: retry: no child processes
bash: fork: retry: no child processes
bash: fork: retry: no child processes
bash: fork: Resource temporarily unavailable
These are system-level error message. In this case, jvm will be a corresponding error, but I'm afraid you do not have a chance to see (you can use other system users to view oh).
- Cannot create GC thread. Out of system resources
- java.lang.OutOfMemoryError: unable to create new native thread
the reason
The cause is not enough resources, specifically the process resources.
Linux thread is actually a process, so java is also, in particular, called "light weight process (LWP)" - lightweight processes.
LWP shared with other processes all (or most of) the logical address space and system resources, a process can create multiple LWP, so that they share most of the resources; LWP has its own process identifier, and other processes and have a parent-child relationship; . LWP and as ordinary as scheduled by the kernel process management
Use the following command to see how much a user process resource
ps -eLf | grep xjjbot(uid) | wc -l
Use the following command to view specific for each process opens up how many threads
ps -o nlwp,pid,lwp,args -u xjjbot(uid) | sort -n
solve
Everything is according to the rules linux file, the first thought is to modify the parameters ulimit, but is not, because it is already big enough. Cross Recall elasticsearch, at the time of installation, you need to configure a known nprocthing, the problem probably lies in this, are not enough resources to process it.
Related configuration files:
/etc/security/limits.conf
On a different version of the kernel, there are some small differences. For example
/etc/security/limits.d/*
file, it will cover limits.conf configuration at some point. Therefore, the case does not take effect, remember to check.
For these reasons, the configuration can be limits.d all commented, unified configuration in limits.conf in.
The following is the original configuration
* soft nproc 4096
root soft nproc unlimited
4096 will be replaced by large numbers point, or directly into unlimited on it.
ElasticSearch system parameter configuration
As mentioned in the es, es then we look at what is required to install the system to change the configuration. These experiences are common, you can draw inferences.
https://www.elastic.co/guide/en/elasticsearch/reference/master/setting-system-settings.html
Disable swap
swap is a performance killer, so ES also could not bear, directly off.
sudo swapoff -a
In the configuration file can be added to this parameter, jvm locked memory, and let them swap exchange.
bootstrap.memory_lock: true
Virtual Memory
ES used mmapfsto map some of the data, but the default system parameters it is too small, need to be modified.
sysctl -w vm.max_map_count=262144
Permanent need to modify /etc/sysctl.conf
File handle
ulimit
The number of open file descriptors linux is limited. If your application needs and at the same time dealing with many small files, you need to configure this parameter.
sudo su
ulimit -n 65536
su elasticsearch
/etc/security/limits.conf
ok, this is what we have just changed files. To the above configuration permanent, you need to change this file.
elasticsearch - nofile 65536
The number of threads
That is on top of it we can quickly think of it, but also because es installed -.-
So, it should not open a lot of threads, in addition to increasing the scheduled time, but also easy to top-to-ceiling system.
Under von Neumann architecture, the software, you do not have a routine?
He had the same fate, struggling but can not escape.