http://zhaoyanblog.com/archives/753.html
Operating System and Process Section
The meaning of the operating system and the process part is very clear and will not be described in detail here. They list basic resource statistics such as CPU and load. The operating system part describes the situation of the entire operating system, and the process part just describes the usage of Elasticsearch's JVM process.
This is obviously a useful statistic, but is often overlooked. Some stats include the following:
> CPU
> Load
> Memory usage
> swap usage
> Open file handles
JVM section
The jvm section contains some key information about the jvm process running elasticsearch. Most importantly, it contains the details of garbage collection, which can have a big impact on the stability of your elasticsearch cluster.
> Getting Started with Garbage Collection (GC)
Before we describe this, it is necessary to introduce GC and its impact on elasticsearch. If you are familiar with GC in jvm, you can skip this chapter.
Java is a language that performs garbage collection by itself, which means that programmers do not need to actively manage the allocation and release of memory. Programmers only need to concentrate on writing their own code, the Java virtual machine manages the process of allocating memory as needed, and then when the memory is no longer in use, it will free itself.
When memory is allocated to a JVM process, it is allocated into a large area called the heap. The JVM divides this heap into two groups, called "generations":
Young Generation (or Garden of Eden)
Newly instantiated objects are allocated space here, and the young generation space is usually small, around 100MB-500MB. The young generation contains two survivor areas
old age
The area where those old objects are stored. These objects are long-lived and persist for a long time. The old generation is usually much larger than the young generation. You can see that the old generation of elasticsearch nodes can be as large as 30GB
When an object is instantiated, it is placed in the young generation, and when the space in the young generation is full, a young generation garbage collection is initiated. Those still alive are moved to one of the survivor areas. And dead objects are cleared. If an object survives multiple GCs in the young generation, it will be promoted to the old generation.
A similar process happens in the old generation. When the space in the old generation becomes more and more full, a garbage collection is started, and dead objects are cleaned up at the same time.
There is no free lunch in the world, and both young and old garbage collections include a "stop-the-world" phase. During this time, the JVM will stop the execution of the program, mark and collect objects. During this stop-the-world phase, nothing happens, the request will not be processed, and the ping will not be responded to. shards will no longer be migrated. The whole world really stopped.
For the young generation this is not a problem because it is small and the GC is performed very quickly. But for older generations, slow GC means pauses of 1s or even 15s, which is unacceptable for a server software.
Garbage collection is a very complex algorithm in the JVM, and a lot of work is done to reduce pauses. At the same time, Elasticsearch works hard to adapt to GC, such as through the reuse of internal objects, using network buffers, and expensive features such as the number of documents. But the frequency and length of GC is information that you need to pay special attention to, because it is the number one culprit of cluster instability.
If a cluster has frequent long-term GC, then your cluster must be out of memory and the load is particularly high. These long GCs cause nodes to periodically leave the cluster. This instability will cause the shard data to be continuously regenerated to ensure balance within the cluster and a sufficient number of shards. This increases network lending and disk IO, while your cluster is responsible for normal indexing data and queries.
In short, long GCs are bad and need to be minimized as much as possible.
Because GC is so important to Elasticsearch, you must be particularly familiar with this part of the node stats API display.
"jvm": {
"timestamp": 1408556438203,
"uptime_in_millis": 14457,
"mem": {
"heap_used_in_bytes": 457252160,
"heap_used_percent": 44,
"heap_committed_in_bytes": 1038876672,
"heap_max_in_bytes": 1038876672,
"non_heap_used_in_bytes": 38680680,
"non_heap_committed_in_bytes": 38993920,
The jvm section first lists general information about heap memory usage, you can see how much heap is used, how much can be used (already allocated threads), and how much heap memory can grow. Ideally heap_committed_in_bytes should be the same as heap_max_in_bytes. If the allocated heap is small, the JVM will have to adjust the size of the heap, which is expensive. If your two values are different, please see the "Heap: Sizing and Swapping" chapter to make sure you have configured them correctly.
heap_used_percent is a useful parameter you have to stare at. Elasticsearch is configured to perform GC when the heap is at 75%. If your node is always around 75%, then your node is under memory pressure, which is a warning that you will have a slow GC soon.
If your heap usage is always above 85%, then you are in trouble. There is a 90-95% probability of performance problems due to 10-30s GC. This is good, but the worst is memory overflow.
"pools": {
"young": {
"used_in_bytes": 138467752,
"max_in_bytes": 279183360,
"peak_used_in_bytes": 279183360,
"peak_max_in_bytes": 279183360
},
"survivor": {
"used_in_bytes": 34865152,
"max_in_bytes": 34865152,
"peak_used_in_bytes": 34865152,
"peak_max_in_bytes": 34865152
},
"old": {
"used_in_bytes": 283919256,
"max_in_bytes": 724828160,
"peak_used_in_bytes": 283919256,
"peak_max_in_bytes": 724828160
}
}
},
The young, survivor, and old sections show the GC usage of each generation for you to analyze. This data is convenient for you to see their relative sizes, but is often not very important for your investigation question.
gc": {
"collectors": {
"young": {
"collection_count": 13,
"collection_time_in_millis": 923
},
"old": {
"collection_count": 0,
"collection_time_in_millis": 0
}
}
}
The number and time of GC displayed in the gc area, including the young generation and the old generation. Most of the time, you can ignore the number of cell phones about the young generation, which tends to be large, which is normal.
On the contrary, the GC of the old generation should be less, and the collection_time_in_millis should be smaller. This is a cumulative number, it's hard to give you a number when you should be worried (for example, a node that has been running for a year, may have many GCs, but the node is healthy and stable). This is why some tools such as Maverl are particularly useful. The number of GCs over a period of time is an important consideration.
The time spent on GC is also important. For example, when indexing data, a certain amount of memory garbage is generated. This is normal and will cause GC to happen from time to time. These GCs are usually fast and not many heroes to the node. The young generation only takes a millisecond or two. The old generation can take hundreds of milliseconds. This is very different from a ten-second GC.
Our best suggestion is to collect the number and time of GC periodically (or use Marvell) and pay attention to frequent GC, you can also turn on the slow GC log and record it in the log.