A blog post about the Redis foreigners is mentioned an interesting thing: they delay map obtained during the test. For persistent Redis data to disk (for example: RDB persistence), Redis need to call fork () system command.
Most commonly used physical servers and virtual machine management program fork is very fast, even though a lot of progress as well. However, Xen's fork () very slow, so (providers and other virtual servers) For some EC2 instance types, each parent process calls fork () for persistence when RDB, there may be significant delays peak. Shown below clearly show the peak delay:
You can imagine, if you fork () when doing a delay test, then the parent process fork () when all requests will be delayed one second (more than figure as an example). This will produce a large number of samples with high latency, and 99% will affect the results.
Example To change the type, configuration, or any other content provided to improve this behavior is a good idea, and some even have a single request delay is unacceptable excessive use cases. However, it is obvious that the peak delay occurs 1 second every 30 minutes is not obvious, because the delay in the request with a uniform distribution peak are very different.
If the peak is evenly distributed, access to a page if you need to perform a large number of requests for Redis server, then access the page are likely to encounter delays: it would seriously affect the quality of service.
However, as shown above, the delay of one second every 30 minutes of operation are completely different things. A good percentage of delayed performance increases with the number of requests and get better, because the more requests, the less likely this delay over-represented in the sample, it will be hidden. If you only have one request per minute, and in which a request is just across fork () due to high latency, it will delay the test results are very ugly.
Also: most page views are not affected. Because the only few users encountered one second delay is just their request and fork () at the same time, other users will only have a very low probability request encountered such a bad thing. Also note that the fork () page views hit (even by the 100-requests) will not be delayed for more than one second, because the fork () after the completion of the request will be completed, we do not need to wait until the completion of RDB persistence.
Only fork () will cause a delay glitches, fork out the child in the process of generating RDB file does not have a great impact on the system. Unless the child process generates RDB files (this procedure uses a copy-on-write mechanism operating systems) have a lot to write, but not much memory available to the server, this time may be swapping occurs that causes a delay.
EC2 instances in the runtime environment of today's most popular, fork delay is one of the worst user experience Redis, so redis author is working to test Redis and EC2: I believe there will soon be specifically optimized for EC2 in Redis official document the explanation, that time there will be more secure than disabling program persistence operations in the master-slaves.
If you now need to EC2 + Redis host and disabled persistence, the easiest deployment is to disable automatic restart Redis instance, and use the Sentinel failover, the host server to crash does not automatically return to a usable state. After checking the failover is successful and there are new available master, the system administrator can manually restart the instance.
It should be noted that not all EC2 instances are the same, on the contrary, various EC2 instance fork still a big difference in performance. Shown below, it is a foreigner to do some tests:
to sum up
- Called when Redis persistence fork can cause serious delays (RDB persistence and AOF rewrite calls the fork command);
- Just cause delay, after the fork is completed, the child process generates RDB / AOF files when the fork process does not cause delays;
- Even Redis single-threaded processing command, but the best server configuration requires dual core, Redis if there are multiple instances, then the corresponding number of double core;
- In addition to the outer fork can cause delay, the Redis those time complexity of O (n) performance can also cause glitches, and other command keys, for example.