Background: The use -ano netstat | grep ESTABLISH | grep 11883 monitoring send_Q value and recv_Q, since the amount of data a single push of more than 1w bar, resulting in falcon-agent process, however, result in loss of data, graphical breakpoint.
1. Why falcon graphic will break?
There are the following two reasons:
- tcp disconnected, resulting in no current monitoring point value.
- Loss of data, point blank period data is not uploaded.
2, verification of data loss
- Find a long connector on the machine monitor, and packed to the log file with the script, and graphics than the loss of falcon-point connection exists. After the actual verification, we found that the data is really lost.
- test.sh
- result:
3, why the data will be lost?
- Because our connections between 5k ~ 1w,
- The value per minute, and beat up the two indicators,
This is what we will push every minute 1w ~ 2w of data to the falcon,
The total single-threaded execution took about 37s,
The total cost of multi-threaded execution time 0s
At first we suspected to be the cause of slow single-threaded execution, worried about the data is not complete insert was discarded, so the use of elongated push time, instead try 2/3/5 per minute, the result is failure.
Later attempts to change multithreading, but the problem remains the same, and later through discussions and consult other colleagues suspect that is not because a short time to push too much data (according to colleagues provide historical single push in 2000, there is no data loss), leading to falcon -agent to handle, however, so that part of the data discarded.
So based on the original, to cut data push, push batches (500 / List) + delay (20ms, avoid multi-process execution too fast), control data push within the total time spent in the 15s, the perfect solution data loss problem.
to sum up
The reason: a short time push data overload
Solution: batches push data delay +
Additional knowledge:
1⃣️send_Q what represents recv_Q?
https://stackoverflow.com/questions/36466744/use-of-recv-q-and-send-q?noredirect=1&lq=1
https://milestone-of-se.nesuke.com/sv-basic/linux-basic/ss-netstat/
2⃣️ What is tcp three-way handshake?