I found data loss from report A, and then found the program running log. I found that there was only a small part of the log, usually 12G, but today it was only 243M.
However, it is found in Report B that the program has completed running. So it can be inferred that the program is running, but the logs are missing.
Find relevant information based on the proc file system
Found several log files
Open the file and find that it is indeed the file lost today. Tail -f the file and find that there is a thread waiting for it.
A deadlock occurred in the thread in the description file.
Use the Java memory analysis command (jstack 5669) to analyze the memory: it is found that the SocksSocketImpl service is locked
Take a look at where the exception occurred before the log was lost: there are also exceptions caused by sockets
Therefore, this exception needs to be solved, because this exception causes another child process to be in a deadlock state.
It can be found that an exception occurred in an object in the memory, but this exception was not well captured, causing the thread to remain in a deadlock state.
Finally, looking at the source code, we found that waiting is mainly because the variable aliveMutThreadNum does not perform atomic operations.
As shown below: Suoran uses locking when operating the aliveMutThreadNum variable, but synchronized is still not the safest form, because
1 synchronized is generally used to obtain class locks
2 Although the method is synchronized, if the method is not used in the appropriate place, the result will still be a deadlock. This deadlock is because aliveMutThreadNum was not processed in finally when catching the exception. Although the relevant method was called, the deadlock was still caused because the program did not execute to this step.
The correct approach should be to use the volatile keyword for modification: and process it in the finally of try catche finally.
In addition: 1. It is best not to write the program counter like this. It is best to use the countdownlatch class for processing.
2. If a deadlock occurs, you should be able to manually command Java to skip the deadlock part and continue executing other programs. You can use hotswap to achieve this purpose.