Preface
Only a bald head can become stronger.
The text has been included in my GitHub repository, welcome to Star: https://github.com/ZhongFuCheng3y/3y
Encountered a problem sending emails online, record it.
One, first talk about the background
One day, Xiao Wang gave me a feedback: "Thank you to check the sending status of the online email, I found out that the sending failed."
I went to DB to check the recent mail sending situation, and said: "It looks normal, there is no abnormal situation online. Maybe the mail has accumulated in redis and has not been consumed yet."
select * from email order by id desc limit 100
Let me first talk about how I send emails roughly:
How to handle email messages
What are the benefits of doing this? Think of Redis as a message queue, and throw all requests on Redis, which can cut peaks . Machine A / B / C of the thread in certain intervals to take a message to Redis pull, and then calls the message transmission interface.
And I will provide a function on the page for the business side to check whether various messages are sent successfully, because sending emails is an asynchronous operation, and former colleagues pursue real-time when writing .
The current logic is: if the push to Redis is successful, and there are no messages accumulated in Redis (indicating that the machine A/B/C can process the email in time), then the email is considered to be sent successfully.
PS: (If there is no problem in the system, this implementation is actually OK. Because the amount of mail sent is generally not too large (Redis does not accumulate messages), and the success rate of sending mail is quite high .
Going back to the question, because of the background above, I guessed whether this email was still piled up on Redis when Xiao Wang was checking the results, so it just returned directly and failed. Sure enough, I checked Redis, and there were still 200 emails without news.
So I asked Xiao Wang: "How many emails have you sent?" Xiao Wang said: "500 in 20 minutes, less than 1qps." I thought for a moment: "Then we have four machines, and logically, there won't be so many piles up."
So I went to the online server to look at the consumption log and found that only one machine was consuming Redis data. I went to check the error log again to see if there were a lot of error messages, but no error log was found...
So I checked the monitoring information of the machine and found nothing unusual. So the question is: Why is there only one machine consuming Redis messages? The logs and monitoring information of the other three machines are normal.
Two, solve
From the log and machine information, I couldn’t tell what was wrong. Then I remembered a command in Java:jstack
The jstack command is mainly used to view the call stack of a Java thread and can be used to analyze thread problems (such as deadlock).
jstack detailed usage and tutorial:
https://www.cnblogs.com/kongzhongqijing/articles/3630264.html
So I executed a jstackcommand, searched for "Email" in the information, and I found it out:
Stuck email
That's easy, just search for keywords like "Java sending mailbox thread blocked" and there should be a solution.
solution
Finally, it was found that the timeout period was not configured when sending mail , which caused some threads to be blocked when sending mail (the specific reason is unknown)
mail.smtp.connectiontimeout : connection time limit, in milliseconds. It is about the length of time to establish a connection with the mail server. The default is unlimited.
mail.smtp.timeout : time limit for mail receiving, in milliseconds. This is about the length of time the mail is received. The default is unlimited.
mail.smtp.writetimeout : time limit for sending mail, in milliseconds. Regarding the length of time the content is uploaded when sending the email. The default is also unlimited.
Article navigation of the official account : navigation of all articles of the official account
Long press to scan the code to follow to get