Business needs:
It is necessary to send advertisements, questionnaires and other emails to registered members on a regular basis, and millions of emails need to be sent every day.
Server:
Several postfix servers
Problems encountered:
For millions of mail sending tasks per day, the postfix server is easy to complete, and the Java client includes the business logic processing time when sending, and it is easy to do with a few more threads.
However, if the sending speed is too fast, the mail service provider will be blacklisted and reject the email, and each mail service provider has different requirements for the speed of receiving emails.
Solution:
In this way, you need to send tasks in batches for each email, and send emails at a speed that each email service provider can receive.
For example, 163 3 per second, sina 4 per second, etc.
So we designed an email sending system with the following structure:
Use Redis to store the last sending time of each domain name,
repeat a, b
a) to obtain and lock the domain name that can be sent.
b) For each domain name that can be sent,
1. If there is an email to be sent in the corresponding domain name queue, send and update the last sending time of the domain name change.
2. Unlock the domain name sending record.
In this way, a non-blocking queue is used for the mail queue, which improves the concurrency of the system.
It seems to solve the problem, but it is still blacklisted by some service providers.
Therefore, the system is analyzed as follows:
This is a typical producer-consumer problem. The mail produced by the Java client is pushed to the Postfix queue regularly, and then Postfix consumes the mail on the queue and sends it to the target mail server.
If you produce one and send one, there will be no problem, but if there are multiple messages in the queue, the speed of delivery to the target mail server cannot be controlled.
So check the postfix queue:
postqueue -p
Found that some mails did pile up.
Reason: When the system is running, the delay in sending a certain email will cause the sending of subsequent emails to be delayed.
Disposal method:
Controlling the sending speed on the Java side is obviously a cure for the symptoms but not the root cause. Therefore, the same logic on the Java side is used to modify the mail sending code in PostFix, and the
same configuration file is used to fundamentally solve the problem of controlling the sending speed of the mail. question.