RabbitMQ
Producers lost data
producers to send data to RabbitMQ, when the data is probably halfway to engage in lost because of network problems or anything, is possible.
At this point you can choose to provide transactional capabilities with RabbitMQ, that is, the producers turned RabbitMQ Affairs channel.txSelect before sending data, then sends a message, if the message has not been received RabbitMQ to succeed, then the producer will receive an exception error, then you can roll back the transaction channel.txRollback, then retry sending the message; if you receive a message, you can commit the transaction channel.txCommit.
But the problem is, RabbitMQ transaction mechanism (synchronous) in a practice, basically throughput will come down, because the consumption of too much performance.
So, in general, if you want to make sure that write RabbitMQ messaging do not lose, you can turn confirm mode, set to open later confirm mode, every time you write a message will be assigned a unique id in the producers, and if written in RabbitMQ , RabbitMQ will give you a return ack message that tells you that the news ok. If you can not deal with this RabbitMQ message, the callback you a nack interfaces, to tell you the message reception fails, you can try again. And you can combine this mechanism to maintain their own state id for each message in memory, if more than a certain time has not received the news of callbacks, then you can re-issued.
The biggest difference affairs mechanisms and confirm the mechanism that transaction mechanism are synchronized, then you commit a transaction will be blocked there, but confirm the mechanism is asynchronous, after you send a message you can send the next message, then that message RabbitMQ reception after the asynchronous callback will be your interface to inform you that a message has been received.
This is generally the producer to avoid data loss, confirm the mechanism are used.
RabbitMQ lost data
previously stored data is automatically read RabbitMQ own lost data, you must turn on the persistent RabbitMQ, that is, after the message is written to be persisted to disk, even if it is RabbitMQ himself hung up, after restoration , general data will not be lost. Unless it is extremely rare, RabbitMQ not persistent, and that they hung up, it could result in a small amount of data loss, but the probability is small.
There are two steps to set persistence:
The first is to create a queue when it is set to persist, so as to ensure the metadata RabbitMQ persistent queue, but it is not persistent
queue in the data.
The second message is sent when the message is set to deliveryMode 2, the message is provided for persistent, in which case RabbitMQ
will be persisted to disk up message.
Both must be set to persist for the job, RabbitMQ even if it is hung up, reboot again will restart recovery queue from disk, restore the data in the queue at the same time.
Note that even if you give RabbitMQ open the persistence mechanism, there is also a possibility that the news wrote in RabbitMQ, but not enough time persisted to disk, the results unfortunately, at this time RabbitMQ hung up, it will lead to memory in a little bit of data loss.
Therefore, the persistence mechanism can cooperate with producers confirm the up side, the message is only after the disk, will inform the producer ack persistence, even before it is persisted to disk, RabbitMQ hung, data lost producers can not receive ack, you also can own retransmission.
Consumer side lost data
RabbitMQ If you have lost data, mainly because when you consume, just to consume, not processed and the results linked to the process, such as restart, then embarrassed, and RabbitMQ think you are a consumer, this data is lost.
ack mechanism this time starting RabbitMQ provides, in simple terms, is that you have to turn off the automatic ack RabbitMQ can be called via an api on the line, then each time your own code to ensure that when processed, and then in the program a ack the. In this case, if you have not been processed, not on the no ack? That RabbitMQ think you have not been processed, this time the consumer will RabbitMQ assigned to another consumer to deal with, the message is not lost.
Kafka
consumer side lost data is
the only possible result in the case of lost consumer data, is that you consume to this message, and then automatically submit the consumer side offset, let Kafka thought you were a consumer good news, but in fact you just ready to deal with this news, you have not treated yourself hung up, then this message is thrown slightly.
It's not almost like RabbitMQ it, we all know that Kafka will automatically submit offset, you can turn off auto-commit offset, offset to manually submit after processing, can ensure that the data will not be lost. But this time may indeed still have a repeat consumption, such as you have just processed, not submitting offset, the result himself hung up, then consumption will certainly be repeated once promised myself idempotency just fine.
One problem encountered in a production environment, that is to say after our Kafka consumer spending data is written to the queue in a first buffer memory, the results sometimes, you just had a message written to the memory queue, then consumers will automatically submit offset. Then we restart the system at this time, it will cause data in memory queue not had time to deal with the lost.
Kafka lost data
piece of a more common scenario is that Kafka a broker goes down, then the re-election of the leader partition. Think about it, after this time if the other follower just some data is not synchronized, the result this time leader hung up, and then the election of a follower into a leader, not less some data? It lost some data ah.
Production environments encountered, we also, before Kafka's leader machine goes down, then the follower is switched to the leader, you will find this to say on the lost data.
At this time, it is usually the minimum requirement set the following four parameters:
- Replication.factor parameter set to topic: This value must be greater than one, it requires that each partition must have at least two copies.
- Set min.insync.replicas parameters Kafka server: This value must be greater than 1, this requires a leader is at least perceived to have at least one follower was kind enough to keep in touch with yourself, not left behind, so as to ensure that the leader and a follower hang of it .
- Set acks the producer end = all: this is to ask each of the data must be written after all the replica, believed to be written in order to succeed.
- In the producer side setting retries = MAX (a lot of very big value, meaning infinite retries): This is the required write once failed, would infinitely retry card here.
After such configuration, at least at the end of Kafka broker can guarantee when the leader is located broker has a fault leader handover, data is not lost.
Producers will not lose data?
If you set up in accordance with the above ideas acks = all, must not be lost, the requirement is that your leader a message is received, all of the follower are synchronized to the news only after considers that the write succeeded. If you do not meet this condition, the producer will automatically continue to retry, retry unlimited.