In order to avoid losing information after unexpected downtime, it is necessary to restore the message queue after restarting, and the message system generally adopts a persistence mechanism.
The default is KahaDB, which is an embeddable transactional persistence mechanism.
ActiveMQ's message persistence mechanisms include JDBC, AMQ, KahaDB, and LevelDB. No matter which persistence method is used, the message storage logic is consistent.
That is, after the sender sends the message, the message center first stores the message in a local data file, an in-memory database or a remote database, etc., and then tries to send the message to the receiver. If the message is successfully sent, the message will be deleted from the storage, and if it fails, it will continue. try.
After the message center is started, the specified storage location must be checked first. If there is a message that has not been sent successfully, the message needs to be sent out.
1. JDBC persistence method
Using JDBC persistence, the database will create three tables: activemq_msgs, activemq_acks and activemq_lock.
activemq_msgs is used to store messages, Queue and Topic are stored in this table.
(1) Configuration method
The way to configure persistence is to modify the conf/acticvemq.xml file in the installation directory.
First define a mysql-ds MySQL data source, then configure jdbcPersistenceAdapter in the persistenceAdapter node and reference the data source just defined.
<persistenceAdapter>
<jdbcPersistenceAdapter dataSource="#mysql-ds" createTablesOnStartup="false" />
</persistenceAdapter>
dataSource specifies the bean of the persistent database. Whether createTablesOnStartup creates a data table at startup? The default value is true, so that the data table will be created every time it starts. Generally, it is set to true when it is started for the first time, and then changed to false. .
Configure JDBC persistence with MySQL:
<beans>
<broker brokerName="test-broker" persistent="true" xmlns="http://activemq.apache.org/schema/core">
<persistenceAdapter>
<jdbcPersistenceAdapter dataSource="#mysql-ds" createTablesOnStartup="false"/>
</persistenceAdapter>
</broker>
<bean id="mysql-ds" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
<property name="driverClassName" value="com.mysql.jdbc.Driver"/>
<property name="url" value="jdbc:mysql://localhost/activemq?relaxAutoCommit=true"/>
<property name="username" value="activemq"/>
<property name="password" value="activemq"/>
<property name="maxActive" value="200"/>
<property name="poolPreparedStatements" value="true"/>
</bean>
</beans>
(2) Database table information
activemq_msgs is used to store messages, Queue and Topic are stored in this table:
ID: self-incrementing database primary key
CONTAINER: message destination
MSGID_PROD: message sender client primary key
MSG_SEQ: is the order in which messages are sent, MSGID_PROD+MSG_SEQ can form JMS MessageID
EXPIRATION: The expiration time of the message, which stores the number of milliseconds from 1970-01-01 to the present
MSG: The binary data of the Java serialized object of the message body
PRIORITY: Priority, from 0-9, the larger the value, the higher the priority higher
activemq_acks is used to store subscription relationships. If it is a persistent topic, the subscription relationship between subscribers and servers is stored in this table:
The main database fields are as follows:
CONTAINER: Destination of the message
SUB_DEST: If Static cluster is used, this field will contain information about other systems in the cluster
CLIENT_ID: Each Subscribers must have a unique client ID to distinguish
SUB_NAME: subscriber name
SELECTOR: selector, you can choose to consume only messages that meet the conditions. Conditions can be implemented with custom attributes, which can support multi-attribute AND and OR operations
LAST_ACKED_ID: record the ID of the consumed message.
The table activemq_lock is only useful in a cluster environment. Only one Broker can get the message, which is called the Master Broker.
The others can only be used as backups and wait for the Master Broker to be unavailable before becoming the next Master Broker.
This table is used to record which Broker is the current Master Broker.
2. AMQ method
The performance is higher than that of JDBC. When writing a message, the message will be written to the log file. Since it is written sequentially, the performance is very high. To improve performance, create a message primary key index and provide a caching mechanism to further improve performance. The size of each log file is limited (the default is 32m, which can be configured by yourself).
When this size is exceeded, the system will recreate a file. When all messages are consumed, the system will delete the file or archive it (depending on configuration).
The main disadvantage is that AMQ Message will create an index for each Destination. If a large number of Queues are used, the size of the index file will occupy a lot of disk space.
And because the index is huge, once the Broker crashes, rebuilding the index will be very slow.
The configuration snippet is as follows:
<persistenceAdapter>
<amqPersistenceAdapter directory="${activemq.data}/activemq-data" maxFileLength="32mb"/>
</persistenceAdapter>
3. KahaDB way
Although the performance of AMQ is slightly higher than the Kaha DB method below, it is not recommended because it takes too long to rebuild the index and the index file occupies too much disk space.
KahaDB is the default persistence plugin since ActiveMQ 5.4 , and it is also the persistence method used by our project now.
KahaDb recovery time is much shorter than its predecessor AMQ and uses fewer data files, so it can completely replace AMQ.
The persistence mechanism of kahaDB is also based on log files, indexes and caches.
Configuration method:
(1) Main features of KahaDB
1. Store messages in log form;
2. Message index is stored in B-Tree structure, which can be updated quickly;
3. Fully supports JMS transactions;
4. Supports multiple recovery mechanisms;
(2) Structure of KahaDB
Messages are stored in a file-based data log. If the message is sent successfully, the variable is marked as deleteable.
The system periodically clears or archives log files.
The location index of the message file is stored in memory so that it can be quickly located. Regularly save the in-memory message index to the metadata store to prevent the message index from occupying too much memory space when a large number of messages are not sent.
Data logs:
Data logs are used to store message logs, and the entire contents of the messages are in Data logs.
Like AMQ, if the size of a Data logs file exceeds the specified maximum value, a new file will be created. It is also appending at the end of the file, and the writing performance is very fast.
Each message has a count reference in Data logs, so when all messages in a file are no longer needed, the system will automatically delete the file or put it in an archive folder.
Metadata cache:
The cache is used to store messages for online consumers. If the consumer is already consuming quickly, then these messages don't need to be written to disk anymore.
The Btree index will create an index based on MessageID to quickly find messages. This index also maintains the relationship between persistent subscribers and Destinations, as well as pointers to messages consumed by each consumer.
The metadata store
saves the metadata of the messages in the message log in the db.data file, which is also stored in the B-Tree structure, and regularly updates the data from the metadata cache. The metadata store also backs up some information that exists in the message log, which allows the Broker instance to start quickly.
Even if the metadata store file is corrupted or deleted by mistake. The broker can read the data logs and recover it, but the speed will be relatively slow.
4. LevelDB method
After ActiveMQ 5.6, the persistence engine of LevelDB was introduced.
At present, the default persistence method is still KahaDB, but the persistence performance of LevelDB is higher than that of KahaDB, which may be a future trend.
ActiveMQ 5.9 provides a data replication method based on LevelDB and Zookeeper, which is the preferred data replication scheme for the Master-slave method.