Personal business card:
Blogger: Drunkardᝰ.
Personal profile: Indulge in wine, and use the energy of wine to fight for a future.
This article is inspirational: When three of us travel together, we must be my teacher.
This project is based on Java "SpringCloud Microservice Technology Stack" by the dark horse programmer of station B , SpringCloud+RabbitMQ+Docker+Redis+search+distributed
[SpringCloud+RabbitMQ+Docker+Redis+search+distributed, detailed system explanation springcloud microservices technology stack course | Dark horse programmer Java microservices] Click to watch
Table of contents
3. SpringAMQP
2.Work Queue work queue model
Work queue, work queue, can improve message processing speed and avoid queue message accumulation.
RabbitMQ's Work Queue is an important message queue mode, which plays the role of processing intensive tasks in distributed systems.
1. Background and Definition
In distributed systems, message queues are a common data exchange method. RabbitMQ, as a popular open source message broker software, provides a variety of message queue modes, of which work queue is a special form. It mainly solves the problem of multiple consumers executing intensive tasks in an orderly manner, ensuring that each task can only be executed correctly by one consumer.
2. Work queue characteristics
- Orderliness: RabbitMQ work queue ensures orderly consumption of messages. When multiple consumers get messages from the queue at the same time, each consumer processes the messages one by one in their original order.
- Reliability: Once a consumer gets a message from the work queue and starts processing it, the message will be locked to ensure that other consumers cannot process the message at the same time. If processing fails, the message is not removed from the queue and can be tried again by another consumer.
- Efficiency: By distributing intensive tasks to multiple consumers for parallel processing, RabbitMQ work queues can improve the processing capabilities of the system.
- Flexibility: The work queue supports customized message processing logic, and consumers can implement different business processing logic as needed to meet diverse application needs.
3. Implementation principles and specific applications
- Message storage: RabbitMQ work queue stores messages persistently on disk to ensure that messages are not lost when the system crashes. At the same time, it supports memory caching of messages to improve reading efficiency.
- Message processing: When the consumer obtains a message from the work queue, it will be sent to the designated processor for processing. The processor can implement custom business logic as needed, such as processing data, calling other services, or updating the database.
- Feedback mechanism: After processing the message, the consumer can feedback the results to the producer. This mechanism can help producers understand how messages are processed so that they can perform subsequent processing or retry.
4. Comparison with other technologies
- Compare Kafka: Kafka is a stream processing platform. Although it also supports distributed message queues, it is different from RabbitMQ's work queue. Kafka emphasizes the processing of real-time streaming data and is suitable for big data analysis, log collection and other scenarios. The RabbitMQ work queue pays more attention to the orderliness and reliability of messages, and is suitable for distributed systems that need to ensure data consistency.
- Compare ActiveMQ: ActiveMQ is another popular message broker software that also supports work queue mode. Compared with RabbitMQ, ActiveMQ is slightly inferior in performance and stability. RabbitMQ has more efficient memory management, higher throughput and better scalability.
5. Application design and practice
When designing an application system with high efficiency, high revenue, strong reliability, strong flexibility, and convenient monitoring and management, we can use the characteristics of the RabbitMQ work queue for optimization. Here are some tips from experience:
- Properly allocate resources: Properly allocate RabbitMQ and consumer resources based on business needs and system load. Ensure that the system can withstand and maintain stable operation during peak workload periods.
- Optimize message processing logic: When implementing a custom processor, IO operations and time-consuming operations during processing should be minimized to increase message processing speed. At the same time, thread pools or process pools can be used to improve processing concurrency.
- Exception handling and monitoring: During the process of consumer processing messages, exceptions should be caught and handled. At the same time, the system's performance indicators and message queue status are monitored in real time through monitoring and management tools, so that problems can be discovered and adjusted in a timely manner.
- Reasonable use of the retry mechanism: For message processing tasks that may fail, the retry mechanism can be enabled. When a consumer fails to process the message, it can automatically put the message back into the queue and let other consumers try to process it again. It should be noted that the retry mechanism may cause message delays, and the pros and cons need to be weighed before using it.
- Properly adjust work queue parameters: RabbitMQ provides many configurable parameters, such as message survival time, number of concurrent consumers, etc. Based on actual needs and performance test results, these parameters are reasonably adjusted to optimize system performance.
- Monitoring and analysis: With the monitoring tools and analysis plug-ins provided by RabbitMQ, the performance of the message queue and the behavior of consumers and producers can be analyzed and optimized. This plays an important role in discovering and solving system bottlenecks and improving overall performance.
Summarize
RabbitMQ's work queue, as an important message queue mode, plays the role of processing intensive tasks in distributed systems. Through the characteristics of orderliness, reliability, efficiency and flexibility, it provides an efficient, reliable and flexible message processing mechanism for application systems. Compared with other message processing technologies, RabbitMQ work queue has unique advantages and precautions in practical applications. Mastering its implementation principles and application skills can help us better design application systems with high efficiency, high profits, strong reliability, strong flexibility, and convenient monitoring and management. As technology continues to develop and application scenarios continue to change, we expect that RabbitMQ work queues will continue to play an important role in future practice and may be further expanded.
Case: Simulate WorkQueue and implement a queue bound to multiple consumers.
Step 1: The producer sends messages to simple.queue in a loop.
Add a test method in the publisher service and send 50 messages to the simple.queue queue in a loop.
@Test
public void testWorkQueue() throws InterruptedException {
String queueName = "simple.queue";
String message = "hello, message_";
for (int i = 0; i < 50; i++) {
rabbitTemplate.convertAndSend(queueName, message + i);
Thread.sleep(20);
}
}
Step 2: Write two consumers, both listening to simple.queue.
Add a consumer in the consumer service, also listening to simple.queue:
@RabbitListener(queues = "simple.queue")
public void listenWorkQueue1(String msg) throws InterruptedException {
System.out.println("消费者1接受到消息：【"+ msg +"】" + LocalTime.now());
Thread.sleep(20);
}
@RabbitListener(queues = "simple.queue")
public void listenWorkQueue2(String msg) throws InterruptedException {
System.out.println("消费者2接受到消息：【"+ msg +"】" + LocalTime.now());
Thread.sleep(200);
}
Consumer 1 is an odd number, consumer 2 is an even number.
Consumption prefetch limit
Modify the application.yml file and set the preFetch value to control the upper limit of prefetched messages:
spring:
rabbitmq:
listener:
simple:
prefetch: 1
Summarize:
Use of the Work model:
Multiple consumers are bound to a queue, and the same message will only be processed by one consumer.
Control the number of messages prefetched by the consumer by setting prefetch.