Introduction combat
RabbitMQ as the widely used messaging middleware, acts as an important role in enterprise applications, micro-services applications. In particular, it plays an important role in some typical application scenarios, and a service module, such as a business service module decoupled, asynchronous communication, high concurrency limit, service timeout, the data delay processing.
Learning courses which link address: https: //edu.csdn.net/course/detail/9314
RabbitMQ official website Replying
First, let us first have read technical manuals RabbitMQ official website development and related Features, interested friends can patiently read them related presentations, I believe there will be some gains, address visible:
http://www.rabbitmq.com/getstarted.html
Read the manual process, we can see that RabbitMQ fact, the core is around the "message model" to expand, including the relevant components news model: producers, consumers, queues, switches, routers, and other news! And we in the actual application, actually revolves around "message model" to expand the line and code!
Now, I will introduce the evolution model of the news, of course, in the course of the official website RabbitMQ can also get a glimpse of the view!
The above has outlined several figures on a few points, and that these points can be said that the meaning of a word to its name!
Manufacturer: program sends a message
consumer: listener program receives messages consumption of
message: a string of binary data stream
queues: a message temporary area / store
Switch: message transfer station for receiving the distribution message. There fanout, direct, topic, headers four kinds of
routes: equivalent key / third party, and bind to the switch for routing messages to a specified queue!
As the evolution of messaging model of the figure shows, in the form of code then we will combat all kinds of typical business scenario!
SpringBoot integrate RabbitMQ combat
We must first of its profits. We first need the help of IDEA Spring Initializr SpringBoot build a project with Maven, and rely on third parties to introduce framework RabbitMQ, Mybatis, Log4j and so on. After the build is complete, you can simply write a RabbitMQController to test whether the project to build a successful (you can temporarily build a single module mode)
Then, we enter the real core stage, use RabbitMQ in the project or service, in fact, nothing more than a few core elements must firmly grasp, these core elements in the process line and the code needs "time wandering in their own mind ", including:
I want to send a message of what
I should need to create a model of what the message: DirectExchange + RoutingKey? TopicExchange + RoutingKey? And so
my message to be processed in real time or require a delay / delay?
Where producers need to write a message, listen consumers need to write a message of where their processing logic is valid
based on a few key points, we start with a small test to merit some, the use of asynchronous write RabbitMQ combat log and asynchronous e-mail . Of course, before performing actual combat, we need to install RabbitMQ console and back-end applications, and configure the look RabbitMQ relevant parameters and related Bean component in the project.
RabbitMQ After the installation is complete, open the back-end console application: http: // localhost: 15672 Enter the guest guest log in, see the following means that the installation was successful
Then the project configuration file-level configuration application.properties
= 127.0.0.1 spring.rabbitmq.host
spring.rabbitmq.port = 5672
spring.rabbitmq.username = Guest
spring.rabbitmq.password = Guest
spring.rabbitmq.listener.concurrency = 10
spring.rabbitmq.listener.max-20 is Concurrency =
spring.rabbitmq.listener.prefetch = 5
the maximum value of concurrent consumer initialization value, concurrent consumers, each consumer can monitor: wherein the latter three parameters are mainly used for "concurrency configuration" represents the number of messages can pull handle.
Next, we need to be disposed of RabbitMQ Configuration and displayed in a manner Bean RabbitMQ injection process when transmitting and receiving messages related Bean component configuration is a typical configuration wherein RabbitTemplate and SimpleRabbitListenerContainerFactory, the former serving as a message transmission component, which is the container factory management RabbitMQ listener listener, the code is as follows:
@Configuration
public class RabbitmqConfig {
private static final Logger log= LoggerFactory.getLogger(RabbitmqConfig.class);
@Autowired
private Environment env;
@Autowired
private CachingConnectionFactory connectionFactory;
@Autowired
private SimpleRabbitListenerContainerFactoryConfigurer factoryConfigurer;
/**
* 单一消费者
* @return
*/
@Bean(name = "singleListenerContainer")
public SimpleRabbitListenerContainerFactory listenerContainer(){
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
factory.setConnectionFactory(connectionFactory);
factory.setMessageConverter(new Jackson2JsonMessageConverter());
factory.setConcurrentConsumers(1);
factory.setMaxConcurrentConsumers(1);
factory.setPrefetchCount(1);
factory.setTxSize(1);
factory.setAcknowledgeMode(AcknowledgeMode.AUTO);
return factory;
}
/**
* 多个消费者
* @return
*/
@Bean(name = "multiListenerContainer")
public SimpleRabbitListenerContainerFactory multiListenerContainer(){
SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
factoryConfigurer.configure(factory,connectionFactory);
factory.setMessageConverter(new Jackson2JsonMessageConverter());
factory.setAcknowledgeMode(AcknowledgeMode.NONE);
factory.setConcurrentConsumers(env.getProperty("spring.rabbitmq.listener.concurrency",int.class));
factory.setMaxConcurrentConsumers(env.getProperty("spring.rabbitmq.listener.max-concurrency",int.class));
factory.setPrefetchCount(env.getProperty("spring.rabbitmq.listener.prefetch",int.class));
return factory;
}
@Bean
public RabbitTemplate rabbitTemplate(){
connectionFactory.setPublisherConfirms(true);
connectionFactory.setPublisherReturns(true);
RabbitTemplate rabbitTemplate = new RabbitTemplate(connectionFactory);
rabbitTemplate.setMandatory(true);
rabbitTemplate.setConfirmCallback(new RabbitTemplate.ConfirmCallback() {
@Override
public void confirm(CorrelationData correlationData, boolean ack, String cause) {
log.info("消息发送成功:correlationData({}),ack({}),cause({})",correlationData,ack,cause);
}
});
rabbitTemplate.setReturnCallback(new RabbitTemplate.ReturnCallback() {
@Override
void returnedMessage public (the Message Message, int replyCode, replyText String, String Exchange, String routingKey) {
log.info ( "message loss: exchange ({}), route ({}), replyCode ({}), replyText ({} ), Message: {} ", Exchange, routingKey, replyCode, replyText, Message);
}
});
return rabbitTemplate;
}}
RabbitMQ combat: decoupling and asynchronous communication service module
in some enterprise systems, we often see an execution function is typically composed of many sub-modules, the function in the implementation process, need to be synchronized to the code executed from the beginning to the end, i.e., the execution flow is module_A -> module_B -> module_C - > module_D, typical cases can be process-oriented application development language assembler or C language, etc. see, now there are a number of JavaWeb applications such wording.
And we know that this is actually for the whole execution process is a certain function in terms of drawbacks, the main points:
The entire execution function of the response time will be slow;
if an exception occurs while the module is not handled properly, may affect the implementation process and the results of other module or even the whole function of;
the whole function of the code can be very lengthy, it may take between the module and the module be strong communication and interaction data, difficult to locate and maintain, or even fall into "a change of code as a whole," the embarrassing situation when problems arise!
Therefore, we need to find ways to optimize, we need to decouple the service module and the strong association between some implementation of asynchronous communication module! Take the following two scenarios to combat our optimization measures!
Scene One: asynchronous log records of user actions
For micro-enterprise application or service applications, we often need to keep track of retrospective operation log of the user, which is part of the business in a way that should not be coupled together with the main service module, and therefore we need to be alone extracted and asynchronously with a main asynchronous communication module interaction data.
Here we use the RabbitMQ messaging model is also DirectExchange + RoutingKey implementation scenario "user logs in successfully logging" in. As previously said, we need echoed a few key points in mind:
Message model: DirectExchange + RoutingKey message model
news: Entity user login information, including user name, logon events, IP sources, such as your log module
transmits and receives: Achieving sent Controller Log in to achieve a listener receives the and listening to the news consumption in data table; real-time transmission and reception
first we need to create a message in the above model RabbitmqConfig class: including the establishment of Queue, Exchange, RoutingKey, etc., as follows:
Env get the figure above information, we need to be configured in application.properties, which mq.env = local
At this point, we will throughout the project / service up and running, and open the RabbitMQ back-end console application, you can see the queue and switch its binding has been created, as follows:
Next, we need to implement user login logic in the Controller, record user login log, query to get user roles vision and other information resources, lack of space, we focus to achieve here is to use MQ achieve "asynchronous record user login log," the logic, that is where the Controller to act as a "producer", the core code is as follows:
@RestController
public class UserController {
private static final Logger log= LoggerFactory.getLogger(HelloWorldController.class);
private static final String Prefix="user";
@Autowired
private ObjectMapper objectMapper;
@Autowired
private UserMapper userMapper;
@Autowired
private UserLogMapper userLogMapper;
@Autowired
private RabbitTemplate rabbitTemplate;
@Autowired
private Environment env;
@RequestMapping(value = Prefix+"/login",method = RequestMethod.POST,consumes = MediaType.MULTIPART_FORM_DATA_VALUE)
public BaseResponse login(@RequestParam("userName") String userName,@RequestParam("password") String password){
BaseResponse response=new BaseResponse(StatusCode.Success);
try {
//TODO：执行登录逻辑
User user=userMapper.selectByUserNamePassword(userName,password);
if (user!=null){
//TODO：异步写用户日志
try {
UserLog userLog=new UserLog(userName,"Login","login",objectMapper.writeValueAsString(user));
userLog.setCreateTime(new Date());
rabbitTemplate.setMessageConverter(new Jackson2JsonMessageConverter());
rabbitTemplate.setExchange(env.getProperty("log.user.exchange.name"));
rabbitTemplate.setRoutingKey(env.getProperty("log.user.routing.key.name"));
Message message=MessageBuilder.withBody(objectMapper.writeValueAsBytes(userLog)).setDeliveryMode(MessageDeliveryMode.PERSISTENT).build();
message.getMessageProperties().setHeader(AbstractJavaTypeMapper.DEFAULT_CONTENT_CLASSID_FIELD_NAME, MessageProperties.CONTENT_TYPE_JSON);
rabbitTemplate.convertAndSend(message);
}catch (Exception e){
e.printStackTrace();
}
//TODO：塞权限数据-资源数据-视野数据
}else{
= new new BaseResponse Response (StatusCode.Fail);
}
} the catch (Exception E) {
e.printStackTrace ();
}
return Response;
}}
In the above "transmit logic" code, in fact, reflects the evolution of our presentation beginning several models in the message, such as we are sending a message to the Exchange rather than the Queue, the message in the form of a binary stream for transmission, and so on. When a request to the controller postman way, we can see the application of the rear end of the console RabbitMQ one unacknowledged message, details of which can be seen by the GetMessage, as follows:
Finally, we will develop the consumer side of the business code, as follows:
@Component
public class CommonMqListener {
private static final Logger log= LoggerFactory.getLogger(CommonMqListener.class);
@Autowired
private ObjectMapper objectMapper;
@Autowired
private UserLogMapper userLogMapper;
@Autowired
private MailService mailService;
/**
* 监听消费用户日志
* @param message
*/
@RabbitListener(queues = "${log.user.queue.name}",containerFactory = "singleListenerContainer")
public void consumeUserLogQueue(@Payload byte[] message){
try {
UserLog userLog=objectMapper.readValue(message, UserLog.class);
log.info ( "listen to the message consumer users logging listener: {}", userlog);
// the TODO: log into the data table
userLogMapper.insertSelective (userlog);
} the catch (Exception E) {
e.printStackTrace ();
}
}
will go up after the service, we can monitor consumption to the top of Queue messages that the current user login information, and we can also see the "record user login log," the logic is different from a main service thread asynchronous thread to perform:
Case "asynchronous log recording user actions," I think the theory of knowledge sufficient for the interpretation of the above stated, and in subsequent chapters, because of space limitations, I will focus on its core business logic!
Scene 2: Asynchronous sending mail
Send e-mail scenes, it is actually more common, such as user registration requires email verification, remote login users send e-mail notification, and so, here I send a message to RabbitMQ asynchronous. The steps to achieve a nearly identical with the scene!
1. Create a message model
2. Create a configuration information
3. Production end
4. Consumer end
RabbitMQ combat: concurrency and message acknowledgment mechanism disposed
actual background
For message listener model, the default is "single instance of consumption" configuration, i.e., "a listener corresponding to a consumer", such a configuration for the above stated "Asynchronous log records of user actions," "asynchronous transmission under message "and other high-concurrency scenario is not applicable. But for the spike in the system, such as mall grab a single scene may seem very difficult!
We all know that, with the mall to grab a single spike system has a significant feature in common, that is, the request will reach millions of our interface at some point, that moment this great influx of traffic to our system, we can use the following chart to roughly a manifestation of this phenomenon:
When the "Start spike", "began to grab the single" moment, then the system may occur such phenomena:
Application of the system configuration can not carry this unit instantaneous flow, cause the system to hang directly, that is the legendary "down" phenomenon;
interface logic does not consider concurrency, database write lock conflict, leading to the final processing results with theoretical results data inconsistent (e.g., only the amount of commodity stock 100, but a highly concurrent, the amount of data actually recorded user table records grab is far greater than 100);
the application occupies resources of the server directly soar, such as CPU, memory, and other broadband instant direct soared, leading to the same table with the library may hang up or even Caton phenomenon with other services or systems of host;
Ever since, we need to find a solution! For now, the Internet has a lot more good solution, and I, by the way common solution used by our applications, including:
We will deal with grab one of the overall business logic independent of service and do a clustered deployment;
huge traffic we will reject that kind of system in the upper, ie not directly transferred to the influx of MQ our interface, thus reducing database write lock conflicts occur because the interface logic and complex threads appear blockage caused by the application server resources occupied soar;
we'll grab a single source of data with other business systems or even the same table where business is split out the services of an independent, and based on some go RPC protocol HTTP communication for data exchange, communication services and so on;
using a distributed lock resolved with a phone number at the same time, at the same time with a single brush IP phenomenon;
Here, we use RabbitMQ to combat the second point above ! I.e., we will "request" -> "grab one interface processing service" message middleware layer intermediate frame do "buffer", "pressure relief" process, as shown below:
Concurrency configuration message acknowledgment.
As talked about above, for rush orders, spike high concurrency system, if we need to use RabbitMQ in the "request" - act as a pressure limiting relief between the "interface" that would require us to put forward higher RabbitMQ the requirement that support high concurrent configuration, where we need to be clear, "concurrent consumer" is actually configured for the listener, when the configuration is successful, we can see in the MQ application back-end console of consumers number, as shown below:
Wherein, the listener 10 has a configuration where instances of the consumer, the consumption of each consumer may be pre-listening number of messages to pull 5 (if the same never-ending time, the client will cache the waiting processing mq!)
In addition, for some news, we sometimes need to know whether strict message has been dealt with consumer consumption monitor that we have a mechanism to ensure that our message is to confirm whether the message has really been processed consumption. In RabbitMQ, the message confirmation process, there are three mechanisms: Auto - Auto, Manual - manual, None - without confirmation, and confirmation mechanisms need to implement ChannelAwareMessageListener listener interface, and consumer confirm rewriting logic. Here we will use the "Manual confirmation" mechanism to combat the mall to grab a single user scenario.
1. Configure confirm consumption and concurrency mechanism disposed in the RabbitMQConfig
2. Model configuration information message
3.RabbitMQ back-end console app to view the configuration concurrency of this queue
4.listener confirm consumption processing logic: Here we need to develop grab one business logic, that "only when the merchandise inventory> 0, grab a single success, the deduction of inventory, and grab one of the user information into the record table, grab a single-user asynchronous notification success! "
Then we request the multithreaded CountDownLatch generating simulated high concurrency (or with jmeter embodiment pressure may be measured!), Each request will carry a random number generated: acts as a phone number -> serving as messages, grab a single queue into the final! Here, I simulated 50,000 requests, the equivalent request 50000 phone number at the same time grab a single occurrence, while the stock portfolio is set to 100, which can be set up in the product database tables
6. The grab a single phone number information request queue pressed, the processing queue waiting
7. Finally, we write a Junit or write Controller, be initService.generateMultiThread (); Call of the simulation to produce a high concurrent requests can grab one
@RestController
public class ConcurrencyController {
private static final Logger log= LoggerFactory.getLogger(HelloWorldController.class);
private static final String Prefix="concurrency";
@Autowired
private InitService initService;
@RequestMapping(value = Prefix+"/robbing/thread",method = RequestMethod.GET)
public BaseResponse robbingThread(){
BaseResponse response=new BaseResponse(StatusCode.Success);
initService.generateMultiThread();
return response;
}}
8. Finally, of course, we are up and running, we can observe in the console system constantly generating new request (thread) - equivalent to continue to have a single phone number influx grab our system and then into the queue, listener consumer grab a single logic processing after listening to requests! Finally, we can observe two database tables: Table commodity stocks, commodity successful grab a single user record table - the table only when inventory merchandise inventory corresponding to 0, commodity successful grab a single user record 100 just means that our combat the purpose and effect has been reached! !
Summary: In this way, we put the request to transfer our mq, to some extent eased the pressure on our application and interface! Of course, in reality, our configuration may be far more than the configuration on the code level, such as our mq might do the cluster configuration, load balancing, updated commodity stocks may be considered sub-library sub-table, update inventory may be considered independent Dubbo to inventory and service through asynchronous communication and interaction Rest Api independent arrangements, among others. The purpose of these optimizations and improvements in fact nothing more than to be able to limit the flow, pressure relief, to ensure system stability, data consistency and so on! Our MQ, where you can play an indelible role, its word to its name: "Message Queue", and the queue has a "FIFO" characteristics, therefore all messages entering the MQ will be "well-behaved" in MQ on lined up, first come, first line, first come, first be processed consumption and thus to avoid at least a case of "instant rush of time request the influx of our interface," the!
Note: In solving the above-mentioned high concurrent with RabbitMQ combat grab a single program, in fact, I also optimized the database level, namely the use of a "similar optimistic lock" written during read and write warehousing, ensure that: grab a single request comes when there is inventory, inventory can be updated to ensure that there are updates to keep the library!