Collection of messaging middleware: MQ (ActiveMQ/RabbitMQ/RocketMQ) + Kafka + notes

Collection of messaging middleware: MQ (ActiveMQ/RabbitMQ/RocketMQ) + Kafka + notes

Recently, a lot of friends have gone to post and approve the autumn recruitment in advance. After the interview, they told me that they were dead when they encountered problems such as message middleware. Uh, I’m a bit at a loss, so I just wanted to do a message middleware topic, categorized and sorted out pure hand-drawn knowledge system diagrams such as MQ (ActiveMQ/RabbitMQ/RocketMQ) + Kafka , interviews and related study notes .

In order to better sort out relevant knowledge, let’s first look at the pure hand-drawn knowledge system diagram

1.1 Outline of Kafka Knowledge System

Since I used the xmind software to hand-paint these knowledge system outlines and cannot be uploaded, they are all displayed in the form of screenshots, and the details are not clear (after all, the picture format is limited), but the original file can be downloaded for free

1.2 Outline of RabbitMQ Knowledge System

1.3 RocketMQ knowledge system outline

After reading the knowledge outline, it’s time for the interview

2.1 Brush Kafka interview

1. What are the uses of Kafka? What is the usage scenario?

2. What do ISR and AR in Kafka stand for? What does the scaling of ISR mean?

3. What do HW, LEO, LSO, LW, etc. in Kafka represent?

4. How does Kafka reflect the order of messages?

5. Do you know the partitioner, serializer, and interceptor in Kafka? What is the processing order between them?

6. What is the overall structure of the Kafka producer client?

7. How many threads are used in the Kafka producer client for processing? What are they?

8. What are the flaws in the design of Kafka's old Scala consumer client?

9. "If the number of consumers in the consumer group exceeds the topic partition, then some consumers will not be able to consume data." Is the sentence correct? If it is correct, is there any hack?

10. What are the circumstances that will cause repeated consumption?

11. What scenarios will cause message leakage consumption?

12. KafkaConsumer is not thread-safe, so how to achieve multi-threaded consumption?

13. Briefly describe the relationship between consumers and consumer groups

14. After you use kafka-topics.sh to create (delete) a topic, what logic will be executed behind Kafka?

15. Can the number of topic partitions be increased? If it can be increased? If not, why?

16. Can the number of topic partitions be reduced? If it can be reduced? If not, why?

17. How to choose the appropriate number of partitions when creating a topic?

18. What internal topics does Kafka currently have, and what are their characteristics? What is their role?

19. What is a priority copy? What is its special function?

20. Where does Kafka have the concept of partition allocation? Briefly describe the general process and principle

21. Briefly describe the log directory structure of Kafka

22. What index files are there in Kafka?

23. If I specify an offset, how can Kafka find the corresponding message?

24. If I specify a timestamp, how can Kafka find the corresponding message?

25. Talk about your understanding of Kafka's Log Retention

26. Talk about your understanding of Kafka's Log Compaction

27. Talk about your understanding of Kafka's underlying storage

28. Talk about the principle of Kafka's delayed operation

29. Talk about the role of Kafka controller

30. What are the flaws in the design of Kafka's old Scala consumer client?

31. What is the principle of consumption rebalance? (Hint: Consumer Coordinator and Consumer Group Coordinator)

32. How is idempotence in Kafka achieved?

33. How are transactions implemented in Kafka?

34. What is an invalid copy? What are the countermeasures?

35. Under multiple copies, the evolution of HW and LEO in each copy

36. What improvements has Kafka made in terms of reliability? (HW, LeaderEpoch)

37. Why does Kafka not support read-write separation?

38. How to implement the delay queue in Kafka

39. How to implement dead letter queue and retry queue in Kafka?

40. How to do message audit in Kafka?

41. How to do message trajectory in Kafka?

42. How to calculate Lag? (Note the difference between read_uncommitted and read_committed states)

43. What metrics does Kafka need to focus on?

44. What design of Kafka makes it have such high performance?

2.2 Brush ActiveMQ interview

1. What is ActiveMQ?

2. What should I do if the ActiveMQ server is down?

3. What should I do if I lose a message?

4. Persistent messages are very slow

5. Uneven consumption of news

6. Dead letter queue

7. Is the message retransmission interval and retransmission times in ActiveMQ?

2.3 Brush the RabbitMQ interview

1. What is the broker in RabbitMQ? What does cluster mean?

2. What is metadata? What are the types of metadata? What is included? What are the metadata related to cluster? How is metadata stored? How is the metadata distributed in the cluster?

3. What is the difference between RAM node and disk node?

4. Is there a limit on the number of messages stored in a queue on RabbitMQ?

5. Channel, exchange and queue in RabbitMQ concept are logical concepts, or are they corresponding to process entities? What role do these things play?

6. What is vhost? What role does it play?

7. What is the difference between declaring queue, exchange, and binding in a cluster system composed of a single node system and multiple nodes?

8. Can the client connect to any node in the cluster to work normally?

9. If the owner node of a certain queue in the cluster fails and the queue is declared to have the durable attribute, can the queue be successfully re-declared from other nodes?

10. What impact will the failure of a node in the cluster have on the consumer? If a mirrored queue is created in the cluster, what impact will the node failure have on the consumer at this time?

11. Can RabbitMQ cluster be used in different geographically separated data centers?

12. Why is disk node not recommended for heavy RPC usage scenarios?

13. What happens when I send a publish message to a non-existent exchange? What happens when you execute a consume action to a non-existent queue?

14. What is the maximum length of routing_key and binding_key?

15. What is the maximum message size allowed by RabbitMQ?

16. Under what circumstances is it safe for the producer not to actively create a queue?

17. What is the purpose of "dead letter" queue?

18. Why do we say that the condition to ensure that the message is reliably persisted is that queue and exchange have durable attributes, while message has persistent attributes?

19. Under what circumstances will blackholed problems occur?

20. How to prevent blackholed problems?

21. What scenario is the Consumer Cancellation Notification mechanism used for?

22. What is the usage of Basic.Reject?

23. Why shouldn't you use a persistence mechanism for all messages?

24. What problems do the cluster, mirrored queue, and warrens mechanisms in RabbitMQ solve? What are the problems?

The answers to all the questions have been sorted out, and the PDF files shown in the following questions are now available. There is no longer a screenshot for the length

After reading the system outline + interview questions, if there are knowledge loopholes, then continue to read the study notes

3.1 Kafka source code analysis and actual combat

Chapter 1 Introduction to Kafka

1.1 The background of the birth of Kafka

1.2 Kafka's internal application in LinkedIn

1.3 The main design goals of Kafka

1.4 Why use a messaging system

Chapter 2 Kafka Architecture

2.1 Basic composition of Kafka

2.2 Kafka's topology

2.3 Communication protocol inside Kafka

Chapter 3 Broker Overview

3.1 Start of Broker

3.2 The module composition inside the Broker

Chapter 4 Broker's basic modules

4.1 SocketServer

4.2 KafkaRequestHandlerPool

4.3 KafkaApis

4.4 KafkaHealthcheck

Chapter 5 Broker's Control Management Module

5.1 KafkaController's election strategy

5.2 Initialization of KafkaController

5.3 Topic's partition state transition mechanism

5.4 Leader copy election strategy for topic partition

5.5 Replica state transition mechanism of topic partition

5.6 Listener inside KafkaController

5.7 Load balancing process of Kafka cluster

5.8 Topic deletion process of Kafka cluster

5.9 Communication module of KafkaController

Chapter 6 Topic Management Tools

6.1 kafka-topics.sh

6.2 kafka-reassign-partitions.sh

6.3 kafka-preferred-replica-election.sh

Chapter 7 Producers

7.1 Design principles

7.2 Sample code

7.3 Module composition

7.4 Send mode

Chapter 8 Consumers

8.1 Simple consumer

8.2 Senior Consumer

Chapter 9 Typical Applications of Kafka

9.1 Integration of Kafka and Storm

9.2 Integration of Kafka and ELK

9.3 Integration of Kafka and Hadoop

9.4 Integration of Kafka and Spark

Chapter 10 A Comprehensive Example of Kafka

10.1 Main applications of security big data

10.2 The role of Kafka in the overall security solution

10.3 Typical business

3.2 Handwritten RocketMQ notes

Section 1: Introduction to RocketMQ

1.1 Core concepts

1.2 RocketMQ's design philosophy and goals

Section 2: Sending of messages in RocketMq

2.1 OneWay sending

2.2 Reliable synchronous transmission

2.3 Reliable asynchronous transmission

2.4 The trade-off of message sending in RocketMQ

Section 3: RocketMQ message consumption

3.1 Cluster consumption and broadcast consumption

3.2 Consumption mode

Section 4: In-depth message sending

4.1 Message Producer Process

4.2 Batch message sending

4.3 Message retry mechanism

Section 5: In-depth message mode

5.1 Pull mode

5.2 Push mode

Section 6: Sequence Message

6.1 Global Sequence Message

6.2 Partial sequence message

Section 7: Delayed Message

7.1 Concept introduction

7.2 Applicable scenarios

7.3 How to use

Section 8: Dead Letter Queue

8.1 Concept introduction

8.2 Applicable scenarios

Section 9: Consumption Idempotence

9.1 What is message idempotence

9.2 Scenarios to be processed

9.3 Treatment method

Section 10: Message Filtering

10.1 Concept introduction

10.2 Expression filtering

10.3 Class filtering

Section 11: RocketMQ storage outline design

11.1 Message storage structure

11.2 Memory mapping

11.3 File flashing mechanism

11.4 Delete expired files

Section 12: Transaction messages in RocketMQ

12.1 Transaction message realization ideas

12.2 Two-phase commit

12.3 Transaction status review mechanism

12.3 Code Implementation

Section 13: RocketMQ master-slave synchronization (HA) mechanism

13.1 RocketMQ cluster deployment mode

13.2 Principle of master-slave replication

13.3 Read and write separation mechanism

13.4 Integration with Spring

13.5 Integration with SpringBoot

Section 14: Actual combat of time-limited orders

14.1 What is a time-limited order

14.2 How to implement time-limited orders

14.3 Realizing Time-Limited Orders with RocketMQ

Section 15: RocketMQ source code analysis

15.1 RocketMQ overall architecture

15.2 NameServer

15.3 RocketMQ service start

15.4 The ins and outs of the news of source code analysis

3.3 RabbitMQ practical study guide

Chapter 1 Introduction to RabbitMQ

1.1 What is a message middleware

1.2 The role of message middleware

1.3 The origin of RabbitMQ

1.4 Installation and simple use of RabbitMQ

Chapter 2 Getting Started with RabbitMQ

2.1 Introduction to related concepts

2.2 Introduction to AMQP protocol

Chapter 3 Client Development Wizard

3.1 Connect RabbitMQ

3.2 Using switches and queues

3.3 Send message

3.4 Consumer news

3.5 Confirmation and rejection on the consumer side

3.6 Close connection

Chapter 4 RabbitMQ Advanced

4.1 Where does the news go

4.2 Expiration time (TTL)

4.3 Dead letter queue

4.4 Delay queue

4.5 Priority Queue

4.6 RPC implementation

4.7 Endurance

4.8 Producer confirmation

4.9 Introduction to Consumer Key Points

4.10 Message transmission guarantee

Chapter 5 RabbitMQ Management

5.1 Multi-tenancy and permissions

5.2 User Management

5.3 Web management

5.4 Application and cluster management

5.5 Server status

5.6 HTTPAPI interface management

Chapter 6 RabbitMQ Configuration

6.1 Environment variables

6.2 Configuration file

6.3 Parameters and strategies

Chapter 7 RabbitMQ Operation and Maintenance

7.1 Cluster construction

7.2 View service log

7.3 Single node failure recovery

7.4 Cluster migration

7.5 cluster monitoring

Chapter 8 Crossing Cluster Boundaries

8.1 Federation

8.2 Shovel

Chapter 9 RabbitMQ Advanced

9.1 Storage mechanism

9.2 Memory and Disk Alarm

9.3 Flow Control

9.4 Mirror queue

Chapter 10 Network Partitioning

10.1 The meaning of network partition

10.2 Judgment of network partition

10.3 Simulation of network partition

10.4 Impact of network partition

10.5 Handling network partitions manually

10.6 Automatic processing of network partitions

10.7 Case: Multi-partition scenario

Chapter 11 RabbitMQ Extension

11.1 Message tracking

11.2 Load Balancing

A wave of operations are fierce, let me summarize middleware

Regarding the message middleware MQ (ActiveMQ/RabbitMQ/RocketMQ) + Kafka , the editor starts with the outline of each knowledge system, wakes up and sorts out the knowledge, and then brushes the major interview high-frequency questions (with detailed answers) , And finally make up as soon as possible when knowledge is missing (look for study notes). This complete set of knowledge outline + interview high-frequency real questions + study notes are indispensable. If you also want to check for omissions or prepare for an interview, then come directly to the editor and share it with you for free! (below)