Introduction to pulsar
Apache Pulsar is a cloud-native, distributed messaging and streaming platform originally created at Yahoo! and now a top-level Apache Software Foundation project
This is an introduction from the official website that Apache Pulsar is a cloud-native distributed message and stream processing platform, originally created in Yahoo! Now a top-level project at the Apache Software Foundation
For the details of the specific pulsar, you can learn through the official documents
need
When updating the transportation status of the main order, there is a business requirement. For example, the detail line of the main order has four materials A, B, C, and D. The A and B material lines correspond to the transportation order number T0001, and the C and D materials correspond to the transportation order number T0002. There is a logic in the background business that only when the shipping order numbers corresponding to all materials are signed, the main order shipping status can be closed.
The original message consumer is consumed in Shared mode. The official website explains this mode in this way. Messages are distributed to different consumers through the round robin polling mechanism, and each message will only be distributed to one consumer. When a consumer disconnects, all unacknowledged messages sent to it will be rescheduled and distributed to other surviving consumers.
So what are the problems with this model in demand? First of all, the service is deployed on multiple nodes (if there are 2 consumers), now the upstream service pushes a message (A, B material transportation order number T0001), and then pushes a message (C, D material transportation order number T0002), these two consumers consume this piece of logic at the same time, then our main order status judgment logic will have problems or it will be very troublesome to handle.
How can the message middleware pulsar easily fulfill such a requirement? My first thought at the time was to use Exclusive mode, but due to the impact on the throughput of consumption, I just gave up. According to my previous impression of Kafka, I wondered if pulsar has such content as partitioning? Sure enough, I found a partition topic to implement this problem message Apache Pulsar
producer
Specify the topic partition to be created through the admin API and specify the number of partitions when creating it.
The class encapsulated by the producer sending the partition message
@Slf4j
@Component
public class ProducerPartionFactory {
@Autowired
private PulsarClient client;
@Autowired
private PulsarAdmin pulsarAdmin;
// 消息生产者集合
private ConcurrentHashMap<String, Producer<byte[]>> producerMap = new ConcurrentHashMap<>();
private Producer<byte[]> getTheProducer(PulsarProducePartionRoter pulsarProducePartionRoter) throws Exception {
String topic = pulsarProducePartionRoter.getTopic();
Producer<byte[]> producer = producerMap.get(topic);
if (producer == null) {
synchronized (topic.intern()) {
producer = producerMap.get(topic);
if (producer == null) {
PartitionedTopicMetadata metadata = null;
String localIpAddress = null;
try {
metadata = pulsarAdmin.topics().getPartitionedTopicMetadata("persistent://public/test/" + topic);
if (metadata.partitions == 0) {
//使用pulsarAdmin来初始化创建分区消息，这里的分区建议按照消费者数量或者消费者数量*n来定义
pulsarAdmin.topics().createPartitionedTopic("persistent://public/test/" + topic, 4);
}
//因为我们这里是多个节点，这里使用ip地址来区别我们不同的生产者名称
localIpAddress = IPUtils.getLocalIpAddress();
if(StringUtils.isNotBlank(localIpAddress)){
localIpAddress.replaceAll("\\.","_");
}
} catch (Exception ex) {
log.error("创建分区主题异常:{}", ex);
throw ex;
}
try {
// topic 生产者消息写到哪个主题中
// producerName 生产者的名字，不唯一即可，如果不设置，会有默认值
// sendTimeout 超时时间
// 默认情况下，当队列已满时，所有对Send和SendAsync方法的调用都将失败，除非您将BlockIfQueueFull设置为true
producer = client.newProducer(Schema.BYTES)
.topic("persistent://public/test/" + topic)
.producerName(topic + localIpAddress)
.sendTimeout(10, TimeUnit.SECONDS)
// .blockIfQueueFull(true)
.create();
producerMap.putIfAbsent(topic, producer);
return producer;
} catch (Exception e) {
e.printStackTrace();
}
}
}
}
return producer;
}
public boolean send(PulsarProducePartionRoter pulsarProducePartionRoter, String message) {
boolean isSuccess = false;
try {
Producer<byte[]> producer = this.getTheProducer(pulsarProducePartionRoter);
MessageId messageId = producer.newMessage().key(pulsarProducePartionRoter.getKey()).value(message.getBytes()).send();
log.info("获取分区发送信息:{}", JSON.toJSONString(messageId));
if (messageId != null) {
isSuccess = true;
}
} catch (Exception e) {
log.error("pulsar发送异常:", e);
}
return isSuccess;
}
}
@Data
public class PulsarProducePartionRoter {
private String topic;
private String key;
}
IPUtils tool class
public class IPUtils {
/**
* 获取本地IP地址
*
* @return 本地IP地址
*/
public static String getLocalIpAddress() {
try {
Enumeration<NetworkInterface> interfaces = NetworkInterface.getNetworkInterfaces();
while (interfaces.hasMoreElements()) {
NetworkInterface iface = interfaces.nextElement();
// 排除回环和虚拟接口
if (iface.isLoopback() || iface.isVirtual() || !iface.isUp()) {
continue;
}
Enumeration<InetAddress> addresses = iface.getInetAddresses();
while (addresses.hasMoreElements()) {
InetAddress addr = addresses.nextElement();
if (!addr.isLinkLocalAddress() && !addr.isLoopbackAddress() && addr.isSiteLocalAddress()) {
return addr.getHostAddress();
}
}
}
} catch (Exception e) {
e.printStackTrace();
}
return null;
}
}A piece of logic for producers to send messages
String topic ="KEY_NEW_ORDER_SIGN";
PulsarProducePartionRoter pulsarProducePartionRoter = new PulsarProducePartionRoter();
pulsarProducePartionRoter.setTopic(topic);
pulsarProducePartionRoter.setKey(requestBean.getStoreNo());
boolean isSuccess = producerPartionFactory.send(pulsarProducePartionRoter, JSON.toJSONString(requestBean));
We can see that the same stroeNo is assigned the same partition number
consumer
After the project is started, we start the consumer thread pool to consume partition message business logic
@Slf4j
@Component
public class SignKeyNewShardConsumerLister implements CommandLineRunner {
private volatile boolean isRunning = true;
private ThreadPoolExecutor threadPoolExecutor = null;
@Autowired
private PulsarClient pulsarClient;
private final EmitterProcessor<FailedMessage> exceptionEmitter = EmitterProcessor.create();
private Consumer<byte[]> consumer;
@Autowired
private PulsarAdmin pulsarAdmin;
@Override
public void run(String... args) throws Exception {
threadPoolExecutor = new ThreadPoolExecutor(1, 1,
0L, TimeUnit.MILLISECONDS,
new LinkedBlockingQueue<>(100),
Executors.defaultThreadFactory(),
new ThreadPoolExecutor.AbortPolicy());
try {
PartitionedTopicMetadata metadata = null;
String topic = null;
try {
topic = "persistent://public/test/KEY_NEW_ORDER_SIGN" ;
metadata = pulsarAdmin.topics().getPartitionedTopicMetadata(topic);
if (metadata.partitions == 0) {
pulsarAdmin.topics().createPartitionedTopic(topic, 4);
}
} catch (Exception ex) {
if (ex.getMessage().contains("Topic not exist")) {
pulsarAdmin.topics().createPartitionedTopic(topic, 4);
} else {
log.error("创建分区主题异常:{}", ex);
throw ex;
}
}
consumer = pulsarClient.newConsumer(Schema.BYTES)
.topic(topic)
.subscriptionName(Topic.KEY_NEW_ORDER_SIGN.getCode())
.receiverQueueSize(1)
.subscriptionType(SubscriptionType.Key_Shared)
.keySharedPolicy(KeySharedPolicy.KeySharedPolicySticky.autoSplitHashRange())
// .autoUpdatePartitions(true)
.subscribe();
} catch (Exception e) {
log.error("初始化客户端SignConsumerLister 异常", e);
}
threadPoolExecutor.submit(() -> {
Message<byte[]> receiveMsg = null;
while (isRunning) {
try {
receiveMsg = consumer.receive();
if (receiveMsg != null) {
try {
// todo 执行业务逻辑
} catch (Exception ex) {
log.error("处理签收业务异常:{}", ex);
}
consumer.acknowledge(receiveMsg);
}
} catch (Exception e) {
log.error("SignConsumerLister exec processMessage Exception: {}", e);
consumer.negativeAcknowledge(receiveMsg);
exceptionEmitter.onNext(new FailedMessage(e, consumer, receiveMsg));
}
}
});
}
@PreDestroy
public void doDestroyListen() {
isRunning = false;
if (threadPoolExecutor != null) {
threadPoolExecutor.shutdown();
}
try {
consumer.close();
} catch (Exception e) {
log.error("SignConsumerLister exec destroy Exception: {}", e);
}
}
}
We also saw that the consumer service processed this message on the same instance node and consumed it sequentially
