Getting started with Pulsar middleware

The article has been included in my Github selection, welcome Star!

a brief introdction

Pulsar is a server-to-server messaging system with multi-tenancy, high performance, and more. Originally developed by Yahoo, it is currently managed by the Apache Software Foundation. It is a top-level project of the Apache Software Foundation and a next-generation cloud-native distributed message flow platform that integrates messaging, storage, and lightweight functional computing. Cross-regional data replication, with streaming data storage features such as strong consistency, high throughput, low latency, and high scalability, is regarded as the best solution for real-time message streaming, storage, and computing in the cloud-native era.

characteristic

• A single instance of Pulsar natively supports multiple clusters, and can seamlessly replicate messages between clusters across computer rooms.
• Extremely low publishing and end-to-end latency.
• Scales seamlessly to over a million topics.
• Simple and easy to use client API supporting Java, Go, Python and C++.
• Supports multiple topic subscription modes (exclusive subscription, shared subscription, failover subscription).
• Message delivery is guaranteed through the persistent message storage mechanism provided by Apache BookKeeper.
• Pulsar IO, a serverless connector framework based on Pulsar Functions, makes it easier to move data in and out of Apache Pulsar.
• Stream-native data processing is implemented by the lightweight serverless computing framework Pulsar Functions.
• Tiered storage offloads data from hot storage to cold/long term storage (eg S3, GCS) when data becomes stale.

Architecture

This is the architecture diagram on the official website, involving several components, here is a brief description:

Broker: Broker is responsible for message transmission, topic management and load balancing. Broker is not responsible for message storage and is a stateless component.

Bookie：负责消息的的持久化，采用Apache BookKeeper组件，BookKeeper是一个分布式的WAL系统。

Producer：生产者，封装消息并将消息以同步或者异步的方式发送到Broker。

Consumer：消费者，以订阅Topic的方式消费消息，并确认。Pulsar中还定义了Reader角色，也是一种消费者，区别在于，它可以从指定置位获取消息，且不需要确认。

Zookeeper：元数据存储，负责集群的配置管理，包括租户，命名空间等，并进行一致性协调。

四种订阅模式

在介绍Pulsar特性时，讲过支持多种订阅模式，总共有四种，分别是独占(exclusive)订阅、共享(shared)订阅、故障转移(failover)订阅、键(key_shared)共享。

独占（Exclusive）

独占模式：同时只有一个消费者可以启动并消费数据；通过 SubscriptionName 标明是同一个消费者），适用范围较小。

共享（Shared）

可以有 N 个消费者同时运行，消息按照 round-robin 轮询投递到每个 consumer 中；当某个 consumer 宕机没有 ack 时，该消息将会被投递给其他消费者。这种消费模式可以提高消费能力，但消息无法做到有序。

故障转移（Failover）

故障转移模式：在独占模式基础之上可以同时启动多个 consumer，一旦一个 consumer 挂掉之后其余的可以快速顶上，但也只有一个 consumer 可以消费；部分场景可用。

键共享（KeyShared）

基于共享模式；相当于对同一个topic中的消息进行分组，同一分组内的消息只能被同一个消费者有序消费。

下载安装

我这里安装的是2.9.1版本的pulsar，链接地址如下：

www.apache.org/dyn/mirrors…

After the download is complete, upload it to the Linux server, and then extract it using the command:

tar -zxvf apache-pulsar-2.9.0-bin.tar.gz

For the stand-alone version, use the command to start in the background:

./bin/pulsar-daemon start standalone

Terminate the command running in the background:

./bin/pulsar-daemon stop standalone

SpringBoot integration

After the startup on the Linux server is completed, it is time to use the Java client to operate. First, introduce Maven dependencies:

<dependency>
    <groupId>io.github.majusko</groupId>
    <artifactId>pulsar-java-spring-boot-starter</artifactId>
    <version>1.1.0</version>
</dependency>
复制代码

application.yml configuration file plus configuration:

#pulsar的服务地址
pulsar:
  service-url: pulsar://192.168.0.105:6650
复制代码

Add a configuration class PulsarConfig:

@Configuration
public class PulsarConfig {

    @Bean
    public ProducerFactory producerFactory() {
        return new ProducerFactory().addProducer("testTopic", String.class);
    }
}
复制代码

Add a constant class that records the topic name:

/**
 * Pulsar中间件的topic名称
 *
 * @author yehongzhi
 * @date 2022/4/9 5:57 PM
 */
public class TopicName {

    private TopicName(){}
    /**
     * 测试用的topic
     */
    public static final String TEST_TOPIC = "testTopic";
}
复制代码

Add message producer PulsarProducerclass:

/**
 * Pulsar生产者
 *
 * @author yehongzhi
 * @date 2022/4/9 5:23 PM
 */
@Component
public class PulsarProducer<T> {

    @Resource
    private PulsarTemplate<T> template;

    /**
     * 发送消息
     */
    public void send(String topic, T message) {
        try {
            template.send(topic, message);
        } catch (PulsarClientException e) {
            e.printStackTrace();
        }
    }
}
复制代码

Add a consumer with topic name "testTopic":

/**
 * topic名称为"testTopic"对应的消费者
 *
 * @author yehongzhi
 * @date 2022/4/9 6:00 PM
 */
@Component
public class TestTopicPulsarConsumer {

    private static final Logger log = LoggerFactory.getLogger(TestTopicPulsarConsumer.class);

    //SubscriptionType.Shared，表示共享模式
    @PulsarConsumer(topic = TopicName.TEST_TOPIC,
                    subscriptionType = SubscriptionType.Shared,
                    clazz = String.class)
    public void consume(String message) {
        log.info("PulsarRealConsumer content:{}", message);
    }

}
复制代码

Finally add a PulsarControllertest send message:

@RestController
@RequestMapping("/pulsar")
public class PulsarController {

    @Resource
    private PulsarProducer<String> pulsarProducer;

    @PostMapping(value = "/sendMessage")
    public CommonResponse<String> sendMessage(@RequestParam(name = "message") String message) {
        pulsarProducer.send(TopicName.TEST_TOPIC, message);
        return CommonResponse.success("done");
    }
}
复制代码

CommonResponsePublic response body:

public class CommonResponse<T> {

    private String code;

    private Boolean success;

    private T data;

    public static <T>CommonResponse<T> success(T t){
        return new CommonResponse<>("200",true,t);
    }

    public CommonResponse(String code, Boolean success, T data) {
        this.code = code;
        this.success = success;
        this.data = data;
    }
    //getter、setter方法
}
复制代码

Start the project, then use postman to test:

Summarize

The above is a simple introduction to pulsar middleware, which introduces the characteristics, architecture, subscription mode of pulsar, and a small example of integrating SpringBoot. Finally, make a comparison with other mainstream middleware for your reference:

Thank you all for reading, I hope this article was helpful to you.

Give it a like if you find it useful, your like is the biggest motivation for my creation ~

I'm a programmer who tries hard to be remembered. See you next time! ! !

The ability is limited. If there are any mistakes or inappropriateness, please criticize and correct them, and learn and communicate together!