Apache Pulsar technology series-PulsarClient implementation analysis

[Live broadcast preview] Shangyun vs. Xiayun: Reduce costs and increase laughter? Cut leeks?

Introduction

Apache Pulsar is a multi-tenant, high-performance inter-service message transmission solution that supports multi-tenancy, low latency, read-write separation, cross-region replication (GEO replication), rapid expansion, flexible fault tolerance and other features. At the same time, in order to achieve high performance, low latency, and high availability, Pulsar has also made a lot of optimizations on the client. This article mainly describes the basic principles and implementation of PulsarClient.

Introduction to PulsarClient

The Pulsar client API design is elegant and concise. PulsarClient is used as the main entrance of the client to facilitate users to remember and build specific clients, such as:

Producer: The producer is used to send messages to the specified topic.
Consumer: A consumer is associated with a specified Topic through subscription and receives messages.
Reader: A consumer for manually managing Cursors. (Internally implemented using Consumer).

PulsarClient also manages client system resources in a unified manner and provides some generalized processing for specific clients, including connection management, thread management, memory management, etc. Next, let's take a look at how PulsarClient is implemented.

What functions does PulsarClient have?

As the unified entrance of the client, it is not difficult to see from the following code snippet that the main function of PulsarClient is to build and destroy PulsarClient instances, and to build various specific Client and transaction instances.

public interface PulsarClient extends Closeable {
    ProducerBuilder<byte[]> newProducer();
    <T> ProducerBuilder<T> newProducer(Schema<T> schema);
    ConsumerBuilder<byte[]> newConsumer();
    <T> ConsumerBuilder<T> newConsumer(Schema<T> schema);
    ReaderBuilder<byte[]> newReader();
    <T> ReaderBuilder<T> newReader(Schema<T> schema);
    void updateServiceUrl(String serviceUrl) throws PulsarClientException;
    CompletableFuture<List<String>> getPartitionsForTopic(String topic);
    CompletableFuture<Void> closeAsync();
    void shutdown() throws PulsarClientException;
    boolean isClosed();
    TransactionBuilder newTransaction() throws PulsarClientException;
}

Implementation principle

Initialization process

PulsarClient can be instantiated using the following code.

PulsarClient client = PulsarClient.builder().serviceUrl("pulsar://broker:6650").build();

PulsarClient and specific clients are built using the Builder mode. Each client has a corresponding ConfigurationData to manage the configuration. The core configuration of PulsarClient is as follows:

public class ClientConfigurationData implements Serializable, Cloneable {
    private String serviceUrl;
   // 用来在运行时外部改变url
    private transient ServiceUrlProvider serviceUrlProvider;
    private long operationTimeoutMs = 30000;
    private long statsIntervalSeconds = 60;
    private int numIoThreads = 1;
    private int numListenerThreads = 1;
    private int connectionsPerBroker = 1;
    private boolean useTcpNoDelay = true;
    private int concurrentLookupRequest = 5000;
    private int maxLookupRequest = 50000;
    private int maxLookupRedirects = 20;
    private int maxNumberOfRejectedRequestPerConnection = 50;
    private int keepAliveIntervalSeconds = 30;
    private int connectionTimeoutMs = 10000;
    private int requestTimeoutMs = 60000;
    private long initialBackoffIntervalNanos = TimeUnit.MILLISECONDS.toNanos(100);
    private long maxBackoffIntervalNanos = TimeUnit.SECONDS.toNanos(60);
    private boolean enableBusyWait = false;
    private String listenerName;
   // 全局内存限制(producer使用)
    private long memoryLimitBytes = 0;
    private String proxyServiceUrl;
    private ProxyProtocol proxyProtocol;
    long tickDuration = 1;
    // transaction
    private boolean enableTransaction = false;
}

The initialization process of PulsarClient is relatively simple. Internal modules are initialized one by one. The following code snippet shows the main modules inside Client.

public class PulsarClientImpl implements PulsarClient {
    // 配置
    protected final ClientConfigurationData conf;
   // 本地元数据管理器，主要负责topic分区个数、topic对应的owner节点以及schema信息
    private LookupService lookup;
   // 共享连接池 双层map结构
    private final ConnectionPool cnxPool;
   // 时间轮
    private final Timer timer;
   // 执行外部逻辑线程组(主要消费使用)
    private final ExecutorProvider externalExecutorProvider;
   // 执行内部逻辑线程组(主要消费使用)
    private final ExecutorProvider internalExecutorService;
    private final AtomicReference<State> state = new AtomicReference<>();
   //producer集合
    private final Set<ProducerBase<?>> producers;
   //consumer集合
    private final Set<ConsumerBase<?>> consumers;
   //producer自增Id
    private final AtomicLong producerIdGenerator = new AtomicLong();
   //consumer自增Id
    private final AtomicLong consumerIdGenerator = new AtomicLong();
   // 请求自增Id
    private final AtomicLong requestIdGenerator = new AtomicLong();
   // netty 线程组
    protected final EventLoopGroup eventLoopGroup;
   // 生产本地buffer内存限制器
    private final MemoryLimitController memoryLimitController;
  ...
}

When PulsarClient is initialized, it mainly creates Netty client, connection pool, time wheel and other objects. It only prepares resources and does not establish a connection with the server for any interaction. Only when a specific client is created will it interact with the server.

Producer Create

Pulsar provides external services at topic granularity. A multi-partition topic is equivalent to a collection of topics with multiple different numerical suffixes. The Topic-Partition mentioned below includes a Partition in a single-partition Topic and a multi-partition Topic. The implementation of Pulsar client Topic-Partition is independent of each other, and a specific client will be created for each Topic-Partition internally in the SDK. We only introduce the initialization process of Producer here (Consumer is similar).

The Producer can be built with the following code.

Producer<byte[]> producer = client.newProducer().topic("my-topic").create();

When My-topic is a Non-partitioned Topic, a ProducerImpl object will be instantiated and returned. When the number of My-topic partitions is greater than 0, a PartitionedProducerImpl object will be created. The PartitionedProducerImpl object contains List. It can be understood that when PulsarClient creates a Producer, it will eventually create ProducerImpl objects with the same number of partitions. Each ProducerImpl works independently and does not affect each other (similar to Consumer).

When creating a Producer, the client and server command words interact as follows:

PulsarClient selects a url through the user-specified ServiceUrl to connect to the server and perform authentication-related operations.
Use LookupService to send the PARTITIONED_METADATA command word to query the number of partitions for a given Topic.
Loop to create ProducerImpl objects based on the number of partitions in the Metadata return result.

3.1 When the ProducerImpl object is initialized, it will use LookupService to send a LOOKUP request to query the Owner node of the corresponding partition. For the Lookup process, please refer to https://km.woa.com/articles/show/555638.

3.2 Connect to the Owner node based on the LOOKUP response, and send a PRODUCER request to the server to create a Producer.

At this point the Producer has been created and can be officially used to send messages.

ps: If the number of partitions changes after the Producer is created, for example, the server is expanded, can the client detect it and increase the number of ProducerImpl objects? The answer is yes. By default, a detection will be initiated every minute, and any partition changes will be handled accordingly.

Connection management

Like most components, the client and server communicate using long connections. The Pulsar protocol is not designed in a traditional response mode and can simultaneously support multiple clients using the same connection to send and receive requests in parallel (the server will serially process requests on a single Topic-partition to ensure message sequence).

Thanks to connection sharing, the number of connections consumed by the client is very small. PulsarClient will create a connection pool for each Broker. The default number of connections is 1. Users can use the ConnectionsPerBroker configuration to set the maximum number of connections for each Broker. When ProducerImpl and ConsumerImpl are initialized, they will randomly obtain a connection from the connection pool to communicate with the server.

In the figure below, maxConnectionsPerHosts=2, 2 connections are created for each Broker in the connection pool, and 6 clients will randomly select a connection to bind in the corresponding Topic owner node.

Connected health management

Pulsar keepAlive detection is bidirectional. After the connection is successfully established, both the client and the server will send a Ping request to the peer regularly for 30 seconds (KeepAliveIntervalSeconds configuration can be modified). After receiving the Ping request, a Pong will be responded to indicate survival. In the following situations, the client and server will actively disconnect:

The handshake action was not completed within the timeout period.
When sending a Ping or Pong command, the Netty callback fails to be sent.
The connection isAutoRead is opened and no request (including Ping and Pong) is received within the timeout period.

After the connection is disconnected, all clients bound to the connection will be notified, and these clients will re-obtain healthy connections from the connection pool. Idle connections are not automatically recycled in Pulsar.

Threading model

PulsarClient uses Netty as the network communication framework and is a standard Netty client. Both protocol processing and event driving rely on Netty. The core processing class directly inherits from Netty Handler.

Therefore, the threading model mainly revolves around Netty's EventLoopGroup. As mentioned above, client resource management converges on PulsarClient, that is, specific clients created using the same PulsarClient all share resources such as threads in the PulsarClient. For example, using the ClientA object to create one or more Producer, Consuemer, Reader clients, these clients all share the thread resources in Client.

PulsarClient threads and thread groups are as follows:

The solid line in the figure indicates that the client will select a thread from the thread pool to bind and run.

Pulsar-client-io : io thread (Netty internal thread), responsible for network connection, reading and writing. NumIoThreads parameter configuration, the default value is 1. The client does not bind the IO thread directly, but uses its internal connections to bind the IO thread. Therefore, it is best to configure the number of IO threads less than or equal to the total number of connections, otherwise some threads will not be used.
Pulsar-client-internal : Mainly used for Consumer internal processing, such as placing messages in the receiving queue after receiving them. It is also configured through the NumIoThreads parameter, and the default value is 1.
Pulsar-external-listener : Mainly used for Consumer external processing, such as user consumption logic callbacks. NumListenerThreads parameter configuration, the default value is 1.
Pulsar-timer : The internal thread of the time wheel is responsible for all timing operations, such as connection reconnection, sending timeout detection, etc. One PulsarClient corresponds to one thread.

Briefly describe how threads interact during production and consumption:

Production : The user thread creates the message and places it in the local cache, and the IO thread is responsible for sending the message to the server.
Consumption : The IO thread receives the message push from the server, uses the Pulsar-client-internal thread to put the message in the local cache queue, and then uses the Pulsar-external-listener thread to execute the user message processing logic.

Summary and reflections

This article introduces the overall client architecture of Pulsar, explains the initialization process of PulsarClient and Producer, as well as the client's connection management and thread model. It does not involve detailed production and consumption processes. It is not difficult to find that the biggest difference between the Pulsar client and other component clients is that Producer/consumer is created for each Topic-partition. If the number of Topic-partitions associated with the client is large, the number of Producers/consumers will expand rapidly, causing the client to consume more resources. It is precisely because the number of Producers/consumers may be large that resources such as connections and threads cannot be independent and can only be shared by Producers/consumers. When sharing resources, it is inevitable that clients will affect each other. For example, current limiting is controlled in the connection dimension, but since the connection is shared, the current limiting of certain topics will affect all clients on the connection. It is recommended that when the number of Topic-partitions associated with the user client is large, the size of the connection pool and thread pool can be appropriately increased to alleviate the impact, or different PulsarClients can be used for client isolation.