Apache Pulsar 2.10.0 version introduction

This article is translated from the StreamNative blog "What's New in Apache Pulsar 2.10" by Li Penghui and Dave Duggins. The original address is https://streamnative.io/blog/release/2022-05-12-whats-new-in-apache -pulsar-210/.

The much-anticipated Apache Pulsar 2.10.0 release has recently been released! The new release includes improvements and bug fixes from 99 contributors, with over 800 changes committed. In addition to the ZooKeeper dependency mitigation  and automated cluster failover  features introduced earlier, Pulsar 2.10.0 also includes more than ten major feature changes. This blog condenses the highlights of the Pulsar 2.10.0 release for you. Let's take a look at the features of the new version that are worth paying attention to, for the convenience of developers and users.

Version highlights:

• ・PR-13316: Provide automatic failback between primary and backup clusters. (Original PIP: PR-13315[1] )

• ・PR-10279:  PartitionedProducer More efficient use of broker memory with fewer producers by adding lazy loading to it.

• TableView ・PR-12838: Add topic map support with new type via key value in received message  .

This blog presents the most noteworthy developments in version 2.10.0, grouped by project feature updates. For a complete list of all performance upgrades and bug fixes, see the  Pulsar 2.10.0 release notes  [2] .

Bug fixes and feature enhancements

Client (supported by Java Client only)

PR-13316: Implement Pulsar cluster-level automatic failover on clients.

Problem: The Pulsar administrator must manually failover the cluster.

Solution: Add automatic failover at the Pulsar cluster level. When a failover event is detected, it automatically and seamlessly switches from the primary cluster to one or more secondary clusters; when the primary cluster recovers, clients automatically switch back.

PR-12517: Support setting topic policies across multiple clusters.

Problem: The topic policy only takes effect on the local cluster 

Solution: Added support for setting topic policies that can be applied to multiple clusters.

• • Set the properties of the message for the local topic policy  replicateTo to avoid being copied to the remote.

• ・ Retention supports setting global parameters.

• ・ SystemTopicBasedTopicPoliciesService Added global theme policy for.

producer

PR-10279: Add lazy loading feature for partition producers.

Problem: The number of partitions is set based on the highest rate producers, the lowest rate producers do not always need to be connected to each partition, so the extra producers take up broker memory.

Solution: Reduce the number of producers to use broker memory more efficiently by introducing lazy loading for partitioned producers; add a round-robin routing pattern class to limit the number of partitions.

PR-12403: Introduce chunk message ID.

Issue: When sending a chunked message, the producer returning the message-id of the last chunk causes an error in some processes.

Solution: A new  ChunkMessage-ID type was introduced. MessageIdImpl Added two new methods for chunk message-id inherited from  : getFirstChunkMessageId and  getLastChunkMessageID. For other method implementations, direct calls  lastChunkMessageIDare compatible with existing business logic.

Broker

PR-12536: Extend the broker to provide more control and flexibility for operators of enterprise clusters.

Problem: Enterprise Pulsar cluster operators need more flexibility and control to intercept broker events (including ledger writes/reads) for template validation, observability and access control.

solution:

• ・ Enhanced org.apache.pulsar.broker.intercept.BrokerInterceptor interface to include additional events for tracing

• ・ Created a new interface org.apache.pulsar.common.intercept.MessagePayloadProcessor, which allows to intercept ledger read and write operations

consumer

PR-10478: Add epoch to redelivery command.

Problem: In older versions, pull and redelivery operations were asynchronous, so client-side consumers might receive a new message and perform a cumulative ack based on the new messageID and not be able to consume the old message.

Solution: Pulsar client synchronizes redelivery and pull message operations through incremental epochs of server and client consumers.

PR-12269: Support pluggable Entry filters in Dispatcher.

Issue: Older versions do not support native message tags.

Solution: Implement the Entry filter framework at the broker level and support namespace and topic levels in subsequent upcoming releases.

PR-13355: Create initial subscription before sending messages to dead letter queue.

Problem: Dead-letter queue data in unprocessed messages is automatically deleted without the need for namespaced data retention policies or dead-letter queue subscriptions.

Solution: Create an initial subscription before sending messages to the dead letter queue. When  deadLetterProducer initialized, the consumer  DeadLetterPolicy initially subscribes according to the settings.

PR-13707: Add redelivery backoff policy for ack timeout.

Issue:   The redelivery backoff policy introduced in PIP-106 [3] only works for negative ack APIs. The backoff policy is bypassed if an ack timeout is used instead of a negative acknowledgement API to trigger message redelivery.

solution:

• ・ Added message redelivery policy for ack timeout.

• ・ RedeliveryBackoff Added alarm  NegativeAckBackoff interface to .

• ・ ConsumerBuilder Exposure  AckTimeoutRedeliveryBackoffin.

PR-13599: (Modify only covers Java clients at the moment) Fix the issue where chunked messages fail when setting topic level maxMessageSize.

Issue: In older versions, chunked message generation failed if topic level maxMessageSize was set to [1].

Solution:  PublishContext add  isChunkedin . If chunked, the  maxMessageSize check is skipped.

Function

PR-13205: Pulsar Functions preload and release external resources.

Problem: External resource initialization and release is either done manually or by using complex initialization logic.

Solution: Introduce an  RichFunction interface to extend by providing setup and tearDown APIs  Function.

PR-12104: Update the authentication interface to include async authentication methods.

Problem: Pulsar's current AuthenticationProvider interface only exposes synchronous methods for authenticating connections. In the absence of any provider that relies on network calls, the functionality is sufficient for now, but in some cases when looking at the OAuth2.0 specification, network calls are required to validate the token.

solution:

AuthenticationProvider

• ・Add  AuthenticationProvider#authenticateAsync. Introduce a default implementation that calls the authentication method.

• ・ Deprecated  AuthenticationProvider#authenticate.

• ・Add  AuthenticationProvider#authenticateHttpRequestAsync.

• ・ Deprecated  AuthenticationProvider#authenticateHttpRequest.

AuthenticationState

• ・Add  AuthenticationState#authenticateAsync.

• ・ Deprecated  AuthenticationState#authenticate and recommended usage  AuthenticationState#authenticateAsync .

• ・ Deprecated  AuthenticationState#isComplete. AuthenticationState#authenticateAsync This approach can be avoided by inferring authentication integrity from  the result.

AuthenticationDataSource

• ・ Deprecated  AuthenticationDataSource#authenticate. The asynchronous version of this method is not required.

PR-12838: Initialize TableView submission.

Problem: In many scenarios, the application uses a Pulsar consumer or reader to get all updates from a topic and construct a map with the latest value for each key for which the message was received. This operation is common when building local caches of data. The community does not provide support for this access mode not included in the Pulsar client API.

Solution: Add new  TableView types and update Pulsar Client.

theme

PR-12818: Support theme metadata (part 1) - create a theme with attributes.

Problem: Unable to store topic metadata.

solution:

• • Added new storage method in topics.java.

• ・Added two new REST API paths to reduce compatibility issues.

metadata storage

PR-13225: Added Etcd MetadataStore implementation.

Issue: The community is working on adding metadata backends that support non-ZooKeeper implementations.

Solution: Add Etcd support for:

• ・ Batch read/write requests

• ・ Session Watch Point

• • Lease manager