Apache Kafka 2.5 stable version released, new features first look

Apache Kafka 2.5 stable version released, new features first look

Past memory big data Past memory big data
Apache Kafka 2.5.0 stable version was officially released on April 15, 2020, local time in the United States. This version includes a series of important feature releases. The more important features include:

•Support TLS 1.3 (currently 1.2 is used by default)
•Kafka Streams DSL supports Co-groups;
•Kafka Consumer supports incremental rebalance (Incremental rebalance)
•In order to better insight into the operation of operators, new indicators have been introduced;
• Apache Zookeeper upgraded to 3.5.7
• Scala 2.11 is no longer supported
• KIP-500 has made new progress.
If you need to download the latest version of Apache Kafka 2.5.0, you can download it here. Due to time and space, the following article lists the more important features and developments this time.

Update of Kafka broker, producer, and consumer

KIP-500 has made new progress

Users who have used Kafka should be aware that Kafka's Broker, topic, and partition-related information are stored in Apache Zookeeper. Apache Zookeeper is an independent system with independent storage. In order to better run the Kafka cluster, develop The readers must be familiar with both Kafka and Zookeeper systems, which increases our burden to a certain extent. In order to reduce the burden on users, KIP-500 proposes to move all the things previously stored in Zookeeper to the internal maintenance of the Kafka cluster, which means that Kafka can run without relying on Zookeeper in the future.

Apache Kafka 2.5.0 version has made new progress in this work. For example, KIP-555 no longer needs to request Zookeeper in the administrator tool, and KIP-543 makes it no longer need to request Zookeeper for dynamic configuration.

KIP-500 is a big feature, and its final solution needs to be reflected in the next few versions.

Improved Exactly once semantics

KIP-447: Producer scalability supports only and once semantics

This KIP simplifies the API of applications that read and write from Kafka in a transactional manner. Previously, this use case usually required a separate producer instance for each input partition. Now this restriction has been removed. This makes it easier to build a one-time semantic application that consumes a large number of partitions. For details, please refer to KIP-447.

KIP-360: Improve the reliability of idempotent/transactional producers

This KIP solves the problem of keeping the producer state on the broker, making idempotence guarantees possible. Before that, when the log was truncated to force retention or truncated from the call to delete the record, the broker would delete the state of the producer, which would cause an UnknownProducerId exception. With this improvement, the broker will retain the producer status until it expires. This KIP also provides a powerful method for producers to recover from accidental errors.

Kafka Streams

KIP-150: Add Cogroup to the DSL

In the past, aggregating multiple streams into one stream can be complicated and error-prone. Usually we need to group and aggregate all flows into a table, and then make multiple outer join calls. The new co-group operator simplifies this process, reduces the number of state storage calls, and improves performance overall. The specific use is as follows:

KTable<K, CG> cogrouped =
  grouped1
    .cogroup(aggregator1)
    .cogroup(grouped2, aggregator2)
    .cogroup(grouped3, aggregator3)
    .aggregate(initializer1, materialized1);

KIP-523: Add toTable() to the DSL

Interpreting the event stream as a change log and materializing it into a table is a powerful method. KIP-523 introduces the toTable() function, which can be applied to the stream and can materialize the latest value of each key. It should be noted that any null value (null) will be interpreted as a deletion of the given key.

KIP-535: Allow state storage to provide outdated data during rebalancing

Previously, interactive queries (IQ) to the state store would fail during rebalancing. This reduces the uptime of applications that rely on the ability to query the Kafka Streams state table. KIP-535 enables applications to query any copy of the state store and be able to observe how far each copy is behind the master copy.

Dropped support and deprecations

We have removed support for Scala 2.11 in Apache Kafka 2.5, and now Scala 2.12 and 2.13 are the only supported versions.

TLS 1.2 is now the default SSL protocol, but TLS 1.0 and 1.1 are still supported.

参考：What’s New in Apache Kafka 2.5：https://www.confluent.io/blog/apache-kafka-2-5-latest-version-updates/