Kafka Study Notes 4 - Kafka consumer client (PHP) Development

First, the preparatory work

While Kafka is Java / Scala language, but without prejudice to its support for multiple languages. In Kafka's official website can CLIENTS see Kafka language support, including C / C ++, Python, Go and other languages.

PHP operation Kafka need to install librdkafka kafka library and the PHP extension.
1. Install librdkafka library

git clone https://github.com/edenhill/librdkafka.git
./configure
 make
 sudo make install

2. Installation extension php-kafka

$ git clone https://github.com/arnaud-lb/php-rdkafka.git
$ cd librdkafka/
$ phpize
$ ./configure 
$ make
$ sudo make install

#在php.ini 文件中配置 rdkafka扩展
extension=rdkafka.so

#查看扩展是否生效
php -m | grep kafka

Second, code implementation

demo from https://github.com/arnaud-lb/php-rdkafka#examples

Normal production logic is as follows:
1. Set the producer client parameters and creates a corresponding instance of the producer;

/**
 * Create a producer
 */
$conf = new RdKafka\Conf();
$conf->set('log_level', LOG_DEBUG);
//$conf->set('debug', 'all');
$rk = new RdKafka\Producer($conf);
$rk->addBrokers("127.0.0.1");

2. Construction of the theme;

/**
 * Create a topic instance from the producer
 */
$topic = $rk->newTopic("test");

3. Send message;

/**
 * Producing messages
 * The first argument is the partition. RD_KAFKA_PARTITION_UA stands for unassigned, and lets librdkafka choose the partition.
 * 第一个参数是分区，RD_KAFKA_PARTITION_UA 表示未分配，并且由 librdkafka 选择分区。
 * The second argument are message flags and should be either 0 or RD_KAFKA_MSG_F_BLOCK to block produce on full queue.
 * 第二个参数是消息标志，为 0 或 RD_KAFKA_MSG_F_BLOCK，当队列满了时阻止生产消息。
 * The message payload can be anything.
 * 消息可以是任何内容。
 */
$topic->produce(RD_KAFKA_PARTITION_UA, 0, "Message payload");

4. Close the producer instance.

/**
 * Proper shutdown
 * This should be done prior to destroying a producer instance
 *   to make sure all queued and in-flight produce requests are completed before terminating.
 * 关闭生产者实例前需确保所有在队列中和正在生产的生产请求都已完成。
 * Not calling flush can lead to message loss!
 * 不调用flush会导致消息丢失！
 */
$timeout_ms = 60000; // 1 minute
$rk->flush($timeout_ms);

Check whether the message is sent successfully
terminal to open a consumer:

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test

Execute php producer.php in another window,

You can see the consumer terminal receives the message.

The complete code is as follows:

<?php
/**
 * Created by PhpStorm.
 * User: liulu
 * Date: 2020/1/1
 * Time: 18:38
 */

/**
 * Create a producer
 */
$conf = new RdKafka\Conf();
$conf->set('log_level', LOG_DEBUG);
//$conf->set('debug', 'all');
$rk = new RdKafka\Producer($conf);
$rk->addBrokers("127.0.0.1");

/**
 * Create a topic instance from the producer
 */
$topic = $rk->newTopic("test");

/**
 * Producing messages
 * The first argument is the partition. RD_KAFKA_PARTITION_UA stands for unassigned, and lets librdkafka choose the partition.
 * 第一个参数是分区，RD_KAFKA_PARTITION_UA 表示未分配，并且由 librdkafka 选择分区。
 * The second argument are message flags and should be either 0 or RD_KAFKA_MSG_F_BLOCK to block produce on full queue.
 * 第二个参数是消息标志，为 0 或 RD_KAFKA_MSG_F_BLOCK，当队列满了时阻止生产消息。
 * The message payload can be anything.
 * 消息可以是任何内容。
 */
$topic->produce(RD_KAFKA_PARTITION_UA, 0, "Message payload");

/**
 * Proper shutdown
 * This should be done prior to destroying a producer instance
 *   to make sure all queued and in-flight produce requests are completed before terminating.
 * 关闭生产者实例前需确保所有在队列中和正在生产的生产请求都已完成。
 * Not calling flush can lead to message loss!
 * 不调用flush会导致消息丢失！
 */
$timeout_ms = 60000; // 1 minute
$rk->flush($timeout_ms);

echo 'finished';exit;