k8s kafka deployment actual combat

Yuxian: CSDN content partner, CSDN new star mentor, 51CTO (Top celebrity + expert blogger), github open source enthusiast (secondary development of go-zero source code, game back-end architecture https://github.com/Peakchen)

 

Kubernetes (k8s) is an open source platform for automating container operations, while Kafka is a distributed stream data processing platform. Deploying Kafka on k8s can make the deployment of Kafka easier to manage and maintain. The following is a simple deployment of k8s Kafka:

1. Install k8s cluster

Before deploying Kafka, you need to prepare a k8s cluster. You can use tools such as Minikube to build a k8s cluster locally for testing. If you need to deploy in a production environment, you can use k8s services provided by cloud service providers, such as AWS EKS, Google GKE, etc.

1. Download the Kafka image

You can download the Kafka image from the official website or use the Kafka image on Docker Hub. After the download is complete, the image needs to be uploaded to the Docker warehouse of the k8s cluster.

1. Create a Kafka Topic

Use the command line tool kubectl provided by k8s to create a Kafka Topic, you can create a topic by command kubectl create -f <topic.yaml>.

1. Create a Kafka deployment file

Create a Kafka deployment file on the k8s cluster to define Kafka's Pod, Service, etc. An example deployment file is as follows:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kafka
  namespace: default
spec:
  selector:
    matchLabels:
      app: kafka
  serviceName: kafka
  replicas: 3
  template:
    metadata:
      labels:
        app: kafka
    spec:
      containers:
      - name: kafka
        image: <kafka-image>
        ports:
        - containerPort: 9092
        volumeMounts:
        - name: data
          mountPath: /var/lib/kafka
      volumes:
      - name: data
        emptyDir: {}

This file defines a StatefulSet for creating Kafka Pods. It defines 3 replicas, uses Kafka mirroring, exposes port 9092, and mounts a data volume.

1. Create Kafka Service

Use the command-line tool kubectl provided by k8s to create Kafka Service and expose the Kafka cluster outside the k8s cluster. An example Service file is as follows:

apiVersion: v1
kind: Service
metadata:
  name: kafka
  namespace: default
  labels:
    app: kafka
spec:
  ports:
  - name: kafka
    port: 9092
    protocol: TCP
  selector:
    app: kafka

This file defines a Service that exposes the Kafka cluster outside the k8s cluster and exposes Kafka's port 9092.

1. Department Kafka

Use the command kubectl apply -f <kafka-deploy.yaml> to deploy Kafka.

1. Test Kafka

After using the command line tools provided by Kafka to test Kafka deployment, you can use the command line tools provided by Kafka to test whether Kafka is working properly.

1. Create a Kafka topic:

kubectl exec -it kafka-0 -- /usr/bin/kafka-topics --create --topic test-topic --partitions 3 --replication-factor 3 --if-not-exists --zookeeper zk-cs:2181

1. View the list of Kafka topics:

kubectl exec -it kafka-0 -- /usr/bin/kafka-topics --list --zookeeper zk-cs:2181

If you can see the test-topic list, it means that Kafka is working normally.

1. Produce messages to Kafka:

kubectl exec -it kafka-0 -- /usr/bin/kafka-console-producer --topic test-topic --broker-list localhost:9092

At this point, you will enter a command line interface, enter the message and press Enter, and the message will be sent to Kafka.

1. Consumer news:

kubectl exec -it kafka-0 -- /usr/bin/kafka-console-consumer --topic test-topic --from-beginning --bootstrap-server localhost:9092

This will take you to a command line interface displaying the received message.

The kafka-0 in the above test command can be replaced with any Pod name in the Kafka cluster, and the test can be performed.