The distributed messaging middleware kafka Lecture

Preliminary Kafka

Scene -> Requirements -> Solutions -> Applications -> Principle

How do I go messaging middleware design - learn / refine

Scenes

Cross-process communication (inter-process production and consumption models)

demand

basic needs

• Sending and receiving messages to achieve.

  NIO communication (serialization / deserialization) - dubbo, avro, protobuf, zk (jute)

• Message storage implementation (persistent / non-persistent)

  Database storage, file storage (disk: sequential read and write, page caching, persistence time (off the plate policy), zero-copy), memory

• Whether to support cross-language (multi-language ecology)

• Acknowledgment messages (confirmation mechanism) -> business logic needs - in the cross-process communication

• Whether to support a cluster

  Achieve their own elections, third party implementation (zk)

High demand

• Whether to support an orderly (business logic)

• Whether to support the transaction message (business logic) -> eventual consistency

• It supports high concurrency and large data storage

• Whether to support the reliability of the memory

• Whether to support multi-protocol

• Whether charges

development of

pub / sub-- financial --TIB (rules)

Non-individual needs, but common needs

IBM websphere mq (Business)

JMS protocol -> Java api-> AMQP (universal)

kafka

1. Origins: LinkedIn activity stream born at the beginning of operational data is to solve the problem of large amounts of data

2. Brief introduction

   Implementation language: scala

    

3. Chart                                               

4. Download and install and start

   Single-node installation:

   sudo wget http://mirrors.tuna.tsinghua.edu.cn/apache/kafka/2.3.0/kafka_2.11-2.3.0.tgz

   be careful:

   • You must install zk (zk must be started at startup), please refer to

     https://www.cnblogs.com/java333/p/11107166.html

   • Remote installation must modify two properties

   # Native IP 
   the Listeners = PLAINTEXT: // 'native IP': 9092 
   #zk address 
   zookeeper.connect = localhost: 2181

           Start command: (server.properties own file copy)
   
                [the root-Master @ K8S bin] # SH kafka-server-start.sh the server.properties                                                                                                                                                                                                                                                                                             

5. Cluster Installation:

   • Multi-machine deployment

   • Pseudo cluster deployment

      

6. Basic Operations