[Advance notice]
article by author: Zhang Yaofeng in conjunction with their own experience in the production of finishing, forming easy to understand article
writing is not easy, reproduced please specify, thank you!
Code Cases Address: ? HTTPS: //github.com/Mydreamandreality/ sparkResearch
Read the article series: kafka core concepts
The term kafka
- The last chapter we already know some of the terms kafka,
- For example: consumers, producers, theme, this chapter is mainly focused on the interpretation of the meaning of these terms
| The term component | Explanation |
|---|---|
| Theme (Topics) | kafka message is stored in the theme, the theme split into partitions |
| Partition (partition) | Topics may have N number of partitions, so it can handle any number of data |
| Offset partition (partition offset) | Each partition has a message called: "offset" unique identification |
| Backup partition (Replicas of partition) | Partition backup, the backup data is not read or write |
| Broker: agent (Brokers) | Simply put, the agent is a maintenance release of data systems, said the agency is easier to understand the message queue itself, a proxy refers to a kafkaserver, then the data will give agents, consumer spending data from the agent |
| kafka cluster | Acting together is more clusters matter, can heat expansion without downtime, |
| Producers (producers) | Producers kafka theme is to send data to each producer the data to the agent when the agent will be the last message is attached to a section of the file, and is attached to the partition, producers can also send messages to the specified partition |
| Consumers (consumers) | Consumers agent reads from the data, you can subscribe to one or more topics |
| Leader (leader) | leader is responsible for the specified partition is written and read nodes, each partition has a server as a leader |
| Followers (follower) | Follow the leader instruction node is called the Follower, if the leadership fails, a follower will automatically become the new leader |
Kafka cluster architecture as shown in FIG.
| Components nouns | Explanation |
|---|---|
| proxy | kafka cluster composed of multiple agents to maintain load balancing, kafka agent is stateless, so it is necessary to maintain zookeeper cluster state, kafka's leader election can be done by zookeeper |
| zookeeper | zookeeper mainly coordination kafka agent, if there is a new agent or agents fail, zookeeper will receive the message, and production and consumers to take decisions and coordinate their agents and other tasks |
| Producers | Producers push data to the agency |
| consumer | kafka agent is stateless, so consumers need to partition offset by maintaining how many messages have been consumed, if consumers shift to confirm a specific message, it means consumers have consumed all previous messages sent asynchronously to pull consumers to the agent fetch request, and the byte buffer ready for consumption, the consumer may skip at any point in the partition by rapid offset, an offset value notified by zookeeper |
Offset assembly language is defined as:
the distance between the actual address segment address of the memory cell section and its location is referred to as segment offset, also called "effective address or offset." Also: the distance between the actual address of its segment address segment of the memory cell is located. In fact, the essence is the "distance between the segment address where the actual address with its segment"
more popular thing about the way data is stored in memory:
"real address" a data storage segment first address = offset +
Kafka small case:
premise:
- First ensure that our kafka normal start and zookeeper
- [Startup and installation can refer to the content of the previous chapter] ? Hit the jump
- Enter the
jpsverification current process - As shown below, Kafka, Zookeeper normal process, QuorumPeerMain the core promoter is based zookeeper
jps是JDK1.5提供的用于查看当前Java程序进程PID的命令
Create Theme
Code Cases
- kafka provides us with a
bin/kafka-topics.shcommand, we can create a theme on the server with the command
We created a theme name is HelloW orld
command we use local zookeeper create a partition and a copy of
which is also known as a single node single agency model
following command to kafka bin / directory, execute
./kafka-topics.sh --create --zookeeper localhost:2181
--replication-factor 1 --partitions 1 --topic HelloWorld
As shown below: Once you've created will create topics helloworldprompt
Get list of topics
How to Get themes just created, Code Cases
./kafka-topics.sh --list --zookeeper localhost:2181
As shown below: acquired HelloWorld
Producers create, send data to the specified topic
We have just created a named HelloWorldtheme, kafka message is stored in a subject, so we now want to send data to the HelloWorldtopic
Code Case:
- Use
kafka-console-producer.shcommand to create producer
./kafka-console-producer.sh --broker-list localhost:9092
--topic HelloWrold
As shown below:
We need to use this command to create a list of agents, in this case 9092, and the theme sent
After creating producers, we can enter the command messages to be transmitted, the
default behavior of a piece of data, as shown below:
Start consumer, consumer news
We produce the above information, then here in relation to the consumer information
Code Cases
- kafka gives us a
kafka-console-consumer.shcommand, in order to start the consumer
./kafka-console-consumer.sh --bootstrap-server localhost:9092
--topic HelloWorld --from-beginning
The above case is a single node single proxy mode, multi-node multi-agency operation of producers and consumers and more cases are basically the same, but a few require multiple configuration files, create a theme when you need to specify the partition and a copy of the argument, with follow-up to say
How kafka subsequent updates of JavaApi, as well as integration and use Spark
