A, broker parameters
Broker.id:kafka uniquely identifies the cluster.
log.dirs: kafka catalog store message log, more comma-separated, need to ensure that the specified directory has sufficient disk space.
zookeeper.connect: you must be configured to specify the address zookeeper cluster kafka cluster registration. The format is: [hostname]: port, multiple separated by commas. When multiple kafka a zookeeper cluster registered to the same cluster must be specified at the end of chroot (ie currently registered kafka zookeeper cluster to the root directory), for
Kafka isolate each cluster, such as: localhost: 2181, localhost: 2182 / kafka_cluster1, / kafka_cluster1 here says chroot.
listeners: The parameter for the connection broker client terminal. Format: [protocol]: // [hostname]: [port], multiple separated by commas. If you do not specify a host name, the default network card binding, if the host name is: 0.0.0.0, then the bind more than a card. kafka current protocol support includes: PLAINTEXT, SSL and SASL_SSL. Is not enabled for security
kafka cluster, use PLAINTEXT it. Whether to use SSL or SASL_SSL.
advertised.listeners: and listeners similar, but also for publishing to clients, but that is mainly used in IaaS environments. The environment of multiple network cards (NICs public network and private network card), binding parameters can be specified public network IP, used for external clients using listeners binding private IP, public use of the internal clients.
unclean.leader.election.enable: whether to open unclean leader election, the default is false. Said that if the ISR becomes empty, leader at a time when there is down, kafka allowed to choose from the remaining non-ISR copy of surviving as a leader, because if allowed, although kafka can continue to provide services to clients, but it can cause data loss .
delete.topic.enable: whether to allow all of the data and delete the topic of management, the new version default allow. This is an asynchronous delete.
log.retention {hours | minutes | ms}:. It is mainly used to set kafka message persistence time, if the same setting, the priority is ms> minutes> hours, the default is stored for seven days.
log.retention.bytes: The main control parameter kafka cluster requires much data for each stored message log. For size exceeds this setting, kafka automatically clean up expired log segment file. The default is 1G.
min.insync.replicas: It is mainly with the use of acks producer. When acks = -1, represents the producer end of the search for a lasting guarantee of the highest level, but only if meaningful min.insync.replicas = -1 acks, it must specify the broker end points clients successfully respond to at least the end of the message the number of pieces to be successful. For chestnut:
Plus the number of each slice 3 is a topic, when min.insync.replicas = 2, a receiving means we kafka machine downtime.
num.network.threads: a broker controls processing threads for background network request, the default 3. Here the "process" is only responsible for forwarding the request, he will forward the received request to subsequent processing thread. In a formal environment, users need to keep monitoring NetworkProcessorAvgIdlePercent JMX index, if the index
Consistently below 0.3, then the need to transfer large parameter.
num.io.threads: This parameter controls the number of threads broker processing network requests end-actual, default 8 ,. kafka default network request to create eight threads polling way non-stop forwarding to listen over and processed. kafka also provides an indicator RequestHandlerAvgIdlePercent JMX monitoring request processing, when the index is less than 0.3,
We need to turn up the parameters.
message.max.bytes: kafka acceptable maximum message size, default is 977KB, the user can be set as desired.
Two, topic parameters
delete.retention.ms: each topic can set their own retention time in order to override the global broker set retention time.
max.message.bytes: override the global message.max.bytes.
retention.bytes: override the global log.retention.bytes.
FAQ
2018-05-02 14:49:18.916 WARN 11916 --- [ntainer#0-3-C-1] org.apache.kafka.clients.NetworkClient : [Consumer clientId=consumer-4, groupId=cwenao-group] Connection to node 0 could not be established. Broker may not be available.
0.10.x broker configuration and advertised.port two abandoned advertised.host.name all configuration items to configure advertised.listeners it.