Linux installation elk

Others 2023-06-18 19:51:05 views: null

elk builds notes, there are too many operating steps, so the records are not detailed. Building the elk environment requires watching videos and other people's blog documents to build a little bit.

Table of contents

00【Reference materials and videos】

01【Install elk】

es stand-alone

es cluster

esHead plugin

kibana

logstash

00【Reference materials and videos】

Linux-ELK installation package download address:

  1. ElasticSearch：https://www.elastic.co/cn/downloads/elasticsearch
  2. Logstash: https://www.elastic.co/cn/downloads/logstash
  3. Kibana：https://www.elastic.co/cn/downloads/kibana
  1. 031-Environment-Linux Single Node Deployment_哔哩哔哩_bilibili
  2. day1-02-Introduction to Elastic Stack_哔哩哔哩_bilibili
  1. Install ELK under linux_Chengnan Yixiu Blog-CSDN Blog
  2. Big data weekly meeting - summary of learning content this week 06 [Linux boot elk]
  3. Linux centOS 7 version ELK(elasticSearch、logstash、kibana)

01【Install elk】

es stand-alone

# 加入如下配置
cluster.name: elasticSearch-7.17.6-standalone
node.name: node001
network.host: 0.0.0.0
http.port: 9200
cluster.initial_master_nodes: ["node001"]

path.data: /export/servers/es/elasticsearch-6.7.0/datas
path.logs: /export/servers/es/elasticsearch-6.7.0/logs
discovery.zen.ping.unicast.hosts: ["node01", "node02", "node03"]
bootstrap.system_call_filter: false
bootstrap.memory_lock: false

http.cors.enabled: true
http.cors.allow-origin: "*"

es cluster

Start the es cluster

[es@node001 ~]$ /opt/module/elk/es-cluster/bin/elasticsearch

[es@node002 ~]$ /opt/module/elk/es-cluster/bin/elasticsearch

[es@node003 ~]$ /opt/module/elk/es-cluster/bin/elasticsearch

http://node001:9200/_cat/nodes

# 加入如下配置
#集群名称
cluster.name: cluster-es-7.17.6
#节点名称，每个节点的名称不能重复
node.name: node-1
#ip 地址，每个节点的地址不能重复
network.host: node001
#是不是有资格主节点
node.master: true
node.data: true
http.port: 9200
# head 插件需要这打开这两个配置
http.cors.allow-origin: "*"
http.cors.enabled: true
http.max_content_length: 200mb
#es7.x 之后新增的配置，初始化一个新的集群时需要此配置来选举 master
cluster.initial_master_nodes: ["node-1"]
#es7.x 之后新增的配置，节点发现
discovery.seed_hosts: ["node001:9300","node002:9300","node003:9300"]
gateway.recover_after_nodes: 2
network.tcp.keep_alive: true
network.tcp.no_delay: true
transport.tcp.compress: true
#集群内同时启动的数据任务个数，默认是 2 个
cluster.routing.allocation.cluster_concurrent_rebalance: 16
#添加或删除节点及负载均衡时并发恢复的线程个数，默认 4 个
cluster.routing.allocation.node_concurrent_recoveries: 16
#初始化数据恢复时，并发恢复线程的个数，默认 4 个
cluster.routing.allocation.node_initial_primaries_recoveries: 16


# 加入如下配置
#集群名称
cluster.name: cluster-es-7.17.6
#节点名称，每个节点的名称不能重复
node.name: node-2
#ip 地址，每个节点的地址不能重复
network.host: node002
#是不是有资格主节点
node.master: true
node.data: true
http.port: 9200
# head 插件需要这打开这两个配置
http.cors.allow-origin: "*"
http.cors.enabled: true
http.max_content_length: 200mb
#es7.x 之后新增的配置，初始化一个新的集群时需要此配置来选举 master
cluster.initial_master_nodes: ["node-1"]
#es7.x 之后新增的配置，节点发现
discovery.seed_hosts: ["node001:9300","node002:9300","node003:9300"]
gateway.recover_after_nodes: 2
network.tcp.keep_alive: true
network.tcp.no_delay: true
transport.tcp.compress: true
#集群内同时启动的数据任务个数，默认是 2 个
cluster.routing.allocation.cluster_concurrent_rebalance: 16
#添加或删除节点及负载均衡时并发恢复的线程个数，默认 4 个
cluster.routing.allocation.node_concurrent_recoveries: 16
#初始化数据恢复时，并发恢复线程的个数，默认 4 个
cluster.routing.allocation.node_initial_primaries_recoveries: 16


# 加入如下配置
#集群名称
cluster.name: cluster-es-7.17.6
#节点名称，每个节点的名称不能重复
node.name: node-3
#ip 地址，每个节点的地址不能重复
network.host: node003
#是不是有资格主节点
node.master: true
node.data: true
http.port: 9200
# head 插件需要这打开这两个配置
http.cors.allow-origin: "*"
http.cors.enabled: true
http.max_content_length: 200mb
#es7.x 之后新增的配置，初始化一个新的集群时需要此配置来选举 master
cluster.initial_master_nodes: ["node-1"]
#es7.x 之后新增的配置，节点发现
discovery.seed_hosts: ["node001:9300","node002:9300","node003:9300"]
gateway.recover_after_nodes: 2
network.tcp.keep_alive: true
network.tcp.no_delay: true
transport.tcp.compress: true
#集群内同时启动的数据任务个数，默认是 2 个
cluster.routing.allocation.cluster_concurrent_rebalance: 16
#添加或删除节点及负载均衡时并发恢复的线程个数，默认 4 个
cluster.routing.allocation.node_concurrent_recoveries: 16
#初始化数据恢复时，并发恢复线程的个数，默认 4 个
cluster.routing.allocation.node_initial_primaries_recoveries: 16

esHead plugin

ElasticSearch plugin es-head installation and use_es head plugin installation_Ordinary Netizen's Blog-CSDN Blog

Start the plugin:

[es@node001 ~]$ cd /opt/module/elk/elasticsearch-head-master/
[es@node001 elasticsearch-head-master]$ npm run start &
[1] 29627
[es@node001 elasticsearch-head-master]$ 
> [email protected] start /opt/module/elk/elasticsearch-head-master
> grunt server

Running "connect:server" (connect) task
Waiting forever...
Started connect web server on http://node001:9100

[es@node001 elasticsearch-head-master]$ 

[es@node001 elasticsearch-head-master]$ npm install
[es@node001 elasticsearch-head-master]$ npm config get registry
[es@node001 elasticsearch-head-master]$ npm info express
[es@node001 elasticsearch-head-master]$ npm install -g cnpm --registry=https://registry.npm.taobao.org
[es@node001 elasticsearch-head-master]$ 
[es@node001 elasticsearch-head-master]$ netstat -nltp | grep 9100
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 192.168.10.101:9100     0.0.0.0:*               LISTEN      87846/grunt         
[es@node001 elasticsearch-head-master]$ 
[es@node001 elasticsearch-head-master]$ 
[es@node001 elasticsearch-head-master]$ history

kibana

[es@node001 ~]$ cd /opt/module/elk/kibana-7.17.6-linux-x86_64/
[es@node001 kibana-7.17.6-linux-x86_64]$ bin/kibana
  log   [15:32:32.515] [info][plugins-service] Plugin "metricsEntities" is disabled.
  log   [15:32:32.652] [info][server][Preboot][http] http server running at http://node001:5601

[es@node001 kibana-7.17.6-linux-x86_64]$ nohup bin/kibana >/dev/null 2>&1 &
[1] 28368
[es@node001 kibana-7.17.6-linux-x86_64]$ 

[es@node001 kibana-7.17.6-linux-x86_64]$ nohup bin/kibana >/dev/null 2>&1 &
[1] 28368
[es@node001 kibana-7.17.6-linux-x86_64]$

logstash

[es@node001 ~]$ cd /opt/module/elk/logstash-7.17.6/
[es@node001 logstash-7.17.6]$ bin/logstash -f /opt/module/elk/logstash-7.17.6/config/json/json01.conf
Using JAVA_HOME defined java: /opt/module/jdk/jdk1.8.0_212
WARNING: Using JAVA_HOME while Logstash distribution comes with a bundled JDK.
DEPRECATION: The use of JAVA_HOME is now deprecated and will be removed starting from 8.0. Please configure LS_JAVA_HOME instead.

[es@node001 logstash-7.17.6]$ bin/logstash -e 'input {stdin{}} output{stdout{codec => rubydebug}}'
Using JAVA_HOME defined java: /opt/module/jdk/jdk1.8.0_212
WARNING: Using JAVA_HOME while Logstash distribution comes with a bundled JDK.
DEPRECATION: The use of JAVA_HOME is now deprecated and will be removed starting from 8.0. Please configure LS_JAVA_HOME instead.

[es@node001 logstash-7.17.6]$ bin/logstash -e 'input {stdin{}} output{stdout{codec => rubydebug}}'
Using JAVA_HOME defined java: /opt/module/jdk/jdk1.8.0_212
WARNING: Using JAVA_HOME while Logstash distribution comes with a bundled JDK.
DEPRECATION: The use of JAVA_HOME is now deprecated and will be removed starting from 8.0. Please configure LS_JAVA_HOME instead.
Sending Logstash logs to /opt/module/elk/logstash-7.17.6/logs which is now configured via log4j2.properties
[2023-05-22T20:48:50,937][INFO ][logstash.runner          ] Log4j configuration path used is: /opt/module/elk/logstash-7.17.6/config/log4j2.properties
[2023-05-22T20:48:50,996][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.17.6", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 Java HotSpot(TM) 64-Bit Server VM 25.212-b10 on 1.8.0_212-b10 +indy +jit [linux-x86_64]"}
[2023-05-22T20:48:50,998][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djdk.io.File.enableADS=true, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true]
[2023-05-22T20:48:51,690][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2023-05-22T20:48:55,848][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2023-05-22T20:48:57,080][INFO ][org.reflections.Reflections] Reflections took 140 ms to scan 1 urls, producing 119 keys and 419 values 
[2023-05-22T20:49:00,863][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["config string"], :thread=>"#<Thread:0x3099c519 run>"}
[2023-05-22T20:49:01,998][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>1.13}
[2023-05-22T20:49:02,254][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
The stdin plugin is now waiting for input:
[2023-05-22T20:49:02,418][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
hello world!^[[D^[[D^[[D^[[D^[[D^[[D^[[D^[[D^[[D^[[D^[[D^[[D
{
       "message" => "hello world!\e[D\e[D\e[D\e[D\e[D\e[D\e[D\e[D\e[D\e[D\e[D\e[D",
      "@version" => "1",
          "host" => "node001",
    "@timestamp" => 2023-05-22T12:49:45.031Z
}
hello,world!
{
       "message" => "hello,world!",
      "@version" => "1",
          "host" => "node001",
    "@timestamp" => 2023-05-22T12:50:01.923Z
}
test logstahsdasdasd
{
       "message" => "test logstahsdasdasd",
      "@version" => "1",
          "host" => "node001",
    "@timestamp" => 2023-05-22T12:50:19.944Z
}
^[[A
{
       "message" => "\e[A",
      "@version" => "1",
          "host" => "node001",
    "@timestamp" => 2023-05-22T12:50:22.392Z
}
test logstash
{
       "message" => "test logstash",
      "@version" => "1",
          "host" => "node001",
    "@timestamp" => 2023-05-22T12:50:28.245Z
}
[2023-05-22T20:51:22,757][INFO ][logstash.javapipeline    ][main] Pipeline terminated {"pipeline.id"=>"main"}
[2023-05-22T20:51:23,005][INFO ][logstash.pipelinesregistry] Removed pipeline from registry successfully {:pipeline_id=>:main}
[2023-05-22T20:51:23,116][INFO ][logstash.runner          ] Logstash shut down.
[es@node001 logstash-7.17.6]$ 
[es@node001 logstash-7.17.6]$ 
[es@node001 logstash-7.17.6]$ bin/logstash -f /opt/module/elk/logstash-7.16.6/config/json/json01.conf
Using JAVA_HOME defined java: /opt/module/jdk/jdk1.8.0_212
WARNING: Using JAVA_HOME while Logstash distribution comes with a bundled JDK.
DEPRECATION: The use of JAVA_HOME is now deprecated and will be removed starting from 8.0. Please configure LS_JAVA_HOME instead.
Sending Logstash logs to /opt/module/elk/logstash-7.17.6/logs which is now configured via log4j2.properties
[2023-05-23T14:48:43,728][INFO ][logstash.runner          ] Log4j configuration path used is: /opt/module/elk/logstash-7.17.6/config/log4j2.properties
[2023-05-23T14:48:43,771][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.17.6", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 Java HotSpot(TM) 64-Bit Server VM 25.212-b10 on 1.8.0_212-b10 +indy +jit [linux-x86_64]"}
[2023-05-23T14:48:43,772][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djdk.io.File.enableADS=true, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true]
[2023-05-23T14:48:44,628][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2023-05-23T14:48:47,652][INFO ][logstash.config.source.local.configpathloader] No config files found in path {:path=>"/opt/module/elk/logstash-7.16.6/config/json/json01.conf"}
[2023-05-23T14:48:47,673][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
[2023-05-23T14:48:47,883][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2023-05-23T14:48:53,417][INFO ][logstash.runner          ] Logstash shut down.
[2023-05-23T14:48:53,439][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?]
        at opt.module.elk.logstash_minus_7_dot_17_dot_6.lib.bootstrap.environment.<main>(/opt/module/elk/logstash-7.17.6/lib/bootstrap/environment.rb:94) ~[?:?]
[es@node001 logstash-7.17.6]$ 


配置文件的路径写错了会出现上述问题。
[es@node001 ~]$ cd /opt/module/elk/logstash-7.17.6/
[es@node001 logstash-7.17.6]$ bin/logstash -f /opt/module/elk/logstash-7.17.6/config/json/json01.conf
Using JAVA_HOME defined java: /opt/module/jdk/jdk1.8.0_212
WARNING: Using JAVA_HOME while Logstash distribution comes with a bundled JDK.
DEPRECATION: The use of JAVA_HOME is now deprecated and will be removed starting from 8.0. Please configure LS_JAVA_HOME instead.

[2023-05-23T15:14:02,119][WARN ][logstash.inputs.jdbc     ][main][93b0d559fcf0f723136838c3814678237baf3f8d907b611ce23407beefea5279] Exception when executing JDBC query {:exception=>Sequel::DatabaseConnectionError, :message=>"Java::ComMysqlJdbcExceptionsJdbc4::CommunicationsException: Communications link failure\n\nThe last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.", :cause=>"com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure\n\nThe last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server."}

[2023-05-23T15:14:02,119][WARN ][logstash.inputs.jdbc     ][main][93b0d559fcf0f723136838c3814678237baf3f8d907b611ce23407beefea5279] Exception when executing JDBC query {:exception=>Sequel::DatabaseConnectionError, :message=>"Java::ComMysqlJdbcExceptionsJdbc4::CommunicationsException: Communications link failure\n\nThe last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.", :cause=>"com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure\n\nThe last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server."}

Mysql database link address is wrong!

Guess you like

Origin blog.csdn.net/weixin_44949135/article/details/130809687
Recommended
Ranking
error: (-215:Assertion failed) !_img.empty() in function ‘cv::imwrite‘
Database migration between Navicat servers
Minimum number of rotation of the array: Array
balenaEtcher for mac (make a boot disk software) v1.5.67
Custom processing serialization and deserialization in jackson
Mu-en-mask system development software
Mastering Regular Expressions
Find mileage Java--
Web pages can not directly concern the public micro-channel number how to do? A key to arouse public concern number of micro-channel solutions
[CodeForces - 739B] Alyona and a tree Tree + [difference] + bipartite
Daily
More
2024-05-12(28)
2024-05-11(32)
2024-05-10(34)
2024-05-09(32)
2024-05-08(18)
2024-05-07(34)
2024-05-06(6)
2024-05-05(0)
2024-05-04(18)
2024-05-03(8)

Code World

© 2018 codetd.com