Linux安装elk

其他 2023-06-18 19:22:54 阅读次数: 0

elk搭建笔记，操作步骤太多，所以记录的不详细。搭建elk环境需要看着视频和别人的博客文档一点点搭建。

目录

00【参考资料和视频】

01【安装elk】

es单机

es集群

esHead插件

kibana

logstash

00【参考资料和视频】

Linux-ELK安装包下载地址：

  1. ElasticSearch：https://www.elastic.co/cn/downloads/elasticsearch
  2. Logstash：https://www.elastic.co/cn/downloads/logstash
  3. Kibana：https://www.elastic.co/cn/downloads/kibana
  1. 031-环境-Linux单节点部署_哔哩哔哩_bilibili
  2. day1-02-Elastic Stack简介_哔哩哔哩_bilibili
  1. linux下安装ELK_城南一休的博客-CSDN博客
  2. 大数据周会-本周学习内容总结06【Linux启动elk】
  3. Linux centOS 7部署ELK（elasticSearch、logstash、kibana）

01【安装elk】

es单机

# 加入如下配置
cluster.name: elasticSearch-7.17.6-standalone
node.name: node001
network.host: 0.0.0.0
http.port: 9200
cluster.initial_master_nodes: ["node001"]

path.data: /export/servers/es/elasticsearch-6.7.0/datas
path.logs: /export/servers/es/elasticsearch-6.7.0/logs
discovery.zen.ping.unicast.hosts: ["node01", "node02", "node03"]
bootstrap.system_call_filter: false
bootstrap.memory_lock: false

http.cors.enabled: true
http.cors.allow-origin: "*"

es集群

启动es集群

[es@node001 ~]$ /opt/module/elk/es-cluster/bin/elasticsearch

[es@node002 ~]$ /opt/module/elk/es-cluster/bin/elasticsearch

[es@node003 ~]$ /opt/module/elk/es-cluster/bin/elasticsearch

http://node001:9200/_cat/nodes

# 加入如下配置
#集群名称
cluster.name: cluster-es-7.17.6
#节点名称，每个节点的名称不能重复
node.name: node-1
#ip 地址，每个节点的地址不能重复
network.host: node001
#是不是有资格主节点
node.master: true
node.data: true
http.port: 9200
# head 插件需要这打开这两个配置
http.cors.allow-origin: "*"
http.cors.enabled: true
http.max_content_length: 200mb
#es7.x 之后新增的配置，初始化一个新的集群时需要此配置来选举 master
cluster.initial_master_nodes: ["node-1"]
#es7.x 之后新增的配置，节点发现
discovery.seed_hosts: ["node001:9300","node002:9300","node003:9300"]
gateway.recover_after_nodes: 2
network.tcp.keep_alive: true
network.tcp.no_delay: true
transport.tcp.compress: true
#集群内同时启动的数据任务个数，默认是 2 个
cluster.routing.allocation.cluster_concurrent_rebalance: 16
#添加或删除节点及负载均衡时并发恢复的线程个数，默认 4 个
cluster.routing.allocation.node_concurrent_recoveries: 16
#初始化数据恢复时，并发恢复线程的个数，默认 4 个
cluster.routing.allocation.node_initial_primaries_recoveries: 16


# 加入如下配置
#集群名称
cluster.name: cluster-es-7.17.6
#节点名称，每个节点的名称不能重复
node.name: node-2
#ip 地址，每个节点的地址不能重复
network.host: node002
#是不是有资格主节点
node.master: true
node.data: true
http.port: 9200
# head 插件需要这打开这两个配置
http.cors.allow-origin: "*"
http.cors.enabled: true
http.max_content_length: 200mb
#es7.x 之后新增的配置，初始化一个新的集群时需要此配置来选举 master
cluster.initial_master_nodes: ["node-1"]
#es7.x 之后新增的配置，节点发现
discovery.seed_hosts: ["node001:9300","node002:9300","node003:9300"]
gateway.recover_after_nodes: 2
network.tcp.keep_alive: true
network.tcp.no_delay: true
transport.tcp.compress: true
#集群内同时启动的数据任务个数，默认是 2 个
cluster.routing.allocation.cluster_concurrent_rebalance: 16
#添加或删除节点及负载均衡时并发恢复的线程个数，默认 4 个
cluster.routing.allocation.node_concurrent_recoveries: 16
#初始化数据恢复时，并发恢复线程的个数，默认 4 个
cluster.routing.allocation.node_initial_primaries_recoveries: 16


# 加入如下配置
#集群名称
cluster.name: cluster-es-7.17.6
#节点名称，每个节点的名称不能重复
node.name: node-3
#ip 地址，每个节点的地址不能重复
network.host: node003
#是不是有资格主节点
node.master: true
node.data: true
http.port: 9200
# head 插件需要这打开这两个配置
http.cors.allow-origin: "*"
http.cors.enabled: true
http.max_content_length: 200mb
#es7.x 之后新增的配置，初始化一个新的集群时需要此配置来选举 master
cluster.initial_master_nodes: ["node-1"]
#es7.x 之后新增的配置，节点发现
discovery.seed_hosts: ["node001:9300","node002:9300","node003:9300"]
gateway.recover_after_nodes: 2
network.tcp.keep_alive: true
network.tcp.no_delay: true
transport.tcp.compress: true
#集群内同时启动的数据任务个数，默认是 2 个
cluster.routing.allocation.cluster_concurrent_rebalance: 16
#添加或删除节点及负载均衡时并发恢复的线程个数，默认 4 个
cluster.routing.allocation.node_concurrent_recoveries: 16
#初始化数据恢复时，并发恢复线程的个数，默认 4 个
cluster.routing.allocation.node_initial_primaries_recoveries: 16

esHead插件

ElasticSearch插件es-head安装及使用_es head插件安装_普通网友的博客-CSDN博客

启动插件：

[es@node001 ~]$ cd /opt/module/elk/elasticsearch-head-master/
[es@node001 elasticsearch-head-master]$ npm run start &
[1] 29627
[es@node001 elasticsearch-head-master]$ 
> [email protected] start /opt/module/elk/elasticsearch-head-master
> grunt server

Running "connect:server" (connect) task
Waiting forever...
Started connect web server on http://node001:9100

[es@node001 elasticsearch-head-master]$ 

[es@node001 elasticsearch-head-master]$ npm install
[es@node001 elasticsearch-head-master]$ npm config get registry
[es@node001 elasticsearch-head-master]$ npm info express
[es@node001 elasticsearch-head-master]$ npm install -g cnpm --registry=https://registry.npm.taobao.org
[es@node001 elasticsearch-head-master]$ 
[es@node001 elasticsearch-head-master]$ netstat -nltp | grep 9100
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 192.168.10.101:9100     0.0.0.0:*               LISTEN      87846/grunt         
[es@node001 elasticsearch-head-master]$ 
[es@node001 elasticsearch-head-master]$ 
[es@node001 elasticsearch-head-master]$ history

kibana

[es@node001 ~]$ cd /opt/module/elk/kibana-7.17.6-linux-x86_64/
[es@node001 kibana-7.17.6-linux-x86_64]$ bin/kibana
  log   [15:32:32.515] [info][plugins-service] Plugin "metricsEntities" is disabled.
  log   [15:32:32.652] [info][server][Preboot][http] http server running at http://node001:5601

[es@node001 kibana-7.17.6-linux-x86_64]$ nohup bin/kibana >/dev/null 2>&1 &
[1] 28368
[es@node001 kibana-7.17.6-linux-x86_64]$ 

[es@node001 kibana-7.17.6-linux-x86_64]$ nohup bin/kibana >/dev/null 2>&1 &
[1] 28368
[es@node001 kibana-7.17.6-linux-x86_64]$

logstash

[es@node001 ~]$ cd /opt/module/elk/logstash-7.17.6/
[es@node001 logstash-7.17.6]$ bin/logstash -f /opt/module/elk/logstash-7.17.6/config/json/json01.conf
Using JAVA_HOME defined java: /opt/module/jdk/jdk1.8.0_212
WARNING: Using JAVA_HOME while Logstash distribution comes with a bundled JDK.
DEPRECATION: The use of JAVA_HOME is now deprecated and will be removed starting from 8.0. Please configure LS_JAVA_HOME instead.

[es@node001 logstash-7.17.6]$ bin/logstash -e 'input {stdin{}} output{stdout{codec => rubydebug}}'
Using JAVA_HOME defined java: /opt/module/jdk/jdk1.8.0_212
WARNING: Using JAVA_HOME while Logstash distribution comes with a bundled JDK.
DEPRECATION: The use of JAVA_HOME is now deprecated and will be removed starting from 8.0. Please configure LS_JAVA_HOME instead.

[es@node001 logstash-7.17.6]$ bin/logstash -e 'input {stdin{}} output{stdout{codec => rubydebug}}'
Using JAVA_HOME defined java: /opt/module/jdk/jdk1.8.0_212
WARNING: Using JAVA_HOME while Logstash distribution comes with a bundled JDK.
DEPRECATION: The use of JAVA_HOME is now deprecated and will be removed starting from 8.0. Please configure LS_JAVA_HOME instead.
Sending Logstash logs to /opt/module/elk/logstash-7.17.6/logs which is now configured via log4j2.properties
[2023-05-22T20:48:50,937][INFO ][logstash.runner          ] Log4j configuration path used is: /opt/module/elk/logstash-7.17.6/config/log4j2.properties
[2023-05-22T20:48:50,996][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.17.6", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 Java HotSpot(TM) 64-Bit Server VM 25.212-b10 on 1.8.0_212-b10 +indy +jit [linux-x86_64]"}
[2023-05-22T20:48:50,998][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djdk.io.File.enableADS=true, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true]
[2023-05-22T20:48:51,690][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2023-05-22T20:48:55,848][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2023-05-22T20:48:57,080][INFO ][org.reflections.Reflections] Reflections took 140 ms to scan 1 urls, producing 119 keys and 419 values 
[2023-05-22T20:49:00,863][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["config string"], :thread=>"#<Thread:0x3099c519 run>"}
[2023-05-22T20:49:01,998][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>1.13}
[2023-05-22T20:49:02,254][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
The stdin plugin is now waiting for input:
[2023-05-22T20:49:02,418][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
hello world!^[[D^[[D^[[D^[[D^[[D^[[D^[[D^[[D^[[D^[[D^[[D^[[D
{
       "message" => "hello world!\e[D\e[D\e[D\e[D\e[D\e[D\e[D\e[D\e[D\e[D\e[D\e[D",
      "@version" => "1",
          "host" => "node001",
    "@timestamp" => 2023-05-22T12:49:45.031Z
}
hello,world!
{
       "message" => "hello,world!",
      "@version" => "1",
          "host" => "node001",
    "@timestamp" => 2023-05-22T12:50:01.923Z
}
test logstahsdasdasd
{
       "message" => "test logstahsdasdasd",
      "@version" => "1",
          "host" => "node001",
    "@timestamp" => 2023-05-22T12:50:19.944Z
}
^[[A
{
       "message" => "\e[A",
      "@version" => "1",
          "host" => "node001",
    "@timestamp" => 2023-05-22T12:50:22.392Z
}
test logstash
{
       "message" => "test logstash",
      "@version" => "1",
          "host" => "node001",
    "@timestamp" => 2023-05-22T12:50:28.245Z
}
[2023-05-22T20:51:22,757][INFO ][logstash.javapipeline    ][main] Pipeline terminated {"pipeline.id"=>"main"}
[2023-05-22T20:51:23,005][INFO ][logstash.pipelinesregistry] Removed pipeline from registry successfully {:pipeline_id=>:main}
[2023-05-22T20:51:23,116][INFO ][logstash.runner          ] Logstash shut down.
[es@node001 logstash-7.17.6]$ 
[es@node001 logstash-7.17.6]$ 
[es@node001 logstash-7.17.6]$ bin/logstash -f /opt/module/elk/logstash-7.16.6/config/json/json01.conf
Using JAVA_HOME defined java: /opt/module/jdk/jdk1.8.0_212
WARNING: Using JAVA_HOME while Logstash distribution comes with a bundled JDK.
DEPRECATION: The use of JAVA_HOME is now deprecated and will be removed starting from 8.0. Please configure LS_JAVA_HOME instead.
Sending Logstash logs to /opt/module/elk/logstash-7.17.6/logs which is now configured via log4j2.properties
[2023-05-23T14:48:43,728][INFO ][logstash.runner          ] Log4j configuration path used is: /opt/module/elk/logstash-7.17.6/config/log4j2.properties
[2023-05-23T14:48:43,771][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.17.6", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 Java HotSpot(TM) 64-Bit Server VM 25.212-b10 on 1.8.0_212-b10 +indy +jit [linux-x86_64]"}
[2023-05-23T14:48:43,772][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djdk.io.File.enableADS=true, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true]
[2023-05-23T14:48:44,628][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2023-05-23T14:48:47,652][INFO ][logstash.config.source.local.configpathloader] No config files found in path {:path=>"/opt/module/elk/logstash-7.16.6/config/json/json01.conf"}
[2023-05-23T14:48:47,673][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
[2023-05-23T14:48:47,883][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2023-05-23T14:48:53,417][INFO ][logstash.runner          ] Logstash shut down.
[2023-05-23T14:48:53,439][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?]
        at opt.module.elk.logstash_minus_7_dot_17_dot_6.lib.bootstrap.environment.<main>(/opt/module/elk/logstash-7.17.6/lib/bootstrap/environment.rb:94) ~[?:?]
[es@node001 logstash-7.17.6]$ 


配置文件的路径写错了会出现上述问题。
[es@node001 ~]$ cd /opt/module/elk/logstash-7.17.6/
[es@node001 logstash-7.17.6]$ bin/logstash -f /opt/module/elk/logstash-7.17.6/config/json/json01.conf
Using JAVA_HOME defined java: /opt/module/jdk/jdk1.8.0_212
WARNING: Using JAVA_HOME while Logstash distribution comes with a bundled JDK.
DEPRECATION: The use of JAVA_HOME is now deprecated and will be removed starting from 8.0. Please configure LS_JAVA_HOME instead.

[2023-05-23T15:14:02,119][WARN ][logstash.inputs.jdbc     ][main][93b0d559fcf0f723136838c3814678237baf3f8d907b611ce23407beefea5279] Exception when executing JDBC query {:exception=>Sequel::DatabaseConnectionError, :message=>"Java::ComMysqlJdbcExceptionsJdbc4::CommunicationsException: Communications link failure\n\nThe last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.", :cause=>"com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure\n\nThe last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server."}

[2023-05-23T15:14:02,119][WARN ][logstash.inputs.jdbc     ][main][93b0d559fcf0f723136838c3814678237baf3f8d907b611ce23407beefea5279] Exception when executing JDBC query {:exception=>Sequel::DatabaseConnectionError, :message=>"Java::ComMysqlJdbcExceptionsJdbc4::CommunicationsException: Communications link failure\n\nThe last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.", :cause=>"com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure\n\nThe last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server."}

mysql数据库链接地址写错了！

猜你喜欢

转载自blog.csdn.net/weixin_44949135/article/details/130809687
今日推荐
NetBSD 禁止提交由 AI 生成的代码
Apache Doris 2.0.10 版本正式发布！
开源日报 | 大模型开战；大模型独角兽被曝卖身；周鸿祎建议谷歌开源所有产品；最大开源AI社区提供1000万美元共享GPU
开源日报 | Chrome内置Gemini的意义不在于Gemini；中国AI追随之路的五大误区；ECharts创始人“下海”养鱼；谷歌I/O开发者大会什么都有，只是没有惊喜
微软回应中国区AI团队“打包赴美”传闻
基于大语言模型的开源知识库问答系统 MaxKB GitHub Star 数量突破 5,000 个！
周排行
static方法和非static方法的区别（java）
如何查找计算机专业paper
java.lang.ClassFormatError: Incompatible magic value 0 in class file com/sitecha
跳跃游戏II
stm32_之【建立工程】
TeaWeb v0.0.9 发布，统计底层优化、主机监控功能改进
事件分发 -----控制字体大小
JavaScript DOM练习（动态表格添加） December 25，2019
JSF Scope & CDI
实现从零搭建一个登录注册页面（附源代码）
每日归档
更多
2024-05-19(0)
2024-05-18(4)
2024-05-17(34)
2024-05-16(6)
2024-05-15(24)
2024-05-14(0)
2024-05-13(18)
2024-05-12(0)
2024-05-11(38)
2024-05-10(38)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022