一、部署规划

1.1 版本说明

操作系统版本	CentOS release 6.10 (Final)
ElasticSearch版本	7.4
logstash版本	7.4
kibana版本	7.4
内核版本	2.6.32-754.el6.x86_64
CPU	8C
MEM	16G

1.2 安装路径规划

说明项	安装路径	挂载点
软件安装目录	ElasticSearch二进制程序	/usr/local/elasticsearch/bin	/
数据文件目录	ElasticSearch数据文件	/data/elasticsearch/data	/data
配置文件目录	ElasticSearch配置文件	/usr/local/elasticsearch/config	/
日志文件目录	ElasticSearch日志文件	/data/elasticsearch/log	/data

1.3 用户与组规划

根据规划配置用户与组信息：

用户名称	组	用户目录
elasticsearch	elasticsearch	/home/elasticsearch

1.4 副本集节点规划

IP、端口规划如下：

角色	协议类型	服务监听端口
elasticsearch	TCP/IP	172.32.1.59:9200/9300、172.32.1.91:9200/9300、172.32.1.140:9200/9300
kibana	TCP/IP	172.32.1.59:5601
logstash	TCP/IP	172.32.1.59:9600

1.5 架构规划

二、Elasticsearch集群部署

2.1 配置映射

cat >> /etc/hosts<<'EOF'
172.32.1.59 node-1
172.32.1.91 node-2
172.32.1.140 node-3
EOF

2.2 创建elasticsearch用户

useradd  elasticsearch
echo 123456|passwd --stdin elasticsearch

2.3 安装elasticsearch软件

tar xf elasticsearch-7.4.0-linux-x86_64.tar.gz -C /usr/local/ 
mv /usr/local/elasticsearch-7.4.0 /usr/local/elasticsearch

2.4 配置环境变量

cat >> /home/elasticsearch/.bash_profile <<'EOF'
PATH=/usr/local/elasticsearch/bin/:$PATH
EOF

2.5 关闭selinux和防火墙

service iptables stop
chkconfig iptables off
setenforce 0 && sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config

2.6 创建目录并授权

mkdir -p /data/elasticsearch
chown -R elasticsearch:elasticsearch /data/elasticsearch
chown -R elasticsearch:elasticsearch /usr/local/elasticsearch

2.7 操作系统调优

内存优化

在/etc/sysctl.conf添加如下内容
fs.file-max=655360
vm.max_map_count=655360

sysctl -p

修改/etc/security/limits.conf

* soft nofile 65536
* hard nofile 65536
* soft nproc 65536
* hard nproc 65536
* soft memlock unlimited
* hard memlock unlimited

修改/etc/security/limits.d/90-nproc.conf

*      soft   nproc   65536

2.8 jvm调优

/usr/local/elasticsearch/config/jvm.options

-Xms1g 调整为内存一半
-Xmx1g 调整为内存一半

2.9 修改配置文件

cat /usr/local/elasticsearch/config/elasticsearch.yml
cluster.name: mycluster
node.name: node-1
node.master: true
path.data: /data/elasticsearch/data
path.logs: /data/elasticsearch/log
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: ["node-1", "node-2","node-3"]
cluster.initial_master_nodes: ["node-1"]
gateway.recover_after_nodes: 1
action.destructive_requires_name: true
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
xpack.license.self_generated.type: basic

2.10 安全加固

在一个master上执行即可

cd /opt/elasticsearch
./bin/elasticsearch-certutil ca
两次回车
./bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 
三次回车

赋予权限

扫描二维码关注公众号，回复： 11577322 查看本文章

mkdir config/certs
mv elastic-*.p12 config/certs/
chown -R elsearch:elsearch config/certs/

再把证书文件 elastic-certificates.p12 复制到其他master节点并赋予权限。

cat >> config/elasticsearch.yml <<EOF
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: certs/elastic-certificates.p12
EOF

创建密码

elasticsearch-setup-passwords interactive

2.11 启动elasticsearch

su - elasticsearch
elasticsearch -d

三、 kibana部署

3.1 授权目录

chown -R elasticsearch:elasticsearch /usr/local/kibana

3.2 配置环境变量

cat >> /home/elasticsearch/.bash_profile <<'EOF'
PATH=/usr/local/kibana/bin/:$PATH
EOF

3.3 修改配置文件

server.port: 5601
server.host: "0.0.0.0"
elasticsearch.hosts: ["http://192.168.1.9:9200","http://192.168.1.9:9200","http://192.168.1.9:9200"]
kibana.index: ".kibana"
i18n.locale: "zh-CN"

3.4 启动kibana

kibana &

四、 logstach安装导入数据

4.1 安装logstach

tar xf logstash-7.4.0.tar.gz -C /usr/local
mv /usr/local/logstash-7.4.0 /usr/local/logstach
chown -R elasticsearch.elasticsearch /usr/local/logstach

4.2 配置环境变量

cat >> /home/elasticsearch/.bash_profile <<'EOF'
PATH=/usr/local/logstash/bin/:$PATH
EOF

4.3 编辑配置文件

vim /usr/local/logstash/config/logstash.conf

input {

 file {

  path => "/home/es/ml-latest-small/movies.csv"

  start_position => "beginning"

  sincedb_path => "/dev/null"

 }

}

filter {

 csv {

  separator => ","

  columns => ["id","content","genre"]

 }

 mutate {

  split => { "genre" => "|" }

  remove_field => ["path", "host","@timestamp","message"]

 }

 mutate {

  split => ["content", "("]

  add_field => { "title" => "%{[content][0]}"}

  add_field => { "year" => "%{[content][1]}"}

 }

 mutate {

  convert => {

   "year" => "integer"

  }

  strip => ["title"]

  remove_field => ["path", "host","@timestamp","message","content"]

 }

}

output {

 elasticsearch {

  hosts => "http://192.168.1.59:9200"

  index => "movies"

  document_id => "%{id}"

 }

 stdout {}

}

4.4 导入数据

su - elasticsearch
logstash -f /usr/local/logstash/config/logstash.conf

ElasticSearch7部署文档

文章目录