Log collection system architecture and deployment platform ELK-

Note: All software has been unified into  http://op.jieyue.com/softdown/ELK/

1. System Architecture

2.logstash- installation

Binary software installation package, unzip start

Software installation path: / usr / local / logstash

input{
redis {
type => "h3c"
host => "127.0.0.1"
port => 6379
password => "xxxx"
db => 1
data_type => "list"
key => "h3c"
}

redis {
type => "cisco"
host => "127.0.0.1"
port => 6379
password => "xxxx"
db => 4
data_type => "list"
key => "cisco"
}

redis {
type => "netflow"
host => "127.0.0.1"
port => 6379
password => "xxxxx"
db => 2
data_type => "list"
key => "netflow"
}

redis {
type => "sflow"
host => "127.0.0.1"
port => 6379
password => "xxxx"
db => 3
data_type => "list"
key => "sflow"
}

redis {
type => "nginx-log"
host => "127.0.0.1"
port => 6379
password => "xxxxx"
db => 10
data_type => "list"
key => "nginx-log"
}

}

output{
if[type] == "h3c" {
elasticsearch {
hosts=> ["10.50.194.x:9200","10.50.194.x:9200","10.50.194.x:9200"]
index => "%{type}-%{+YYYY.MM.dd}"
}
}

if[type] == "cisco" {
elasticsearch {
hosts=> ["10.50.194.x:9200","10.50.194.x:9200","10.50.194.x:9200"]
index => "%{type}-%{+YYYY.MM.dd}"
}
}

if[type] == "netflow" {
elasticsearch {
hosts=> ["10.50.194.x:9200","10.50.194.x:9200","10.50.194.x:9200"]
index => "%{type}-%{+YYYY.MM.dd}"
}
}

if[type] == "sflow" {
elasticsearch {
hosts=> ["10.50.194.x:9200","10.50.194.x:9200","10.50.194.x:9200"]
index => "%{type}-%{+YYYY.MM.dd}"
}
}

}

3.ES installation

Software packages binary installation, start decompression (three nodes are responsible for storing and managing data, slice 5, 2 copies)

Software installation path: / usr / local / elasticsearch

cat elasticsearch.yml |egrep -v "^#|^$"
cluster.name: cloud-ops
node.name: node-138
path.data: /data/esdata
path.logs: /data/eslogs
network.host: 10.50.194.x
http.port: 9200
discovery.zen.ping.unicast.hosts: ["10.50.194.x", "10.50.194.x", "10.50.194.x"]
http.cors.enabled: true
http.cors.allow-origin: "*"

#head plug-in installation

1.node install
yum install xz -y
tar -xf node-v6.11.0-linux-x64.tar.xz
vi /etc/profile
export PATH=$PATH:/usr/local/node-v6.11.0-linux-x64/bin

cd /usr/local/
git clone https://github.com/mobz/elasticsearch-head.git
npm install grunt --registry=https://registry.npm.taobao.org
vi Gruntfile.js +92
hostname: '0.0.0.0',
##
cd /usr/local/elasticsearch-head/node_modules
npm install grunt-contrib-clean --registry=https://registry.npm.taobao.org
npm install grunt-contrib-concat --registry=https://registry.npm.taobao.org
npm install grunt-contrib-watch --registry=https://registry.npm.taobao.org
npm install grunt-contrib-connect --registry=https://registry.npm.taobao.org
npm install grunt-contrib-copy --registry=https://registry.npm.taobao.org
npm install grunt-contrib-jasmine --registry=https://registry.npm.taobao.org

#start
cd /usr/local/elasticsearch-head/node_modules/grunt/bin
nohup grunt server &

4.kibana AnSo

Binary software installation package, unzip start

Software Location: / usr / local / kibana

cat kibana.yml |egrep -v "^#|^$"
server.port: 9300
server.host: "10.50.194.x"
elasticsearch.url: "http://10.50.194.x:9200"
kibana.index: ".kibana"
logging.dest: /data/kibana-logs/kibana.log
tilemap.url: 'http://webrd02.is.autonavi.com/appmaptile?lang=zh_cn&size=1&scale=1&style=7&x={x}&y={y}&z={z}'

5. Server Information

IP	username		Features
10.50.194.138	Jypp		it is
10.50.194.139	Jypp		it is
10.50.194.140	Jypp		it is
10.50.194.141	Jypp		logstash repeat
10.50.194.142	Jypp		logstash kibana
10.50.194.204	Jypp		logstash、 es-data
10.50.194.205	Jypp		logstash   es-data
10.50.194.161	Jypp		logstash