docker安装logstash7.3.2

 注:按顺序运行,不要一次性把配置文件全都配置好,然后直接启动,这个因为需要安装插件,所以有可能会有问题

docker run -d --restart=always -p 5044:5044 --name logstash logstash:7.3.2

拷贝配置文件:

docker mkdir -p /usr/local/docker/logstash
docker cp logstash:/usr/share/logstash/config /usr/local/docker/logstash/
docker cp logstash:/usr/share/logstash/pipeline /usr/local/docker/logstash/
docker cp logstash:/usr/share/logstash/Gemfile /usr/local/docker/logstash/Gemfile

启动:

#先删除旧容器
docker rm -f logstash
#以挂载方式启动
docker run -d --restart=always -p 5044:5044 --name logstash \
-v /usr/local/docker/logstash/config:/usr/share/logstash/config \
-v /usr/local/docker/logstash/pipeline:/usr/share/logstash/pipeline \
-v /usr/local/docker/logstash/Gemfile:/usr/share/logstash/Gemfile \
logstash:7.3.2

修改Gemfile,并安装输入输出插件
!注意:因为https://rubygems.org/已经停止维护了,所以需要将gem source替换为https://gems.ruby-china.com/

bin/logstash-plugin install logstash-input-jdbc
bin/logstash-plugin install logstash-output-elasticsearch
vi mysql.conf

=============================================================

 修改如下配置文件:

/usr/local/docker/logstash/config/logstash.yml

http.host: "0.0.0.0"
xpack.monitoring.elasticsearch.hosts: [ "http://101.201.150.23:9200" ]

/usr/local/docker/logstash/config/pipelines.yml

- pipeline.id: goods
  path.config: "/usr/share/logstash/pipeline/goods.conf"

/usr/local/docker/logstash/pipeline/goods.conf:

input {
  jdbc {
    jdbc_driver_library => "/usr/share/logstash/pipeline/mysql-connector-java-5.1.48.jar"
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    jdbc_connection_string => "jdbc:mysql://xxxxx:3306/mytest"
    jdbc_user => "xxxxx"
    jdbc_password => "xxxxx"
    schedule => "* * * * *"
    lowercase_column_names => false
    use_column_value => true
    tracking_column_type => "timestamp"
    tracking_column => "update_time"
    statement => "SELECT id,name FROM goods t WHERE t.update_time >= :sql_last_value"
  }
}
output {
	elasticsearch {
		  action => "update"
		  doc_as_upsert => true	
	      # ES的IP地址及端口
	      hosts => ["xxxxx:9200"]
	      # 索引名称 可自定义
	      index => "vc_base"
	      # 需要关联的数据库中有有一个id字段，对应类型中的id
	      document_id => "%{id}"
	}
    stdout {
        # JSON格式输出
        codec => json_lines
    }
}