Logstash

Logstash is a lightweight log collection and processing framework that can easily collect scattered and diversified logs, perform custom processing, and then transfer them to a designated location, such as a server or file.

Docker install Logstash.

1: Find the mirror: docker search Logstash.

docker search Logstash

2: Download the mirror:

docker pull logstash:5.6

Note: To view the version of the mirror and the TAG, you need to view the address on the docker hub as follows: https://hub.docker.com After entering, search in the search box in the upper left corner of the page

3: Check the image we downloaded: docker images

4: Run

Reference: https://blog.csdn.net/qq_33547169/article/details/86629261

Create a folder (for mounting container files)

mkdir /usr/local/logstash/config

Create related configuration files

logstash.yml (empty file will do)
log4j2.properties

logger.elasticsearchoutput.name = logstash.outputs.elasticsearch

logger.elasticsearchoutput.level = debug

pipelines.yml (the small bar is very important)

- pipeline.id: my-logstash  
  path.config: "/usr/share/logstash/config/*.conf"  
  pipeline.workers: 3

*.conf file

input {    
        jdbc {      
                # mysql jdbc connection string to our backup databse  后面的test 对应mysql中的test数据库      
                jdbc_connection_string =>  "jdbc:mysql://127.0.0.1:3306/tensquare_article?characterEncoding=UTF8"      
                # the user we wish to excute our statement as      
                jdbc_user => "root"      
                jdbc_password => "123456"      
                # the path to our downloaded jdbc driver        
                jdbc_driver_library => "D:/logstash‐5.6.8/mysqletc/mysqlconnector‐java‐5.1.46.jar"      
                # the name of the driver class for mysql      
                jdbc_driver_class => "com.mysql.jdbc.Driver"      
                jdbc_paging_enabled => "true"      
                jdbc_page_size => "50000"      
                #以下对应着要执行的sql的绝对路径。      
                statement => "select id,title,content from tb_article"      
                #定时字段 各字段含义（由左至右）分、时、天、月、年，全部为*默认含义为 每分钟都更新        
                schedule => "* * * * *"    
                }  
        } 
output {    
        elasticsearch {      
                #ESIP地址与端口      
                hosts => "localhost:9200"       
                #ES索引名称（自己定义的）      
                index => "tensquare"      
                #自增ID编号      
                document_id => "%{id}"      
                document_type => "article"    
                }    
                stdout {        
                #以JSON格式输出        
                codec => json_lines    
                        }  
                }

Start and run:

docker run -d -p 5044:5044 -p 9600:9600 -it -v /usr/local/logstash/config/:/usr/share/logstash/config/  logstash:5.6

Docker install Logstash