2, Logstash profile
Introduction 2.1 logstash
LogStash written by JRuby language, simple architecture based on message (message-based) and run on the Java Virtual Machine (JVM). Unlike isolated Agent (Agent) or a host (server), LogStash configurable single Agent (Agent) in conjunction with other open source software, in order to achieve different functions.
Four components of 2.2 logStash
-
Shipper: Send Event (events) to LogStash; typically, a remote Agent (agent) only need to run this component;
-
Broker and Indexer: Index of receiving and events;
-
Search and Storage: allows to search and store events;
-
Web Interface: Web-based interface display
-
It is for these components can be deployed independently in LogStash architecture, it provides a better cluster scalability.
2.3, a software package download URL: https: //www.elastic.co/cn/downloads/logstash
2.4, the downloaded tar archive copy to / application / directory, and create a soft link / application / logstash.
2.5, the asymptotic cycle of learning logstash
2.5.1 start a logstash, -e: execute the command line; input input, stdin standard input, is a plug-in; output output, stdout: standard output. The default is to use the output format rubudebug display detailed output, codec as a codec
[root@harlan_ansible ~]# /application/logstash/bin/logstash -e 'input {stdin{}} output {stdout{}}' OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/application/logstash-7.3.2/logstash-core/lib/jars/jruby-complete-9.2.7.0.jar) to field java.io.FileDescriptor.fd WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release Thread.exclusive is deprecated, use Thread::Mutex Sending Logstash logs to /application/logstash/logs which is now configured via log4j2.properties [2019-09-27T21:37:51,409][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified [2019-09-27T21:37:51,440][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.3.2"} [2019-09-27T21:37:53,020][INFO ][org.reflections.Reflections] Reflections took 88 ms to scan 1 urls, producing 19 keys and 39 values [2019-09-27T21:37:53,865][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge] A gauge metric of an unknown type (org.jruby.RubyArray) has been create for key: cluster_uuids. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team. [2019-09-27T21:37:53,868][INFO ][logstash.javapipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, :thread=>"#<Thread:0x1b23cd0d run>"} [2019-09-27T21:37:53,975][INFO ][logstash.javapipeline ] Pipeline started {"pipeline.id"=>"main"} The stdin plugin isWaiting now for INPUT: [ 2019 - 09 -27T21: 37 [ : 54 is , 143 ] [the INFO] [logstash.agent] {Pipelines running: COUNT => . 1 ,: running_pipelines => [: main],: non_running_pipelines => [] } [ 2019 - 09 -27T21: 37 [ : 54 is , 943 ] [the INFO] [logstash.agent] Successfully Started Logstash the API {Endpoint: Port => 9600 } Hello Word # manually entered string of characters, then the following standard on the screen output. / the Application / logstash- 7.3 . 2/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated { "message" => "hello word", "@version" => "1", "@timestamp" => 2019-09-27T13:38:20.241Z, "host" => "harlan_ansible" }
2.5.2 The output string input screen to elasticsearch service
[root@harlan_ansible ~]# /application/logstash/bin/logstash -e 'input { stdin{} } output { elasticsearch { hosts => ["127.0.0.1:9200"] } }'
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/application/logstash-7.3.2/logstash-core/lib/jars/jruby-complete-9.2.7.0.jar) to field java.io.FileDescriptor.fd WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release Thread.exclusive is deprecated, use Thread::Mutex Sending Logstash logs to /application/logstash/logs which is now configured via log4j2.properties [2019-09-27T21:45:48,670][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified [2019-09-27T21:45:48,693][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.3.2"} [2019-09-27T21:45:50,613][INFO ][org.reflections.Reflections] Reflections took 88 ms to scan 1 urls, producing 19 keys and 39 values [2019-09-27T21:45:51,981][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://127.0.0.1:9200/]}} [2019-09-27T21:45:52,255][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://127.0.0.1:9200/"} [2019-09-27T21:45:52,358][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>7} [2019-09-27T21:45:52,378][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7} [2019-09-27T21:45:52,517][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//127.0.0.1:9200"]} [2019-09-27T21:45:52,765][INFO ][logstash.outputs.elasticsearch] Using default mapping template [2019-09-27T21:45:52,779][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge] A gauge metric of an unknown type (org.jruby.specialized.RubyArrayOneObject) has been create for key: cluster_uuids. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team. [2019-09-27T21:45:52,803][INFO ][logstash.javapipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>250, :thread=>"#<Thread:0x228d3610 run>"} [2019-09-27T21:45:53,020][INFO ][logstash.javapipeline ] Pipeline started {"pipeline.id"=>"main"} [2019-09-27T21:45:53,177][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1, "index.lifecycle.name"=>"logstash-policy", "index.lifecycle.rollover_alias"=>"logstash"}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}} The stdin plugin is now waiting for input: [2019-09-27T21:45:53,389][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/logstash [2019-09-27T21:45:53,499][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]} [2019-09-27T21:45:54,409][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600} [2019-09-27T21:45:55,368][INFO ][logstash.outputs.elasticsearch] Creating rollover alias <logstash-{now/d}-000001> [2019-09-27T21:45:56,899][INFO ][logstash.outputs.elasticsearch] Installing ILM policy {"policy"=>{"phases"=>{"hot"=>{"actions"=>{"rollover"=>{"max_size"=>"50gb", "max_age"=>"30d"manually enter a string. Hello #Policy}}}}}} To _ilm Policy / / logstash-
Through a browser to access the address: http://10.0.0.169:9200/_search?pretty
Congratulations, you've successfully exploited so far Elasticsearch and Logstash to collect the log data.
2.6, log collection system conf
file placed in the conf / application / logstash / bin / directory, the specific configuration is as follows:
input { file { path => "/var/log/messages" type => "system" start_position => "beginning" } file { path => "/application/es/to/logs/elasticsearch.log" type => "es-error" start_position => "beginning" } } output { if [type] == "system" { elasticsearch { hosts => ["10.0.0.169:9200"] index => "system-%{+YYYY.MM.dd}" } } if [type] == "es-error" { elasticsearch { hosts => ["10.0.0.169:9200"] index => "es-error-%{+YYYY.MM.dd}" } } }
Logstash execute the command to start the service:
/application/logstash/bin/logstash -f logstash.conf