Elastic Stack开篇
- 1. Elasticsearch简介
- 2. [【Elastic 中国社区官方博客】](https://blog.csdn.net/UbuntuTouch)
- 3. Elasticsearch DSL语法:增删改查
- 4. Kibana入门(构建自己的dashboard)
- 4.1. 数据准备[【百度云盘提取码; ahxe】](https://pan.baidu.com/s/1KWIqfOBxpikUi8iOMyV_yw)
- 4.2. [定义索引模式](https://www.elastic.co/guide/en/kibana/current/tutorial-define-index.html#tutorial-define-index)
- 4.3. [可视化数据和制作dashboard](https://www.elastic.co/guide/en/kibana/current/tutorial-visualizing.html)
- 5. Logstash入门[将数据采集到 Elasticsearch 服务](https://www.elastic.co/guide/en/logstash/current/advanced-pipeline.html)
- 6. Elastic Stack 参考链接
- 7. 寄语:纵浪大化中,不喜亦不惧;重剑无锋,大巧不工!
1. Elasticsearch简介
- Elasticsearch是一个非常强大的搜索引擎
- 事实上Elasticsearch的完整栈有如下的几个:
- Beats
- APM Server
- Elasticsearch
- Elasticsearch Hadoop
- Kibana
- Logstash
- Elastic产品生态
- Elasticsearch的特点是它提供了一个极速的搜索体验。这源于它的高速(speed)。相比较其它的一些大数据引擎,Elasticsearch可以实现秒级的搜索,但是对于它们来说,可能需要数小时才能完成。Elasticsearch的cluster是一种分布式的部署,极易扩展(scale)。这样很容易使它处理petabytes的数据库容量。最重要的是Elasticsearch是它搜索的结果可以按照分数进行排序,它能提供我们最相关的搜索结果(relevance)。
2. 【Elastic 中国社区官方博客】
3. Elasticsearch DSL语法:增删改查
3.1. 访问Elastic Cloud云
3.2. 批量上传文档【百度云盘提取码;ahxe】
curl --user elastic:LqlpkO8LyHT35jPthtZeIoq1 -H 'Content-Type: application/x-ndjson' -XPOST "https://7903c4320e844edd8de8f23ab3a246df.asia-northeast1.gcp.cloud.es.io:9243/bank/_bulk?pretty" --data-binary @accounts.json
注意:文件当中最后要加一个空行
4. Kibana入门(构建自己的dashboard)
4.1. 数据准备【百度云盘提取码; ahxe】
-
创建mapping
PUT /shakespeare { "mappings": { "properties": { "speaker": {"type": "keyword"}, "play_name": {"type": "keyword"}, "line_id": {"type": "integer"}, "speech_number": {"type": "integer"} } } } PUT /logstash-2015.05.20 { "mappings": { "properties": { "geo": { "properties": { "coordinates": { "type": "geo_point" } } } } } }
-
导入数据
curl --user elastic:LqlpkO8LyHT35jPthtZeIoq1 -H 'Content-Type: application/x-ndjson' -XPOST "https://7903c4320e844edd8de8f23ab3a246df.asia-northeast1.gcp.cloud.es.io:9243/bank/_bulk?pretty" --data-binary @accounts.json curl --user elastic:LqlpkO8LyHT35jPthtZeIoq1 -H 'Content-Type: application/x-ndjson' -XPOST "https://7903c4320e844edd8de8f23ab3a246df.asia-northeast1.gcp.cloud.es.io:9243/shakespeare/_bulk?pretty" --data-binary @shakespeare.json curl --user elastic:LqlpkO8LyHT35jPthtZeIoq1 -H 'Content-Type: application/x-ndjson' -XPOST "https://7903c4320e844edd8de8f23ab3a246df.asia-northeast1.gcp.cloud.es.io:9243/_bulk?pretty" --data-binary @logs.jsonl
-
查看elasticsearch中的索引数据
4.2. 定义索引模式
4.3. 可视化数据和制作dashboard
5. Logstash入门将数据采集到 Elasticsearch 服务
-
filebeat.yml的配置文件
- type: log # Change to true to enable this input configuration. enabled: true # Paths that should be crawled and fetched. Glob based paths. paths: - /home/hadoop/fanjh/data/logstash_data/*.log output.logstash: # The Logstash hosts hosts: ["localhost:5044"]
-
logstash.yml的配置文件
# This will overwrite 'var.elasticsearch.hosts' and 'var.kibana.host' cloud.id: FANJIANHAI_ES:YXNpYS1ub3J0aGVhc3QxLmdjcC5jbG91ZC5lcy5pbyQ3OTAzYzQzMjBlODQ0ZWRkOGRlOGYyM2FiM2EyNDZkZiRiZTAwNG M4MzE0NWM0ZGE2OGU5YjNkNDZkODFkY2JmZQ== # # Format of cloud.auth is: <user>:<pass> # This is optional # If supplied this will overwrite 'var.elasticsearch.username' and 'var.elasticsearch.password' # If supplied this will overwrite 'var.kibana.username' and 'var.kibana.password' cloud.auth: elastic:LqlpkO8LyHT35jPthtZeIoq1
-
first-pipeline.conf配置文件