2.8 Logstash安装与导入数据

安装 Logstash

下载并解压缩 Logstash

笔者下载的是二进制安装包中的 MACOS 版本

Elastic 官方网站：https://www.elastic.co/cn/

Logstash介绍页面：https://www.elastic.co/cn/logstash

Logstash下载页面：https://www.elastic.co/cn/downloads/logstash

准备 logstash.conf 配置文件

运行 Logstash

bin/logstash -f logstash.conf

下载Movielens测试数据集

下载最MovieLens最小测试数据集：https://grouplens.org/datasets/movielens/

编写logstash.conf配置文件

创建 logstash.conf 配置文件并写入一下内容（注意文件路径）

input {
    
    
  file {
    
    
    path => "/Users/mankvis/Work/elasticsearchStudy/ml-latest-small/movies.csv"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}
filter {
    
    
  csv {
    
    
    separator => ","
    columns => ["id","content","genre"]
  }

  mutate {
    
    
    split => {
    
     "genre" => "|" }
    remove_field => ["path", "host","@timestamp","message"]
  }

  mutate {
    
    

    split => ["content", "("]
    add_field => {
    
     "title" => "%{
    
    [content][0]}"}
    add_field => {
    
     "year" => "%{
    
    [content][1]}"}
  }

  mutate {
    
    
    convert => {
    
    
      "year" => "integer"
    }
    strip => ["title"]
    remove_field => ["path", "host","@timestamp","message","content"]
  }

}
output {
    
    
   elasticsearch {
    
    
     hosts => "http://localhost:9200"
     index => "movies"
     document_id => "%{
    
    id}"
   }
  stdout {
    
    }
}

尝试通过 Logstash 导入数据

在 logstash/bin 目录下运行一下命令

sudo ./logstash -f logstash.conf

以下为输出信息

查看已经导入的数据

打开 Kibana 的 Dev Tools 并输入 GET /movies/_search

可以看到下载的测试数据集中的数据已经全部倒入 Elasticsearch 中