Elasticsearch核心技术与实战学习笔记系列第二章 Logstash 安装与测试数据导入

一序

本文属于极客时间 Elasticsearch核心技术与实战学习笔记系列。

二安装Logstash

下载路径：https://artifacts.elastic.co/downloads/logstash/logstash-7.6.2.zip

我是使用下载zip，直接解压缩的格式安装的。也可使用homebrew安装。

我为什么使用解压的方式，因为最好是下载与ES相同版本号的logstash，（7.2.0），并解压到相应目录。

homebrew不指定就是最新的版本。没试过不知道兼容性如何。刚开始学还是少折腾自己。

如果使用使用Homebrew进行安装

1.1 首先需要点击Elastic Homebrew存储库

brew tap elastic/tap

1.2 点击Elastic Homebrew存储库后，可以brew install用来安装Logstash的默认发行版

brew install elastic/tap/logstash-full

2.使用Homebrew进行Logstash启动

2.1 要立即启动启动elastic / tap / logstash-full并在登录时重新启动

brew services start elastic/tap/logstash-full

2.2 要运行Logstash，请在前台运行

logstash

官网上还有其他安装方式：

Installing from Package Repositoriesedit

We also have repositories available for APT and YUM based distributions. Note that we only provide binary packages, but no source packages, as the packages are created as part of the Logstash build.

We have split the Logstash package repositories by version into separate urls to avoid accidental upgrades across major versions. For all 7.x.y releases use 7.x as version number.

We use the PGP key D88E42B4, Elastic’s Signing Key, with fingerprint
4609 5ACC 8548 582C 1A26 99A9 D27D 666C D88E 42B4
to sign all our packages. It is available from https://pgp.mit.edu.

APTedit

Download and install the Public Signing Key:
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
You may need to install the apt-transport-https package on Debian before proceeding:
sudo apt-get install apt-transport-https
Save the repository definition to /etc/apt/sources.list.d/elastic-7.x.list:
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
Use the echo method described above to add the Logstash repository. Do not use add-apt-repository as it will add a deb-src entry as well, but we do not provide a source package. If you have added the deb-src entry, you will see an error like the following:
Unable to find expected entry 'main/source/Sources' in Release file (Wrong sources.list entry or malformed file)
Just delete the deb-src entry from the /etc/apt/sources.list file and the installation should work as expected.

Run sudo apt-get update and the repository is ready for use. You can install it with:
sudo apt-get update && sudo apt-get install logstash
See Running Logstash for details about managing Logstash as a system service.

YUMedit

Download and install the public signing key:
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
Add the following in your /etc/yum.repos.d/ directory in a file with a .repo suffix, for example logstash.repo
[logstash-7.x]
name=Elastic repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
And your repository is ready for use. You can install it with:
sudo yum install logstash
The repositories do not work with older rpm based distributions that still use RPM v3, like CentOS5.

See the Running Logstash document for managing Logstash as a system service.

三下载测试数据

下载最MovieLens最小测试数据集：https://grouplens.org/datasets/movielens/
Logstash下载：https://www.elastic.co/cn/downloads/logstash
Logstash参考文档：https://www.elastic.co/guide/en/logstash/current/index.html

因为网速不好，全的文件250M太大了。所以就使用最小的1M哪个测试。

四导入数据

进入到logstash解压缩文件夹。bin下。

创建配置文件如下：

input {
  file {
    path => "/Users/yiruan/dev/elk7/logstash-7.0.1/bin/movies.csv"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}
filter {
  csv {
    separator => ","
    columns => ["id","content","genre"]
  }

  mutate {
    split => { "genre" => "|" }
    remove_field => ["path", "host","@timestamp","message"]
  }

  mutate {

    split => ["content", "("]
    add_field => { "title" => "%{[content][0]}"}
    add_field => { "year" => "%{[content][1]}"}
  }

  mutate {
    convert => {
      "year" => "integer"
    }
    strip => ["title"]
    remove_field => ["path", "host","@timestamp","message","content"]
  }

}
output {
   elasticsearch {
     hosts => "http://localhost:9200"
     index => "movies"
     document_id => "%{id}"
   }
  stdout {}
}

注意，上面的配置文件里面，要把path修改为,你实际的movies.csv路径

#启动Elasticsearch实例，然后启动 logstash，并制定配置文件导入数据
bin ./logstash -f logstash.conf

效果如下：

这是为以后的使用做好数据准备。

好了，本节课就到此结束了。