Logstash：使用ELK堆栈进行API分析

通过API提取数据并不是ELK Stack用户最常见的用例之一，但这种情况对于有些案例非常有用。 例如，使用REST API包装其数据库服务的开发人员可能对出于商业智能目的而分析此数据感兴趣。在今天的文章中，我们来讲述如何使用ELK堆栈来对一个Service API来进行分析。

无论出于何种原因，ELK Stack都提供了一些简单的方法来与该API集成。 这些方法之一是Logstash HTTP poller 输入插件。 在下面的示例中，我将使用此插件获取一些通过公共API公开的天气数据。

要执行这些步骤并为ELK中的HTTP API分析构建自己的管道，您需要进行以下设置：

• Elastic：菜鸟上手指南：安装好自己的Elasticsearch及Kibana
• 如何安装Elastic栈中的Logstash：安装好自己的Logstash
• 在 https://openweathermap.org/api 链接注册一个账号。你可以在https://openweathermap.org/guide处找到注册的方式及在https://home.openweathermap.org/api_keys 找到自己的API keys。 

准备数据

如上所述，我们使用OpenWeatherMap API收集当前的天气数据。 该数据集包括从40,000个气象站收集的有关全世界现有天气状况的统计信息。

数据本身可以JSON，XML或HTML格式提供，但默认格式为JSON，这非常适合我们的用例，因为我们正在使用Elasticsearch索引数据。 调用数据的方式有多种-您可以通过提供城市名称（我们将在此处使用的方法），城市ID，地理坐标或邮政编码来进行调用。 您还可以在特定坐标内呼叫多个城市。

下面的示例在一个包含经度和纬度坐标的边界框中调用天气统计信息：

http://api.openweathermap.org/data/2.5/box/city?bbox=12,32,15,37,10&appid=YourAppKey

在上面， bbox定义为[lon-left,lat-bottom,lon-right,lat-top,zoom]，也即它表示一个长方形的区域。在上面的appid中，我们必须填入自己的App Key才可以工作。

API返回的数据为我们提供了测量当前天气状况所需的所有统计信息，包括温度，湿度，大气压力，风速和风向等。

这是一个例子：

{"cod":200,"calctime":0.00256473,"cnt":15,"list":[{"id":2563191,"dt":1581912559,"name":"Birkirkara","coord":{"Lon":14.46,"Lat":35.9},"main":{"temp":8,"feels_like":6.94,"temp_min":8,"temp_max":8,"pressure":1028,"humidity":93},"visibility":10000,"wind":{"speed":0.5},"rain":null,"snow":null,"clouds":{"today":1},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}]},{"id":2210247,"dt":1581912558,"name":"Tripoli","coord":{"Lon":13.19,"Lat":32.88},"main":{"temp":14.62,"feels_like":12.78,"temp_min":14.62,"temp_max":14.62,"pressure":1027,"sea_level":1027,"grnd_level":1027,"humidity":59},"wind":{"speed":1.53,"deg":42},"rain":null,"snow":null,"clouds":{"today":12},"weather":[{"id":801,"main":"Clouds","description":"few clouds","icon":"02n"}]},{"id":2216885,"dt":1581912558,"name":"Zawiya","coord":{"Lon":12.73,"Lat":32.75},"main":{"temp":10.33,"feels_like":8.14,"temp_min":10.33,"temp_max":10.33,"pressure":1027,"sea_level":1027,"grnd_level":1015,"humidity":77},"wind":{"speed":1.96,"deg":87},"rain":null,"snow":null,"clouds":{"today":0},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}]},{"id":2212771,"dt":1581912558,"name":"Sabratah","coord":{"Lon":12.49,"Lat":32.79},"main":{"temp":14.59,"feels_like":12.27,"temp_min":14.59,"temp_max":14.59,"pressure":1027,"sea_level":1027,"grnd_level":1027,"humidity":57},"wind":{"speed":2.05,"deg":57},"rain":null,"snow":null,"clouds":{"today":0},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}]},{"id":2215163,"dt":1581912558,"name":"Masallatah","coord":{"Lon":14,"Lat":32.62},"main":{"temp":8.75,"feels_like":6.93,"temp_min":8.75,"temp_max":8.75,"pressure":1027,"sea_level":1027,"grnd_level":995,"humidity":77},"wind":{"speed":0.98,"deg":309},"rain":null,"snow":null,"clouds":{"today":0},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}]},{"id":2219905,"dt":1581912558,"name":"Al Khums","coord":{"Lon":14.26,"Lat":32.65},"main":{"temp":12.93,"feels_like":11.96,"temp_min":12.93,"temp_max":12.93,"pressure":1027,"sea_level":1027,"grnd_level":1023,"humidity":71},"wind":{"speed":0.65,"deg":51},"rain":{"3h":0.19},"snow":null,"clouds":{"today":27},"weather":[{"id":500,"main":"Rain","description":"light rain","icon":"10n"}]},{"id":2208425,"dt":1581912558,"name":"Zuwarah","coord":{"Lon":12.08,"Lat":32.93},"main":{"temp":12.65,"feels_like":10.83,"temp_min":12.65,"temp_max":12.65,"pressure":1027,"sea_level":1027,"grnd_level":1027,"humidity":64},"wind":{"speed":1.3,"deg":63},"rain":null,"snow":null,"clouds":{"today":0},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}]},{"id":2210221,"dt":1581912558,"name":"Tarhuna","coord":{"Lon":13.63,"Lat":32.44},"main":{"temp":7.58,"feels_like":6.11,"temp_min":7.58,"temp_max":7.58,"pressure":1027,"sea_level":1027,"grnd_level":992,"humidity":85},"wind":{"speed":0.56,"deg":142},"rain":null,"snow":null,"clouds":{"today":0},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}]},{"id":2208485,"dt":1581912558,"name":"Zlitan","coord":{"Lon":14.57,"Lat":32.47},"main":{"temp":12.93,"feels_like":11.96,"temp_min":12.93,"temp_max":12.93,"pressure":1027,"sea_level":1027,"grnd_level":1023,"humidity":71},"wind":{"speed":0.65,"deg":51},"rain":{"3h":0.19},"snow":null,"clouds":{"today":27},"weather":[{"id":500,"main":"Rain","description":"light rain","icon":"10n"}]},{"id":2217362,"dt":1581912558,"name":"Gharyan","coord":{"Lon":13.02,"Lat":32.17},"main":{"temp":5.59,"feels_like":4.01,"temp_min":5.59,"temp_max":5.59,"pressure":1027,"sea_level":1027,"grnd_level":940,"humidity":90},"wind":{"speed":0.4,"deg":209},"rain":null,"snow":null,"clouds":{"today":0},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}]},{"id":2523693,"dt":1581912495,"name":"Pozzallo","coord":{"Lon":14.85,"Lat":36.73},"main":{"temp":5.56,"feels_like":2.34,"temp_min":3.89,"temp_max":7.78,"pressure":1028,"humidity":87},"visibility":10000,"wind":{"speed":2.6,"deg":60},"rain":null,"snow":null,"clouds":{"today":0},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}]},{"id":2524119,"dt":1581912495,"name":"Modica","coord":{"Lon":14.77,"Lat":36.85},"main":{"temp":5.73,"feels_like":2.54,"temp_min":3.89,"temp_max":7.78,"pressure":1028,"humidity":87},"visibility":10000,"wind":{"speed":2.6,"deg":60},"rain":null,"snow":null,"clouds":{"today":0},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}]},{"id":2208791,"dt":1581912558,"name":"Yafran","coord":{"Lon":12.53,"Lat":32.06},"main":{"temp":7.01,"feels_like":5.31,"temp_min":7.01,"temp_max":7.01,"pressure":1027,"sea_level":1027,"grnd_level":959,"humidity":84},"wind":{"speed":0.68,"deg":209},"rain":null,"snow":null,"clouds":{"today":0},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}]},{"id":2523581,"dt":1581912494,"name":"Rosolini","coord":{"Lon":14.95,"Lat":36.82},"main":{"temp":5.32,"feels_like":2.06,"temp_min":3.89,"temp_max":7.78,"pressure":1028,"humidity":87},"visibility":10000,"wind":{"speed":2.6,"deg":60},"rain":null,"snow":null,"clouds":{"today":0},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}]},{"id":2523650,"dt":1581912495,"name":"Ragusa","coord":{"Lon":14.72,"Lat":36.93},"main":{"temp":5.52,"feels_like":2.29,"temp_min":3.89,"temp_max":7.78,"pressure":1028,"humidity":87},"visibility":10000,"wind":{"speed":2.6,"deg":60},"rain":null,"snow":null,"clouds":{"today":22},"weather":[{"id":801,"main":"Clouds","description":"few clouds","icon":"02n"}]}]}

配置Logstash

将通过API返回的数据导入到ELK堆栈中的方法是使用Logstash http poller输入插件，该插件可以从定义的终结点URL聚合，解码和运送数据。我们首先来创建一个名字叫做logstash_openweathermap.conf的配置文件。

Logstash input

输入部分定义了http_poller输入插件-轮询的URL端点，请求超时，基于CRON的时间表（每5分钟一次）和要使用的编解码器（JSON）。 metadata_target设置是可选的，并将某些字段添加到有关轮询器性能的响应中。

input {
  http_poller {
    urls => {
      url => "http://api.openweathermap.org/data/2.5/weather?q=London,uk&APPID=7dbe7341764f682c2242e744c4f167b0&units=metric"
    }
    request_timeout => 60
    schedule => { every => "5m"}
    codec => "json"
    metadata_target => "http_poller_metadata"
  }
}

我们要在此处插入的API在调用URL后包含以下查询参数：

• q =London，uk–要求返回伦敦的天气数据
• APPID = – OpenWeatherMap API密钥
• units = metrics –将单位格式转换为摄氏度

上面的url中显示的是微服务的接口。它返回的结果是：

{
  "coord": {
    "lon": -0.13,
    "lat": 51.51
  },
  "weather": [
    {
      "id": 802,
      "main": "Clouds",
      "description": "scattered clouds",
      "icon": "03n"
    }
  ],
  "base": "stations",
  "main": {
    "temp": 6.7,
    "feels_like": 0.46,
    "temp_min": 5,
    "temp_max": 8.33,
    "pressure": 1006,
    "humidity": 65
  },
  "visibility": 10000,
  "wind": {
    "speed": 6.2,
    "deg": 230
  },
  "clouds": {
    "all": 40
  },
  "dt": 1581916140,
  "sys": {
    "type": 1,
    "id": 1414,
    "country": "GB",
    "sunrise": 1581923520,
    "sunset": 1581959845
  },
  "timezone": 0,
  "id": 2643743,
  "name": "London",
  "cod": 200
}

你可以使用https://jsonformatter.org/json-viewer来格式化 API返回的结果。

Logstash filter

由于API响应使用JSON，因此Logstash几乎不需要进行任何处理或解析。 因此，我们现在可以将过滤器部分留空。

filter {}

Logstash output

下面的这部分就非常直接了：

output {
   	elasticsearch {
     	index => "openweather"
     	document_type => "_doc"
     	hosts => "localhost:9200"
	} 
 }

在上面，我们把数据导入到localhost:9200的Elasticsearch中，并且索引的名字是openweather。

综合上面的配置，我们最终的longstash_openweather.conf如下：

input {
  http_poller {
    urls => {
      url => "http://api.openweathermap.org/data/2.5/weather?q=London,uk&APPID=7dbe7341764f682c2242e744c4f167b0&units=metric"
    }
    request_timeout => 60
    schedule => { every => "1m"}
    codec => "json"
    metadata_target => "http_poller_metadata"
  }
}

filter {}

output {
  stdout {
    codec => rubydebug
    }

  elasticsearch {
     	index => "openweather"
     	document_type => "_doc"
     	hosts => "localhost:9200"
  }
}

启动Logstash。我们在Logstash的安装目录运行logstash：

./bin/logstash -f ~/data/logstash_openweather.conf

您将看到使用天气数据创建的新Elasticsearch索引。 在Kibana中定义新的索引模式以开始分析：

分析数据

我们可以通过创建一个叫做openweather的index pattern。然后可以通过Discover查看：

运用 openweather index pattern，我们可以做出来我们的dashboard：