Flink de l'entrée au parfum réel (10, sortie de données Sink-Elasticsearch)

Objectif: lire les données d'un fichier txt et les écrire dans es. J'utilise es7.9 ici. Si vous utilisez une version antérieure à es7, il y a une catégorie .type ("_ doc") dans le code ci-dessous qui doit être définie

S'il n'y a pas d'environnement es et kibana (facultatif), vous pouvez d'abord l'installer

Installez es7

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.9.3-x86_64.rpm
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.9.3-x86_64.rpm.sha512
shasum -a 512 -c elasticsearch-7.9.3-x86_64.rpm.sha512 
sudo rpm --install elasticsearch-7.9.3-x86_64.rpm
systemctl restart elasticsearch

Installez kibana (facultatif, vous n'avez pas besoin de l'installer si vous ne souhaitez pas utiliser l'interface)

wget https://artifacts.elastic.co/downloads/kibana/kibana-7.9.3-x86_64.rpm
sudo rpm --install kibana-7.9.3-x86_64.rpm

systemctl start kibana

Commencez par introduire la dépendance pom d'Elasticsearch

<dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-connector-elasticsearch7_2.12</artifactId>
    <version>1.10.1</version>
</dependency>

Créer un nouveau ElasticsearchSinkTest.scala

package com.mafei.sinktest

import java.util

import org.apache.flink.api.common.functions.RuntimeContext
import org.apache.flink.streaming.api.scala.{StreamExecutionEnvironment, createTypeInformation}
import org.apache.flink.streaming.connectors.elasticsearch.{ElasticsearchSinkFunction, RequestIndexer}
import org.apache.flink.streaming.connectors.elasticsearch7.ElasticsearchSink
import org.apache.http.HttpHost
import org.elasticsearch.client.Requests

object ElasticsearchSinkTest {
  def main(args: Array[String]): Unit = {
    //创建执行环境
    val env = StreamExecutionEnvironment.getExecutionEnvironment

    val inputStream = env.readTextFile("/opt/java2020_study/maven/flink1/src/main/resources/sensor.txt")
    env.setParallelism(1)
    inputStream.print()

    //先转换成样例类类型
    val dataStream = inputStream
      .map(data => {
        val arr = data.split(",") //按照,分割数据，获取结果
        SensorReadingTest5(arr(0), arr(1).toLong, arr(2).toDouble) //生成一个传感器类的数据，参数中传toLong和toDouble是因为默认分割后是字符串类别
      })

    //定义es的连接信息
    val httpHosts = new util.ArrayList[HttpHost]()
    httpHosts.add(new HttpHost("127.0.0.1", 9200))

    //自定义写入es的ElasticsearchSinkFunction
    val myEsSinkFunc = new ElasticsearchSinkFunction[SensorReadingTest5] {
      override def process(t: SensorReadingTest5, runtimeContext: RuntimeContext, requestIndexer: RequestIndexer): Unit = {
        //定义一个map作为 数据源
        val dataSource = new util.HashMap[String, String]()
        dataSource.put("id", t.id)
        dataSource.put("temperature", t.temperature.toString)
        dataSource.put("ts", t.timestamp.toString)

        //创建index request ，指定index
        val indexRequest = Requests.indexRequest()
        indexRequest.index("sensors") //指定写入哪一个索引
          .source(dataSource) //指定写入的数据
        //            .type("_doc")  //我这里用的es7已经不需要这个参数了

        //执行新增操作
        requestIndexer.add(indexRequest)
      }
    }

    dataStream.addSink(new ElasticsearchSink.Builder[SensorReadingTest5](httpHosts, myEsSinkFunc)
      .build()
    )
    env.execute()
  }
}

Structure du code:

Vérifiez les données sur le serveur. Le capteur est les données que nous venons de remplir.
Vérifiez toutes les données d'index
[root @ localhost ~] # curl http://127.0.0.1:9200/_cat/indices
green open .kibana-event-log-7.9 .3-000001 NvnP2SI9Q_i-z5bNvsgWhA 1 0 1 0 5,5 ko 5,5 kb
jaune capteurs ouverts PGTeT0MZRJ-4hmYkDQnqIw 1 1 6 0
5,4 kb 5,4 kb vert ouvert .apm-custom-link
IdxoOaP9Sh6ss0ssBd0Q9kP9kb 1 0 6 3195434,2kb 434,2kb
vert ouvert .apm-agent-configuration FG9PE8CARdyKWrdsAg4gbA 1 0 0 0 208b 208b
vert ouvert .kibana_1 uVmly8KaQ5uIXZ-IkArnVg 1 0 18 4 10,4mb 10,4m

Afficher les données stockées

[root@localhost ~]# curl http://127.0.0.1:9200/sensors/_search
{"took":0,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":6,"relation":"eq"},"max_score":1.0,"hits":[{"_index":"sensors","_type":"_doc","_id":"h67gkHUBr1E85RDXoNXP","_score":1.0,"_source":{"temperature":"41.0","id":"sensor1","ts":"1603766281"}},{"_index":"sensors","_type":"_doc","_id":"iK7gkHUBr1E85RDXoNXP","_score":1.0,"_source":{"temperature":"42.0","id":"sensor2","ts":"1603766282"}},{"_index":"sensors","_type":"_doc","_id":"ia7gkHUBr1E85RDXoNXP","_score":1.0,"_source":{"temperature":"43.0","id":"sensor3","ts":"1603766283"}},{"_index":"sensors","_type":"_doc","_id":"iq7gkHUBr1E85RDXoNXP","_score":1.0,"_source":{"temperature":"40.1","id":"sensor4","ts":"1603766240"}},{"_index":"sensors","_type":"_doc","_id":"i67gkHUBr1E85RDXoNXP","_score":1.0,"_source":{"temperature":"20.0","id":"sensor4","ts":"1603766284"}},{"_index":"sensors","_type":"_doc","_id":"jK7gkHUBr1E85RDXoNXP","_score":1.0,"_source":{"temperature":"40.2","id":"sensor4","ts":"1603766249"}}]}}