This article will introduce how to connect real-time computing Flink with other systems. The introduction consists of four parts, namely:
1. Storage and use of Jar;
2. Real-time calculation of how Flink interacts with some typical data sources;
3. How to inject Flink indicators on the VVP platform into the external system of Metrics;
4. How to import the Flink job logs running on the VVP platform to the external system.
1. How to store the Jar running the job on OSS
There are two ways to upload the job jar on the VVP platform.
Method 1 : With the help of the resource upload function provided by VVP, you can directly use this function to upload Jar. Currently, this function supports the upload of Jar packages within 200 megabytes. When using it, just select the jar package uploaded when creating the job. The demonstration is as follows:
- Enter the VVP platform, click the resource upload function on the left, and then click the upload resource in the upper right corner of the opened page, select the Jar package to upload, and complete the upload;
- After the upload is successful, click Create Job on the left to complete the job name and other information. In the Jar URI column, drop down and select the Jar package you just uploaded, click OK to complete the creation job, and then start it to use.
Method two , upload the Jar to be used directly on the OSS console, and then use the Jar link provided by OSS. It is also relatively simple to use, directly use the Jar link provided by OSS. The demonstration is as follows:
- Open the OSS console, select the bucket used when creating the VVP, then select the directory, and click upload file. When uploading, you can set its permission to public read, and click upload file to complete;
- When using it, click "Details" on the right side of the uploaded package on the OSS console to obtain the URL link of the Jar package.
- When creating a job, fill in the Jar URI with the URL of the jar package, as shown in the following figure:
It should be noted that the link provided on the OSS details page is for public network access, and the opened VVP cannot directly access the public network. Therefore, when creating a job using HTTPS, you need to use the endpoint accessed by the VPC to start the job normally.
What if I want to get an HTTPS link from the public network? You can connect the VVP to the public network first. For the operation process of the connection, please refer to the "How Flink Fully Managed Cluster Access the Public Network" in the Alibaba Cloud help document. The simple steps are as follows:
- First, create a NAT gateway. When creating, select "combined purchase ERP", then select the region and add the name and other information, and then bind the elastic public network IP to complete the creation;
- Second, create a SNAT entry. After creating the NAT, click "Create SNAT Entry", select the switch in the pop-up window and add the name information to complete the creation.
After completing the above two steps, the VVP instance has been connected to the public network, and you can directly use the jar package accessible on the https public network when creating the Deployment.
2. How does Flink interact with typical data sources on the VVP platform
This part introduces how to interact with some external data storage systems through SQL and connectors, taking SLS and Kafka as data sources to read and write data as examples.
(Practical demonstration) Click on the SQL editor to create a Datagen Table, which is used to randomly generate data, and then click Run. Then click to generate an SLS Table, add the required parameter information, and click Create to complete.
After the creation is complete, write SQL statements, such as insert into sls select id, name from datagen, and then save it and click Run to create the Deployment and start it.
When the job runs successfully, query the data on SLS. As shown in the figure below, datagen has generated data and successfully written to SLS.
Similarly, we can follow the above steps to read data from SLS and then write to Kafka:
- Create a Kafka table on the sql editor page of vvp
- Use SQL syntax to read data from SLS and write it to Kafka and start it
- After the job runs successfully, it starts to read data from SLS and write it to Kafka
3. How to integrate Flink's indicators on the VVP platform into the external Metrics system
Next, if you want to put the indicators of running jobs into some systems, and perform indicator observations. VVP provides two methods:
Method 1 : VVP will enter the Flink job indicators into the arms by default. No additional processing is required. After running the job directly, you can see it through the indicator button, as shown in the figure below:
Method 2 : If you have an indicator system and want to integrate Flink's job indicators into your own system, there are two main points: first, ensure the connectivity between the job on the VVP and the network of your own indicator system; secondly, configure the corresponding in Flink conf The metrics reporter. As shown in the figure below, in the process of creating a job, configure metrics (reference for metrics reporters configuration: https://ci.apache.org/projects/flink/flink-docs-release-1.11/monitoring/metrics.html ):
Example: Use the pushGateway method of premetheus, so the reporter class chooses org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter. Configure the port and host of pushGateway as shown in the figure above, and the Metric reporter configuration is complete. After the job is started successfully, check the indicators on the configured grafana disk, as shown in the following example.
Fourth, how to enter the Flink job log into the external system
If the job suddenly fails during the running of the job, and we want to view the log of the failed job, we need to save the log of the Flink job. The VVP platform provides two solutions for this purpose. Write Logs into OSS or SLS. Simply put, when creating a job, configure some Log parameters in the Log configuration item.
Method 1: Write the log to OSS. When creating a job, in the Log configuration in the advanced configuration, choose to use user-defined, then put the configuration in the (help document) in the custom configuration, and then replace some parameters with the necessary parameters of OSS. Okay.
When you need to view the log, you can find the file stored in the log through the guidance of the help document, and then click download to view.
Method two is to write the log into SLS. Similar to method 1, except that the LOG configuration items are slightly different; the downloading and viewing methods are the same as method 1.
This article is the original content of Alibaba Cloud and may not be reproduced without permission.