background
In daily work, sometimes leaders are interested in certain data statistics and require analysts to take data and submit them regularly. Faced with this requirement, we can design a large screen to display the data that leaders care about in a visual way; now, with the help of the "Generate Echarts Chart H5 Page" component, we can perform a statistics on the data in the database table, and then display the data as a visual chart and send it directly to business users through email, WeChat, DingTalk, etc., realizing fully automated report statistics and ETLCloudsending SQL.
Tool selection
- ClickHouse database
- Docker deployment ETLCloudV2.2
- ETLCloud's library table input component, generate Echarts chart H5 page component, DingTalk message component
Note: The community version is selected here, and Dockerthe deployment method is lightweight and quick to start: docker pull ccr.ccs.tencentyun.com/restcloud/restcloud-etl:V2.2.
buy components
Because the "Generate Echarts chart H5 page" component is not included in ETLCloudthe community version, so first go to the official component library to choose to buy (the component that generates Echartsthe chart H5page is free). Select Chart Statistical Analysis in the official component library-generate Echarts chart H5 page-click buy, and then you can view it on the component page I purchased (login first).
Install components
After the "Generate Echarts Chart H5 Page" component is successfully purchased, it can ETLCloudbe installed on the "Data Processing Components" page of the management background. Select remote installation, and a list of purchased components will appear; here, choose to install under the general component category, and refresh the page after the installation is complete.
Practice: Data Statistics - Visualization - DingTalk Sending
Next, enter our migration practice: the whole process of zero code, visualization, drag and drop, and the whole process of data statistics, chart visualization, and push can be completed with one click of the mouse.
data source
The data source selects the poetry database from the previous article migration ClickHouse.
-- 柱状图-查询唐诗宋词所有作者的作品数量并排序（仅列出作品数量超过1000的作者）
SELECT author, count(*) AS count FROM poetry.poetry GROUP BY author HAVING count >=1000 ORDER BY count DESC;
Create applications and processes
Create an application and fill in the basic application configuration information. Then, create a data flow and fill in the information.
After creating the process, you can click the "Process Design" button to enter the process visualization configuration page.
Visual configuration process
Before configuring the process, briefly introduce the various areas of this configuration page: the left side is the component area, the middle top is the functional area, and most of the middle is the process drawing area. Double-click the component in the drawing area, and you can see the component detailed configuration item area that pops up in a drawer style.
- Database table batch input: ClickHouse
In the input component on the left, select "Library Table Input", drag it to the central process drawing area, and double-click to enter the configuration stage.
Step 1: Select the data source we configured ClickHouseand select poetrythe database.
Step 2: Configure the SQL statement for statistical data.
Step 3: Select the final output field.
Step 4: Data preview, confirm SQLwhether the statement and query result information meet expectations.
- Generate Echarts chart H5 page
In the general components on the left, select "Generate Echarts Chart H5 Page", drag it to the central process drawing area, and double-click to enter the configuration stage.
Fill in the code to generate the Echarts chart at the code of the H5 page. You can refer to Echartsthe official example to complete the chart display; in addition, pay attention to the fields in the statistical results of the previous step, here are authorand , and get the variable value countthrough in the code .${}
<!DOCTYPE html>
<html lang="zh-CN" style="height: 100%">
<head>
<title>柱状图</title>
<meta charset="utf-8">
</head>
<body>
<center>需要修改item.age和item.total为api返回rows行中的字段id
<div id="container" style="height: 500px;width:100%;"></div>
</center>
<script type="text/javascript" src="https://fastly.jsdelivr.net/npm/[email protected]/dist/echarts.min.js">
</script>
<script type="text/javascript">
var dom = document.getElementById('container');
var myChart = echarts.init(dom, null, {
renderer: 'canvas',
useDirtyRect: false
});
var app = {
};
var option;
option = {
xAxis: {
type: 'category',
data: [#foreach($item in $data)
'$!{item.author}', #end
]
},
yAxis: {
type: 'value'
},
series: [{
data: [#foreach($item in $data)
'$!{item.count}', #end
],
type: 'bar',
showBackground: true,
backgroundStyle: {
color: 'rgba(180, 180, 180, 0.2)'
}
}]
};
if (option && typeof option === 'object') {
myChart.setOption(option);
}
window.addEventListener('resize', myChart.resize);
</script>
</body>
</html>
- Dingding news
In the general components on the left, select "DingTalk Message", drag it to the central process drawing area, and double-click to enter the configuration stage. How to open and configure the DingTalk robot is not introduced here. Select Textthe type of message and enter the content of the message. Here, ${pageurl}get the address variable value of the chart page generated in the previous step through .
Note: When configuring the robots in the DingTalk group, pay attention to the matching of keywords and message content. The keywords I configure here are: Tang Poetry and Song Ci.
- Improve the process
Finally, by 流程线connecting the start , library table input , generation of Echarts chart H5 page , DingTalk message , and end components, the process configuration of data statistics, chart visualization, and push is completed. Done~
run process
Save the process and run the process; then you can view the corresponding process log and visually monitor the migration progress.
Note: After being sent to the DingTalk group by the robot, the IPaddress of the link is Dockerthe address of the container, which cannot be accessed directly. Here, after IPreplacing with Dockerthe host's IP, you can directly open the display in the browser.
To confirm that our graph is correct, query the data from ClickHouseagain .
problem record
When running the process, there is an error problem with the installed components: that is to say, the components we installed ourselves are not found. Afterwards, I contacted the official technical staff, sent me a ETLGenerateHtmlPage.classfile, put it in the specified directory, and cn.restcloud.etl.module.plugin.reportran it successfully.
- Error message:
java.lang.Class-java.lang.Exception: node:GenerateHtmlPagedoes not exist, please download it in the component market or define it yourself! - Solution: Copy the component to the specified directory of the container:
/usr/tomcat/webapps/ROOT/WEB-INF/classes/cn/restcloud/etl/module/plugin/report.
# 进入容器
[root@etl ~]# docker exec -it de63b29c71d0 /bin/bash
root@de63b29c71d0:/usr# cd /usr/tomcat/webapps/ROOT/WEB-INF/classes/cn/restcloud/
# 这个目录一开始不存在，先在容器中创建目录
root@de63b29c71d0:/usr/tomcat/webapps/ROOT/WEB-INF/classes/cn/restcloud# mkdir -p etl/module/plugin/report
# 从宿主机复制文件到容器的指定目录
[root@etl ~]# docker cp /opt/ETLGenerateHtmlPage.class de63b29c71d0:/usr/tomcat/webapps/ROOT/WEB-INF/classes/cn/restcloud/etl/module/plugin/report
Successfully copied 6.14kB to de63b29c71d0:/usr/tomcat/webapps/ROOT/WEB-INF/classes/cn/restcloud/etl/module/plugin/report
# 重启ETLCloud服务
[root@etl ~]# docker restart de63b29c71d0
de63b29c71d0
Summarize
The above is the whole process of converting the statistical data in to ETLCloudcharts and sending them to DingTalk, realizing the full automation of report statistics and sending; at the same time, I also experienced how to install and use new components from the official component market.ClickHouseEcharts
Reference
If you have any questions or any bugs are found, please feel free to contact me.
Your comments and suggestions are welcome!