introduce
FlyFish is a free data visualization orchestration platform open sourced by Cloud Intelligence. Quickly create data models in a simple way, and quickly generate a set of data visualization solutions in the form of drag and drop. In Feiyu products, data can be directly obtained from the data source and displayed on the visualization application by directly connecting to MySQL, Oracle and other databases. This article aims to describe the steps for readers how to connect from the data source to the data display application.
data source type
The types of data sources currently supported by FlyFish are as follows. Among them, MySQL, Postgres, ClickHouse, MariaDB, SqlServer, Dameng, and Oracle are all databases, and you can directly write sql to obtain data. The http data source can fill in information such as request path parameters, and finally process the returned json structure to support application display.
database type data
This article takes the MySQL data source as an example to describe the data source processing process of the database class in detail.
Data source creation
First click the New button on the data source management page, and you can see the following page:
Select the MySQL data source type, and then fill in the corresponding information. The meaning of the specific information is as follows:
- Data source name: Name the newly created data source so that you can find it in the list after it is successfully created.
- Connection address: Fill in the connection address of the MySQL database as shown in the example, pay attention to the correct filling of the ip, port number and db name.
- Database name: This value does not need to be filled in, it will be automatically resolved from the connection address.
- Username: The username to connect to the MySQL database.
- Password: The user password corresponding to the user name used to connect to the MySQL database.
It should be noted that because different databases require different configurations to locate a db, the values to be filled in may be different for different databases. For example, if the Oracle database has the above content, the Schema name needs to be filled in.
After the information is filled in, you can click the link test button to verify whether the current db can be successfully linked. If the content shown in the figure below pops up, it means the connection is successful.
If the following picture pops up, it means that the current db is not connected successfully. You need to check whether the information filled in is correct and whether it is a network problem. For example, the db of some users can only be accessed in their own intranet.
After the test link is successful, click the Save Data Source button to save and set the data source information. You can click Edit to change the data source information, and the Delete button to delete unwanted data sources. On the list, you can see that the data sources that have been successfully saved are as follows:
Data query creation
Creating a good data source is only the first step in using the data source. The next step is to write SQL to get the required content. Click the Data Query button on the sidebar to create a new data query. Click the New Basic Query button and we will see the following page:
If there are too many current data sources, you can enter the previously set data source name for retrieval. Select the previously added data source, and you can see that all data table information will be displayed on the page, as shown in the following figure:
Select the table you want to get data from, and you will be redirected to the page for writing SQL, and the SQL for full table query will be automatically generated as shown in the figure below:
Here you can write the SQL statement you want to implement. It should be noted that this is not limited to querying the currently selected table, but you can still write other table names or do associated queries. In addition, you need to pay attention to the SQL syntax for different types of data sources. For example, the syntax of MySQL is different from that of Oracle, and you need to write identifiable SQL statements according to the type of data source. After writing the SQL, you can click the execute button in the upper right corner of the input to execute the written SQL statement, as shown in the following figure:
Below the input box, you can see the data information returned by the current SQL statement query. If the result does not match the expectation, you can continue to change the SQL to the effect you are satisfied with. In addition, there is a button for setting parameters in the upper right corner of the input box. This button is mainly to solve the problem of SQL reuse. For example, if there are two large screens, one requires an id field and the other requires a name field, so does the user need to write two sentences of SQL? Setting parameters perfectly solves this problem. Let's take a look at how to use setting parameters.
As shown in the figure below, I can set the field I need to query as a parameter called param, that is, use {{}} to wrap the parameter I want to set:
Then click the Set Parameters button, the system will automatically analyze the parameters that need to be set as shown in the pop-up window
Set the type of this parameter here: text, and the default value is: id, fill in a description, and click OK. When the execute button is clicked again, the default value of the parameter will be used, and the new result will be displayed as shown below:
This parameter can be applied to various places in the statement, such as query fields, table names, query conditions, etc. Developers can set variables according to individual needs. How to set such variables on the big screen will be described in detail later.
When the SQL statement is adjusted, click Save in the upper right corner of the page, and the following pop-up window will appear:
It's good to give your query a memorable name so you can find it quickly. Click OK to save successfully. At this point, on the list page of the data query, you can see the newly created and successful data query, as shown in the figure below.
Click the data name to edit the data query. It should be noted that if the data query has been applied to the large screen component, if the returned result after the change is inconsistent with the previous one, it may cause abnormal display on the large screen.
Application of data query
Next, let’s talk about how to apply the configured data query to the large screen.
From the sidebar, click Application Creation->Application Development, you can create a new application or edit it in a previous application, click Development on the application, and enter the following development page:
Take the scrolling list diagram component as an example. Drag a scrolling list component to the editor and select the component, select [Data] in the right sidebar, then select [Data Query] in the data type drop-down, and then enter the previously named data [Query] in the data query input box Name], select it in the search results, and you can find that the previously set parameters will also be displayed below. At this time, you can change the value of the parameter, as shown in the figure below:
In the figure above, you can see that the data query created before is selected, and then the parameter value is changed to name. After clicking Apply, you can see that the information displayed on the component has been changed to the information in the name field. Then save the application. So far, the entire process from configuration to application of the data source has been completed.
http type data
The http data source is different from the data source of the database class and has its own separate logic when added.
Data source creation
On the data source list page, click the New button and select the data source type as http to see the following interface:
Field explanation:
- Data source name: Customize the name of the newly added data source, as long as it meets the requirements.
- Database name: Define the name of a database for subsequent sql processing.
- Request: Select the request type from the drop-down list, and then fill in the corresponding path in the following address bar.
- header request header: fill in the header information of the request.
- Request parameter: fill in the parameter information of the request.
- Request Body: Fill in the request body information.
After filling all the fields, click Save Data Source to save successfully. It should be noted that the current data source will be treated as a db when the subsequent data query is created, so the database name will appear in the SQL query statement, and the naming here must conform to the corresponding rules.
Data table creation
Go back to the data source list page, find the newly created data source and click to enter the data source details page, as shown in the following figure:
Click Add Table in the lower left corner, then enter the name of the custom data table, and click Enter to start building the table. This can be understood as database table building, which is convenient for subsequent SQL processing and table building. The header and parameter information of the data source will be echoed here, and then click the Execute query button to see the data returned by the request in the result preview As shown in FIG. Then you can start to define fields for the returned information for json extraction, click the Add button under the defined fields, and you will see the following pop-up window:
Field explanation:
- Field name: Define a field name, which can be understood as a column name in the database for subsequent sql processing
- Field type: drop down to select the type of field
- Data extraction: fill in the jsonpath path for extracting data in json.
- Data preview: In the process of filling in the jsonpath, the information extracted by the current jsonpath will be displayed in real time
For the specific usage of jsonpath, please refer to: https://blog.csdn.net/software_test010/article/details/125427926. After seeing that the required data has been extracted in the data preview, click OK to save the current field settings. When the required fields are extracted, the rows and columns similar to the database will be displayed as shown below:
The display information of the data table in the figure is the final result of the current data table, click Save. Repeat the above steps to create multiple data tables for the same data source, that is, extract different fields from the returned results to solve the problem of request reuse.
Data query creation
The process of creating a data query is similar to that of a database, please refer to the previous paragraphs.
Click New Data Query, select the created data source and the created data table, and write SQL, as shown in the figure below:
It should be noted that the link table query is not supported here, and the data table created by the Http data source can only be queried in a single table. In addition, the above uses the SQL syntax of Spark, and the bottom layer uses the Spark engine for SQL analysis and processing, so you need to pay attention when writing SQL.
The operation of setting parameters and how to apply the data query to the component for display have been described above and will not be repeated here.
epilogue
The open source data visualization orchestration platform FlyFish is a powerful and easy-to-use data visualization tool that can help users transform large amounts of data into intuitive, easy-to-understand charts and graphs. As a comprehensive data visualization solution, the platform not only provides rich visualization options, but also supports the application technology of multiple data sources.
So far, I believe that all developers should have a general understanding of how to add data sources to the FlyFish platform, and how to obtain the data they want from the data sources and apply them to components. Action is worse than excitement, click the link below, and come and build your own set of big data visualization screens through FlyFish!
GitHub address: https://github.com/CloudWise-OpenSource/FlyFish
Gitee address: https://gitee.com/CloudWise/fly-fish
FlyFish contribution guide: http://bbs.aiops.cloudwise.com/d/717-flyfish
FlyFish Template Center: https://www.cloudwise.ai/flyFishComponents.html
Search xiaoyuerwise on WeChat or scan to identify the QR code below, and note 【Flying Fish】Join the Flying Fish developer exchange group in the AIOps community and communicate face-to-face with the PMC of the FlyFish project.