NIFI synchronizes MySql data source data_to the original database hbase_simultaneously analyzes and processes the data in real time_synchronizes to the cleaning database_actual operation 06---Nifi work notes for big data 0046

 Let’s talk about the requirements first here. We want to take the data out from different data sources. Of course, here is a data source. After taking it out,

It can be seen that the data passes through the GenerateTableFetch processor, obtains the paging SQL, and then executes it through the ExecuteSql processor. After obtaining the data, it converts the data from Avro format to Json format, uses the ConvertAvroToJSON processor, and then converts the multiple pieces of paged data, Split the SplitJson processor, and then the split data can be seen by walking. A line to the right is: directly store the data in the Hbase database, use PutHbaseJSON

processor.

Then the other one goes to the EvaluateJsonPath processor, which is used to extract the specific field data in Json, and then pass the data through ReplaceText to combine into insert SQL, and then import the data into other databases through PutSQL.

The idea is this, so that we can import the data from multiple data sources into multiple data sources. Simple analysis functions can also be added.