need
When Apache NiFi is used to distribute enterprise master data to downstream business systems in real time, the downstream systems include MySQL, PostgreSQL, Oracle and other business systems. Among them, NiFi does not directly support Oracle Upsert semantics, resulting in a large amount of master data updates such as products and materials, through insert-error The -update method distributes Orace data to the downstream, and the performance is insufficient. Therefore, it is planned to implement the Upsert function by customizing the Processor and using Oracle's built-in Merge into syntax.
This article first explains the process of setting up the Apache NiFi custom development Processor environment, and the follow-up article explains how NiFi supports Oracle's built-in Merge into function by rewriting ConvertJsonToSQL or PutDatabaseRecord respectively.
Customize the Processor process experience
This article uses a Mac notebook to demonstrate the development process. If you are using a Windows computer, please use Baidu to complete the 环境准备introduction content of the section.
Environmental preparation
To build a NiFi development environment, you need to complete the following preparatory work:
- Mac computer
- Java
(base) dszhao@IT00039346 / % java -version
java version "1.8.0_251"
Java(TM) SE Runtime Environment (build 1.8.0_251-b08)
Java HotSpot(TM) 64-Bit Server VM (build 25.251-b08, mixed mode)
(base) dszhao@IT00039346 / %
- maven
(base) dszhao@IT00039346 / % mvn -version
Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
Maven home: /Users/dszhao/software/apache-maven-3.6.3
Java version: 1.8.0_251, vendor: Oracle Corporation, runtime: /Library/Java/JavaVirtualMachines/jdk1.8.0_251.jdk/Contents/Home/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "mac os x", version: "10.15.7", arch: "x86_64", family: "mac"
- Git
- idea
- NiFi
This article installs the latest version 1.15.3 of NiFi
# https://www.apache.org/dyn/closer.lua?path=/nifi/1.15.3/nifi-1.15.3-bin.tar.gz
# 下载NiFi
# 本机安装并正常运行，详细过程略
Initialize NiFi maven project
# 创建开发目录
(base) dszhao@IT00039346 source % mkdir nifi-processor
(base) dszhao@IT00039346 source % cd nifi-processor
# s
(base) dszhao@IT00039346 nifi-processor % mvn archetype:generate
Choose a number or apply filter (format: [groupId:]artifactId, case sensitive contains): : nifi
Choose archetype:
1: remote -> org.apache.nifi:nifi-processor-bundle-archetype (-)
2: remote -> org.apache.nifi:nifi-service-bundle-archetype (-)
Choose a number or apply filter (format: [groupId:]artifactId, case sensitive contains): :
Choose a number or apply filter (format: [groupId:]artifactId, case sensitive contains): : 1
Choose org.apache.nifi:nifi-processor-bundle-archetype version:
1: 0.0.2-incubating
2: 0.1.0-incubating
3: 0.2.0-incubating
4: 0.2.1
5: 0.3.0
6: 0.4.0
7: 0.4.1
8: 0.5.0
9: 0.5.1
10: 0.6.0
11: 0.6.1
12: 0.7.0
13: 0.7.1
14: 0.7.2
15: 0.7.3
16: 0.7.4
17: 1.0.0-BETA
18: 1.0.0
19: 1.0.1
20: 1.1.0
21: 1.1.1
22: 1.1.2
23: 1.2.0
24: 1.3.0
25: 1.4.0
Choose a number: 25:
Downloading from cvte: http://mvn.gz.cvte.cn/nexus/content/groups/public/org/apache/nifi/nifi-processor-bundle-archetype/1.4.0/nifi-processor-bundle-archetype-1.4.0.pom
Downloaded from cvte: http://mvn.gz.cvte.cn/nexus/content/groups/public/org/apache/nifi/nifi-processor-bundle-archetype/1.4.0/nifi-processor-bundle-archetype-1.4.0.pom (3.0 kB at 447 B/s)
Downloading from cvte: http://mvn.gz.cvte.cn/nexus/content/groups/public/org/apache/nifi/nifi-maven-archetypes/1.4.0/nifi-maven-archetypes-1.4.0.pom
Downloaded from cvte: http://mvn.gz.cvte.cn/nexus/content/groups/public/org/apache/nifi/nifi-maven-archetypes/1.4.0/nifi-maven-archetypes-1.4.0.pom (1.5 kB at 445 B/s)
Downloading from cvte: http://mvn.gz.cvte.cn/nexus/content/groups/public/org/apache/nifi/nifi/1.4.0/nifi-1.4.0.pom
[WARNING] Could not validate integrity of download from http://mvn.gz.cvte.cn/nexus/content/groups/public/org/apache/nifi/nifi/1.4.0/nifi-1.4.0.pom: Checksum validation failed, no checksums available
[WARNING] Checksum validation failed, no checksums available from cvte for http://mvn.gz.cvte.cn/nexus/content/groups/public/org/apache/nifi/nifi/1.4.0/nifi-1.4.0.pom
Downloaded from cvte: http://mvn.gz.cvte.cn/nexus/content/groups/public/org/apache/nifi/nifi/1.4.0/nifi-1.4.0.pom (98 kB at 19 kB/s)
Downloading from cvte: http://mvn.gz.cvte.cn/nexus/content/groups/public/org/apache/nifi/nifi-processor-bundle-archetype/1.4.0/nifi-processor-bundle-archetype-1.4.0.jar
[WARNING] Could not validate integrity of download from http://mvn.gz.cvte.cn/nexus/content/groups/public/org/apache/nifi/nifi-processor-bundle-archetype/1.4.0/nifi-processor-bundle-archetype-1.4.0.jar: Checksum validation failed, no checksums available
[WARNING] Checksum validation failed, no checksums available from cvte for http://mvn.gz.cvte.cn/nexus/content/groups/public/org/apache/nifi/nifi-processor-bundle-archetype/1.4.0/nifi-processor-bundle-archetype-1.4.0.jar
Downloaded from cvte: http://mvn.gz.cvte.cn/nexus/content/groups/public/org/apache/nifi/nifi-processor-bundle-archetype/1.4.0/nifi-processor-bundle-archetype-1.4.0.jar (12 kB at 1.9 kB/s)
Define value for property 'artifactBaseName': cvte
[INFO] Using property: nifiVersion = 1.4.0
Define value for property 'groupId': com
Define value for property 'artifactId': cvte
Define value for property 'version' 1.0-SNAPSHOT: : 1.0
Define value for property 'package' com.processors.cvte: : nifi
Confirm properties configuration:
artifactBaseName: cvte
nifiVersion: 1.4.0
groupId: com
artifactId: cvte
version: 1.0
package: nifi
Y: : y
[INFO] ----------------------------------------------------------------------------
[INFO] Using following parameters for creating project from Archetype: nifi-processor-bundle-archetype:1.4.0
[INFO] ----------------------------------------------------------------------------
[INFO] Parameter: groupId, Value: com
[INFO] Parameter: artifactId, Value: cvte
[INFO] Parameter: version, Value: 1.0
[INFO] Parameter: package, Value: nifi
[INFO] Parameter: packageInPathFormat, Value: nifi
[INFO] Parameter: package, Value: nifi
[INFO] Parameter: version, Value: 1.0
[INFO] Parameter: artifactBaseName, Value: cvte
[INFO] Parameter: groupId, Value: com
[INFO] Parameter: artifactId, Value: cvte
[INFO] Parameter: nifiVersion, Value: 1.4.0
[INFO] Project created from Archetype in dir: /Users/dszhao/source/nifi-processor/cvte
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 10:26 min
[INFO] Finished at: 2022-03-08T19:09:51+08:00
[INFO] ------------------------------------------------------------------------
(base) dszhao@IT00039346 nifi-processor %
**⚠️Note:** The version of the NiFi development framework generated by this method is 1.4.0.
The current latest version is 1.15.3. The pom file can be modified to refer to the latest version, which will be described in detail later.
Configure the IDEA development environment
Open the maven project created above in the idea
Modify the TODO implement code in MyProcessor.java
@Override
public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
FlowFile flowFile = session.get();
if ( flowFile == null ) {
return;
}
// TODO implement
// 此处新增
System.out.println("This is a custom processor that will receive flow file");
session.transfer(flowFile,MY_RELATIONSHIP);
}
Modify the version number of nifi-nar-bundles in the project pom to 1.15.3
Package the project as a nar compressed file
(base) dszhao@IT00039346 nifi-processor % ll
total 0
drwxr-xr-x 6 dszhao staff 192 Mar 8 19:15 cvte
(base) dszhao@IT00039346 nifi-processor % cd cvte
(base) dszhao@IT00039346 cvte % ll
total 16
-rw-r--r-- 1 dszhao staff 1320 Mar 8 19:15 cvte.iml
drwxr-xr-x 4 dszhao staff 128 Mar 8 19:15 nifi-cvte-nar
drwxr-xr-x 5 dszhao staff 160 Mar 8 19:15 nifi-cvte-processors
-rw-r--r-- 1 dszhao staff 1464 Mar 8 19:09 pom.xml
(base) dszhao@IT00039346 cvte % pwd
/Users/dszhao/source/nifi-processor/cvte
(base) dszhao@IT00039346 cvte % mvn install
# 省略若干下载信息
[INFO] --- maven-install-plugin:2.5.2:install (default-install) @ nifi-cvte-nar ---
[INFO] Installing /Users/dszhao/source/nifi-processor/cvte/nifi-cvte-nar/target/nifi-cvte-nar-1.0.nar to /Users/dszhao/.m2/repository/com/nifi-cvte-nar/1.0/nifi-cvte-nar-1.0.nar
[INFO] Installing /Users/dszhao/source/nifi-processor/cvte/nifi-cvte-nar/pom.xml to /Users/dszhao/.m2/repository/com/nifi-cvte-nar/1.0/nifi-cvte-nar-1.0.pom
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for cvte 1.0:
[INFO]
[INFO] cvte ............................................... SUCCESS [02:55 min]
[INFO] nifi-cvte-processors ............................... SUCCESS [01:08 min]
[INFO] nifi-cvte-nar ...................................... SUCCESS [ 0.162 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 04:05 min
[INFO] Finished at: 2022-03-08T19:26:49+08:00
[INFO] ------------------------------------------------------------------------
If everything is normal, after packaging, generate target, as follows:
(base) dszhao@IT00039346 cvte % ll nifi-cvte-nar/target
total 288
drwxr-xr-x 3 dszhao staff 96 Mar 8 19:26 classes
drwxr-xr-x 3 dszhao staff 96 Mar 8 19:26 maven-archiver
drwxr-xr-x 3 dszhao staff 96 Mar 8 19:26 maven-shared-archive-resources
-rw-r--r-- 1 dszhao staff 146149 Mar 8 19:26 nifi-cvte-nar-1.0.nar
drwxr-xr-x 3 dszhao staff 96 Mar 8 19:26 test-classes
(base) dszhao@IT00039346 cvte % ll nifi-cvte-processors/target/
total 24
drwxr-xr-x 4 dszhao staff 128 Mar 8 19:26 classes
drwxr-xr-x 3 dszhao staff 96 Mar 8 19:26 generated-sources
drwxr-xr-x 3 dszhao staff 96 Mar 8 19:26 generated-test-sources
drwxr-xr-x 3 dszhao staff 96 Mar 8 19:26 maven-archiver
drwxr-xr-x 3 dszhao staff 96 Mar 8 19:26 maven-shared-archive-resources
drwxr-xr-x 3 dszhao staff 96 Mar 8 19:26 maven-status
-rw-r--r-- 1 dszhao staff 9588 Mar 8 19:26 nifi-cvte-processors-1.0.jar
drwxr-xr-x 5 dszhao staff 160 Mar 8 19:26 surefire-reports
drwxr-xr-x 4 dszhao staff 128 Mar 8 19:26 test-classes
(base) dszhao@IT00039346 cvte %
Copy the generated nar to the local NiFi environment
Copy the generated nar file to the lib directory of the nifi installation directory
(base) dszhao@IT00039346 bin % cp /Users/dszhao/source/nifi-processor/cvte/nifi-cvte-nar/target/nifi-cvte-nar-1.0.nar /Users/dszhao/source/nifi-1.15.3/lib/
(base) dszhao@IT00039346 bin % ll
total 368
-rwxr-x---@ 1 dszhao staff 1872 Jan 22 2020 dump-nifi.bat
-rw-r--r-- 1 dszhao staff 146149 Mar 8 19:34 nifi-cvte-nar-1.0.nar
-rwxr-x---@ 1 dszhao staff 1120 Jan 22 2020 nifi-env.bat
-rwxr-x---@ 1 dszhao staff 2719 Jan 22 2020 nifi-env.sh
-rwxr-x---@ 1 dszhao staff 16662 Jan 22 2020 nifi.sh
-rwxr-x---@ 1 dszhao staff 1871 Jan 22 2020 run-nifi.bat
-rwxr-x---@ 1 dszhao staff 1832 Jan 22 2020 status-nifi.bat
(base) dszhao@IT00039346 bin % sh nifi.sh restart
Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_251.jdk/Contents/Home
NiFi home: /Users/dszhao/source/nifi-1.15.3
Bootstrap Config File: /Users/dszhao/source/nifi-1.15.3/conf/bootstrap.conf
2022-03-08 19:34:42,556 INFO [main] org.apache.nifi.bootstrap.Command Apache NiFi is not currently running
Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_251.jdk/Contents/Home
NiFi home: /Users/dszhao/source/nifi-1.15.3
Bootstrap Config File: /Users/dszhao/source/nifi-1.15.3/conf/bootstrap.conf
After restarting nifi, log in to https://127.0.0.1:8443/nifi/ to view
You can see that our custom MyProcessor has been successfully loaded.
Test the custom MyProcessor
This demonstration example simply routes the FlowFile to the downstream Processor through the MY_RELATIONSHIP connection.
Summarize
This article simply demonstrates the process of NiFi custom development Processor, including NiFi processor maven development skeleton construction, the latest version of NiFi application, and NiFi Processor test. Subsequent documents will introduce in detail the rewriting of ConvertJsonToSQL and PutDatabaseRecord respectively to implement the Oracle Merge function, and demonstrate the NiFi Processor custom details through two practical cases.
For any questions in the article, please add WeChat to discuss:
