Deploy kettle8.3 source code environment in idea

Table of contents

Pull the specified version of kettle source code and configure its startup environment

Errors encountered during startup

Import plug-ins or custom plug-ins

idea package jar skip test

---

Pull the specified version of kettle source code and configure its startup environment

Pulling the kettle source code needs to configure the specified maven environment (otherwise many dependencies will not be downloaded), the specific setting file can be downloaded from the official website (right click and save as, and then replace the setting file with the one just saved in the maven configuration in the idea file): https://raw.githubusercontent.com/pentaho/maven-parent-poms/master/maven-support-files/settings.xml

The content of this file is as follows, and the following is copied from the link:

<?xml version="1.0" encoding="UTF-8"?>
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
 <localRepository>D:\repo</localRepository>  <!-- 这个是自己的maven本地仓库 -->
  <!-- This is the recommended settings.xml for development of Hitachi Vantara projects. -->

  <!--
  If your wish to mirror everything through pentaho-public's repo uncomment bellow. Not recommended
  for external developers.
  -->
  <!--
  <mirrors>
    <mirror>
      <id>pentaho-public</id>
      <url>https://repo.orl.eng.hitachivantara.com/artifactory/pnt-mvn/</url>
      <mirrorOf>*</mirrorOf>
    </mirror>
  </mirrors>
  -->

  <!--
  Don't forget to add setting-security.xml so that the password get's decrypted  
  -->
  <servers>
    <server>
      <id>pentaho-public</id>
      <username>devreaduser</username>
      <password>{zIMyJWfHKfoHiBJAVsAgW4E5BcJzR+nhTtgPy0J+/rs=}</password>
    </server>
  </servers>

  <!--
  You might want to tweak the 'updatePolicy' configuration to fit your need on having updated snapshots and
  releases. Our recommendation is to set it to 'never' and run maven with the '-U' flag when needed.
  -->
  <profiles>
    <profile>
      <id>pentaho</id>
      <activation>
        <activeByDefault>true</activeByDefault>
      </activation>
      <repositories>
        <repository>
          <id>pentaho-public</id>
          <name>Pentaho Public</name>
          <url>https://repo.orl.eng.hitachivantara.com/artifactory/pnt-mvn/</url>
          <releases>
            <enabled>true</enabled>
            <updatePolicy>always</updatePolicy>
          </releases>
          <snapshots>
            <enabled>true</enabled>
            <updatePolicy>always</updatePolicy>
          </snapshots>
        </repository>
      </repositories>
      <pluginRepositories>
        <pluginRepository>
          <id>pentaho-public</id>
          <name>Pentaho Public</name>
          <url>https://repo.orl.eng.hitachivantara.com/artifactory/pnt-mvn/</url>
          <releases>
            <enabled>true</enabled>
            <updatePolicy>always</updatePolicy>
          </releases>
          <snapshots>
            <enabled>true</enabled>
            <updatePolicy>always</updatePolicy>
          </snapshots>
        </pluginRepository>
      </pluginRepositories>
    </profile>
  </profiles>

  <!-- this lets you call plugins from these groups in their short form -->
  <pluginGroups>
    <pluginGroup>org.pentaho.maven.plugins</pluginGroup>
    <pluginGroup>com.pentaho.maven.plugins</pluginGroup>
    <pluginGroup>com.github.spotbugs</pluginGroup>
  </pluginGroups>
</settings>

Special attention: When pulling code from a remote warehouse, be sure to pay attention to the source version of the pull. You can use the git method of the command line to complete the acquisition of the warehouse code, so that you can specify the version .

git clone -b 8.3 https://github.com/pentaho/pentaho-kettle.git

The network speed of this pulling process is very slow. Even if the overseas acceleration is turned on, it will still get stuck from time to time. 18 minutes here is already considered fast. It is very normal to pull and download for hours.

 The compiled files are in the following path:

Note: Everyone's error report may be different, and it has a lot to do with editing tools and systems.

First set up the running program: In the opened [ Run/Debug Configurations] window, make the following settings:

-Djava.ext.dirs="lib"

The modified spoon program configuration: (just follow the modification below)

①Create a new directory under the source code directory dist, and create a new subdirectory lib in distthe directory

② Unzip the directory zipin the previously compiled release package to the ui directory in ui (the directory structure of this dist will be shown later)ui<源码目录>\dist

③Copy the following files in the directory zipof the previously compiled release package to :lib4jar<源码目录>\lib

dom4j-2.1.1.jar
jaxen-1.1.6.jar
slf4j-api-1.7.7.jar
slf4j-nop-1.7.7.jar

 The dist directory is as follows:

 After the project modification configuration is completed, start: the Spoon class under kettle-ui-swt is running, and an error will inevitably be reported:

Errors encountered during startup

Error 1: Cannot find EventManager.class

1. Solution: Create this class in org.eclipse.core.commands.common.EventManager:

Then copy the following code into the EventManager class you just created:

Pay attention to the package name stored in this class (also created by yourself): 

package org.eclipse.core.commands.common;

// Source code recreated from a .class file by IntelliJ IDEA
// (powered by FernFlower decompiler)
//
import org.eclipse.core.runtime.ListenerList;

public abstract class EventManager {
    private static final Object[] EMPTY_ARRAY = new Object[0];
    private transient ListenerList listenerList = null;

    public EventManager() {
    }

    protected final synchronized void addListenerObject(Object listener) {
        if (this.listenerList == null) {
            this.listenerList = new ListenerList(1);
        }

        this.listenerList.add(listener);
    }

    protected final synchronized void clearListeners() {
        if (this.listenerList != null) {
            this.listenerList.clear();
        }

    }

    protected final Object[] getListeners() {
        ListenerList list = this.listenerList;
        return list == null ? EMPTY_ARRAY : list.getListeners();
    }

    protected final boolean isListenerAttached() {
        return this.listenerList != null;
    }

    protected final synchronized void removeListenerObject(Object listener) {
        if (this.listenerList != null) {
            this.listenerList.remove(listener);
            if (this.listenerList.isEmpty()) {
                this.listenerList = null;
            }
        }

    }
}

 Continue to start and continue to report errors: can't load library: xxxxxxx\swt-gtk-4623.dll

 solution:

Open: <源码目录>\ui\pom.xml:

Do not comment out the global Linux :

 After the comment in swt, the project will be reloaded, and the source code environment may be damaged, so you need to manually refresh maven to re-download the source code zip :

 Then find the model in the project structure, find the kettle-ui-swt module, and click on this module, you can see all the dependencies of this module (as shown in the figure)

Note: In fact, doing the above may sometimes directly move the project structure or refresh the maven project , then the order of win and linux may be reset, so you need to configure it again at this time. Or just adjust the order of these two dependencies when it continues to report this error.

Continue to run, continue to report errors: Error starting spoon shell java.lang.NoClassDefFoundError:org/jaxen/JaxenException

Of course, you don't need to comment, then you need to find the dependent jar yourself, and then import this dependency. The way I use it is to comment out this scope.

Of course, various ClassNotFoundExceptions may be reported:

If you encounter this error, check which jarpackage , libsearch for the required jarpackage the zip after the compilation is successful), and copy it to <源码目录>\dist\libit.

Other errors encountered: Error: Could not find or load main class org.pentaho.di.ui.spoon.Spoon

A solution found on the web: 

It was indeed solved in this way, but I kept refreshing the maven, and then suddenly started to download the source code zip package and some other things after a while, and then this problem no longer appeared.

Continue to run, the operation is successful, and the following page will appear:

Import plug-ins or custom plug-ins

pay attention:

By configuring the source code environment in the above way, the kettle started in this way has no plug-ins (the plug-ins here mean that the plug-ins in the plugins are not in the startup program, and the original basic plug-ins in the engine module will still be there), if If you want to load the plug-in into the program, you need to copy the plug-in (jar) in the compiled zip package to the plugins of the dist (the jar structure package is copied), so that the plug-in and some custom plug-ins can take effect.

 The following is a demonstration of introducing the s3-csv-input plug-in into the operation of kettle: just copy the jar of this component to the plugins in our dist directory.

Custom plugins can also be packaged here for use and debugging.

For the debugging of custom plug-ins, you can refer to my other blog: (1 message) Remote debugging of kettle client and source code environment --- Remote debugging of deployed java projects (idea version)_The future is very bright Long, don't just look at the blog in front of you-CSDN blog

idea package jar skip test

Add a little trick: you can use this thing when you test it yourself. In real projects, you still need to pass the test before it can be packaged.

The install in the idea skips the test link:

There is the following icon in the toolbar of the maven command bar, select test, and then click the lightning symbol;