Bboss is a good elasticsearch Java rest client. It operates and accesses elasticsearch in a way similar to mybatis.
BBoss Environmental requirements
JDK requirement: JDK 1.7+
Elasticsearch version requirements: 1.x,2.X,5.X,6.X,+
Spring boot： 1.x,2.x,+
kafka2x-Elasticsearch data synchronization tool demo
Kafka_2.12-2.3.0 series compatible version, use this demo application to run brought the container environment, you can write fast, to distribute a data import tool that can be run
Supported kafka_2.12-2.3.0 series version to elasticsearch data synchronization
kafka low version (kafka_2.12-0.10.2.0 series version) Synchronization Tool Case Address: https://gitee.com/bbossgroups/kafka1x-elasticsearch
Supported Elasticsearch version: 1.x, 2.x, 5.x, 6.x, 7.x, +
PB-level data synchronization support the massive import function
Import coordinate maven
<dependency>
<groupId>com.bbossgroups.plugins</groupId>
<artifactId>bboss-elasticsearch-rest-kafka2x</artifactId> <version>5.9.9</version> <scope>compile</scope> </dependency>
Build deployment
Ready to work
Build a release version by gradle, gradle installation configuration reference documents:
https://esdoc.bbossgroups.com/#/bboss-build
Download Source Project - Based gradle
https://github.com/bbossgroups/kafka2x-elasticsearch
From the above address to download the source code project, and then import the idea or eclipse, according to their needs, modify the import program logic
org.frameworkset.elasticsearch.imp.Kafka2ESdemo
If you need to test and debug the import function, main method is to run Kafka2ESdemo to:
public class Dbdemo {
public static void main(String args[]){ Kafka2ESdemo dbdemo = new Kafka2ESdemo(); boolean dropIndice = true;//CommonLauncher.getBooleanAttribute("dropIndice",false);//同时指定了默认值 dbdemo.scheduleTimestampImportData(dropIndice); } ..... }
Es modify the configuration -kafka2x-elasticsearch \ src \ main \ resources \ application.properties
After modification configuration, you can debug the function.
After the test debugging, you can build the release version can run: into the command line, run under the root directory of the project source kafka2x-elasticsearch the following instructions packaged and released version gradle
release.bat
Running jobs
After gradle successfully constructed, at build / distributions directory will generate the zip package can run decompression operation Importer
linux：
chmod +x restart.sh
./restart.sh
windows: restart.bat
Job jvm configuration
Modify jvm.options, set the memory size and other parameters jvm
-Xms1g
-Xmx1g
Operating parameters
In use kafka2x-elasticsearch , in order to avoid the debugging process continues to distribute a data synchronization tool may be part of the control parameters to the startup configuration file resources / application.properties then acquired by the following method parameters in the code:
# Tool main
MainClass = org.frameworkset.elasticsearch.imp.Kafka2ESdemo # parameters # in code acquisition method: CommonLauncher.getBooleanAttribute ( "dropIndice", false ); // specify default values to false dropIndice = to false
Method dropIndice acquisition parameters in the code:
Boolean dropIndice = CommonLauncher . getBooleanAttribute ( "dropIndice" , false ); // specify the default value of false
May additionally / application.properties configuration control parameters in the job execution resources, such as worker threads, the queue number, batch size and the like:
queueSize=50
workThreads=10
batchSize=20
Acquires the job execution method using the above parameters:
int batchSize = CommonLauncher . getIntProperty ( "batchSize" , 10 ); // specify default values int QueueSize = CommonLauncher . getIntProperty ( "QueueSize" , 50 ); // specify default values int workThreads = CommonLauncher . getIntProperty ( " workThreads " , 10 ); // specify default values importBuilder . setBatchSize ( batchSize ); importBuilder .setQueue ( QueueSize ); // Set the thread pool bulk import queue length importBuilder . setThreadCount ( workThreads ); // Set the number of worker thread pool bulk import
elasticsearch technical exchange group: 166 471 282
elasticsearch micro-channel public number: bbossgroup
Cloud project code: kafka2x-elasticsearch