kettle deployment

1. Put jmbi sql on the production environment first, refer to the attachment jmbi.sql

2. Kettle installation and deployment
Select the target machine, install jdk1.8, then log in to the test environment 192.168.1.142, copy the kettle installation file, job, shell script
scp -r /d1/usr/kettle root@target IP:/d1/usr/kettle
scp -r /root/.kettle root@target IP:/root/.kettle
scp -r /root/.pentaho root@target IP:/root /.pentaho

Modify the kettle.properties in the .kettle folder to configure the DB connection in the production environment, hdfs
modify all the conversion tasks in the job folder in the kettle installation directory. The hdfs:// in the .ktr file [email protected]:9000 is changed to the production environment. You can
also replace the job upload in the attachment with the original job


installation and add task scheduling on the kettle cluster.
vim /etc/crontab
0 0 * * * root sh /d1/usr /kettle/ktlsh/jdt_custom.sh >> /d1/usr/kettle/ktllog/jdt_custom_log
5 0 * * * root sh /d1/usr/kettle/ktlsh/jdt_base_ciq.sh >> /d1/usr/kettle/ktllog/ jdt_baseciq_log
10 0 * * * root sh /d1/usr/kettle/ktlsh/jdt_view.sh >> /d1/usr/kettle/ktllog/jdt_view_log
15 0 * * * root sh /d1/usr/kettle/ktlsh/jdt_increase.sh >> /d1/usr/kettle/ktllog/jdt_increase_log


3.Hbase build table, refer to attachment jmbi_hbase.txt


4.MR deployment
Copy jmframework-1.0.1.jar to all installation directories in the hadoop cluster /d1/usr/hadoop Copy the rest of the jar packages in /share/hadoop/common/
to /d1/dailytask/ to the hadoop master (10.88.13.16) machine, such as 192.168.1.141 in the test environment, just this one is enough
jmbitaskmr.sh to the hadoop master (10.88.13.16),
task scheduling
vim /etc/crontab
50 2 * * * root sh /d1/jmbtaskmr.sh


5.service deployment, operation and maintenance routine deployment


6.web deployment, operation and maintenance routine deployment

7. data collection spider Regular deployment, adding SMS reminder function