Apache DolphinSchedule installation and deployment

Others 2021-03-07 04:50:55 views: null

Apache DS installation and deployment

@Author：Michealkz

Mirror download address

Official download address: https://dolphinscheduler.apache.org/zh-cn/download/download.html

Domestic mirror download address:

# 清华源下载地址
https://mirrors.tuna.tsinghua.edu.cn/apache/incubator/dolphinscheduler/1.3.5/apache-dolphinscheduler-incubating-1.3.5-dolphinscheduler-bin.tar.gz

# 北理工源下载地址
https://mirrors.bfsu.edu.cn/apache/incubator/dolphinscheduler/1.3.5/apache-dolphinscheduler-incubating-1.3.5-dolphinscheduler-bin.tar.gz

Deployment mode: stand-alone deployment/cluster deployment

For stand-alone deployment, refer to the official document: https://dolphinscheduler.apache.org/zh-cn/docs/1.3.5/user_doc/quick-start.html

Here is the method of cluster deployment

Cluster deployment (Cluster)

1. Host planning:

Choose three nodes to install DolphinSchedule, the choice of nodes can be free, it is recommended to keep on the same machine as the zookeeper node

192.168.5.72 Monitor DolphinScheduler
192.168.5.19 hdnamenode DolphinScheduler
192.168.5.61 hddatanode01 DolphinScheduler

2. Basic software installation

  • PostgreSQL (8.2.15+) or MySQL (5.7 series): You can choose one of the two, such as MySQL requires JDBC Driver 5.1.47+
  • JDK (1.8+): Required, please configure JAVA_HOME and PATH variables under /etc/profile after installation
  • ZooKeeper (3.4.6+): required
  • Hadoop (2.6+) or MinIO: optional, if you need to use the resource upload function, you can choose to upload to Hadoop or MinIO
 注意：DolphinScheduler本身不依赖Hadoop、Hive、Spark,仅是会调用他们的Client，用于对应任务的提交。

3. Download the binary tar.gz package

  • Please download the latest version of the back-end installation package to the server deployment directory, such as creating /opt/dolphinscheduler as the installation deployment directory, download address: download , upload the tar package to this directory after downloading, and unzip
# 创建部署目录,部署目录请不要创建在/root、/home等高权限目录 
mkdir -p /opt/dolphinscheduler;
cd /opt/dolphinscheduler;

# 解压缩
tar -zxvf apache-dolphinscheduler-incubating-1.3.4-dolphinscheduler-bin.tar.gz -C /opt/dolphinscheduler;
mv apache-dolphinscheduler-incubating-1.3.4-dolphinscheduler-bin  dolphinscheduler-bin

4. Create deployment user and hosts mapping

  • Create deployment users on all deployment scheduling machines, and be sure to configure sudo to avoid passwords. If we plan to deploy scheduling on 4 machines, ds1, ds2, ds3, and ds4, we first need to create deployment users on each machine
# 创建用户需使用root登录，设置部署用户名，请自行修改，后面以dolphinscheduler为例
useradd dolphinscheduler;

# 设置用户密码，请自行修改，后面以dolphinscheduler123为例
echo "dolphinscheduler123" | passwd --stdin dolphinscheduler

# 配置sudo免密
echo 'dolphinscheduler  ALL=(ALL)  NOPASSWD: NOPASSWD: ALL' >> /etc/sudoers
sed -i 's/Defaults    requirett/#Defaults    requirett/g' /etc/sudoers

注意：
 - 因为是以 sudo -u {
    
    linux-user} 切换不同linux用户的方式来实现多租户运行作业，所以部署用户需要有 sudo 权限，而且是免密的。
 - 如果发现/etc/sudoers文件中有"Default requiretty"这行，也请注释掉
 - 如果用到资源上传的话，还需要在`HDFS或者MinIO`上给该部署用户分配读写的权限

5. Configure hosts mapping and ssh to get through and modify directory permissions

  • Use the first machine (hostname is ds1) as the deployment machine, configure the hosts of all machines to be deployed on ds1, and log in as root on ds1
vi /etc/hosts

#add ip hostname
192.168.xxx.xxx ds1
192.168.xxx.xxx ds2
192.168.xxx.xxx ds3
192.168.xxx.xxx ds4

Note: Please delete or comment out the line 127.0.0.1

On ds1, switch to the deployment user and configure ssh local password-free login

su dolphinscheduler;

ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys

Note: After the normal setting, the dolphinscheduler user ssh localhostdoes not need to enter the password when executing the command

On ds1, configure the deployment user dolphinscheduler ssh to connect to other machines to be deployed

On ds1, modify the directory permissions so that the deployment user has operating permissions on the dolphinscheduler-bin directory

sudo chown -R dolphinscheduler:dolphinscheduler dolphinscheduler-bin

6. Database initialization

  • Enter the database, the default database is PostgreSQL, if you choose MySQL, you need to add the mysql-connector-java driver package to the lib directory of DolphinScheduler, here is MySQL as an example
  mysql -h192.168.xx.xx -P3306 -uroot -p
  • After entering the database command line window, execute the database initialization command and set the access account and password. Note: {user} and {password} need to be replaced with specific database username and password
  CREATE DATABASE dolphinscheduler DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
  GRANT ALL PRIVILEGES ON dolphinscheduler.* TO '{root}'@'%' IDENTIFIED BY '{CDPmonitor123456*}';
  GRANT ALL PRIVILEGES ON dolphinscheduler.* TO '{root}'@'localhost' IDENTIFIED BY '{CDPmonitor123456*}';
  flush privileges;
  • Create table and import basic data
    Modify the following configuration in datasource.properties in the conf directory
  vi conf/datasource.properties

If you choose MySQL, please comment out the PostgreSQL related configuration (the same is true for the reverse), and you also need to manually add the [ mysql-connector-java driver jar ] package to the lib directory. The download here is mysql-connector-java-5.1.47.jar , And then correctly configure the database connection related information

  	#postgre
    #spring.datasource.driver-class-name=org.postgresql.Driver
    #spring.datasource.url=jdbc:postgresql://localhost:5432/dolphinscheduler
    # mysql
    spring.datasource.driver-class-name=com.mysql.jdbc.Driver
    spring.datasource.url=jdbc:mysql://xxx:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8&allowMultiQueries=true     需要修改ip
    spring.datasource.username=xxx						需要修改为上面的{
    
    user}值
    spring.datasource.password=xxx						需要修改为上面的{
    
    password}值
   
   # 修改之后
  
  spring.datasource.driver-class-name=com.mysql.jdbc.Driver
  spring.datasource.url=jdbc:mysql://192.168.5.72:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8&allowMultiQueries=true
  spring.datasource.username=root
  spring.datasource.password=CDPmonitor123456*

After modifying and saving, execute the create table and import basic data script in the script directory

  sh script/create-dolphinscheduler.sh

Note: If the execution of the above script reports "/bin/java: No such file or directory" error, please configure JAVA_HOME and PATH variables under /etc/profile

7. Modify operating parameters

Under the modified conf / env directory dolphinscheduler_env.shenvironment variable (related to use of the software is installed in / opt / soft, for example)

export HADOOP_HOME=/opt/cloudera/parcels/CDH/lib/hadoop
export HADOOP_CONF_DIR=/etc/hadoop/conf
#export SPARK_HOME1=/opt/soft/spark1
export SPARK_HOME2=/opt/cloudera/parcels/CDH/lib/spark
export PYTHON_HOME=/bin/python
export JAVA_HOME=/usr/java/jdk1.8.0_45/
export HIVE_HOME=/opt/cloudera/parcels/CDH/lib/hive
export FLINK_HOME=/opt/soft/flink
export  PATH=$HADOOP_HOME/bin:$SPARK_HOME2/bin:$PYTHON_HOME:$JAVA_HOME/bin:$HIVE_HOME/bin:$PATH:$FLINK_HOME/bin:$PATH

注: 这一步非常重要,例如 JAVA_HOME 和 PATH 是必须要配置的，没有用到的可以忽略或者注释掉

Soft link jdk to /usr/bin/java (still take JAVA_HOME=/opt/soft/java as an example)

sudo ln -s /usr/java/jdk1.8.0_45/bin/java /usr/bin/java

Modify a key deployment configuration file conf/config/install_config.conffor each parameter, pay special attention to the following configuration parameters

# 这里填 mysql or postgresql
dbtype="mysql"

# 数据库连接地址
dbhost="192.168.xx.xx:3306"

# 数据库名
dbname="dolphinscheduler"

# 数据库用户名，此处需要修改为上面设置的{user}具体值
username="xxx"

# 数据库密码, 如果有特殊字符，请使用\转义，需要修改为上面设置的{password}具体值
password="xxx"

#Zookeeper地址
zkQuorum="192.168.5.61:2181,192.168.5.65:2181,192.168.5.67:2181"

#将DS安装到哪个目录，如: /opt/dolphinscheduler/app，不同于现在的目录
installPath="/opt/dolphinscheduler/dolphinscheduler"

#使用哪个用户部署，使用第3节创建的用户
deployUser="dolphinscheduler"

# 邮件配置，以qq邮箱为例
# 邮件协议
mailProtocol="SMTP"

# 邮件服务地址
mailServerHost="smtp.qq.com"

# 邮件服务端口
mailServerPort="25"

# mailSender和mailUser配置成一样即可
# 发送者
mailSender="[email protected]"

# 发送用户
mailUser="[email protected]"

# 邮箱密码
mailPassword="xxx"

# TLS协议的邮箱设置为true，否则设置为false
starttlsEnable="true"

# 开启SSL协议的邮箱配置为true，否则为false。注意: starttlsEnable和sslEnable不能同时为true
sslEnable="false"

# 邮件服务地址值，参考上面 mailServerHost
sslTrust="smtp.qq.com"

# 业务用到的比如sql等资源文件上传到哪里，可以设置：HDFS,S3,NONE，单机如果想使用本地文件系统，请配置为HDFS，因为HDFS支持本地文件系统；如果不需要资源上传功能请选择NONE。强调一点：使用本地文件系统不需要部署hadoop
resourceStorageType="HDFS"

#如果上传资源保存想保存在hadoop上，hadoop集群的NameNode启用了HA的话，需要将hadoop的配置文件core-site.xml和hdfs-site.xml放到安装路径的conf目录下，本例即是放到/opt/soft/dolphinscheduler/conf下面，并配置namenode cluster名称；如果NameNode不是HA,则只需要将mycluster修改为具体的ip或者主机名即可
defaultFS="hdfs://192.168.5.19:8020"


# 如果没有使用到Yarn,保持以下默认值即可;如果ResourceManager是HA，则配置为ResourceManager节点的主备ip或者hostname,比如"192.168.xx.xx,192.168.xx.xx";如果是单ResourceManager请配置yarnHaIps=""即可
yarnHaIps=""

# 如果ResourceManager是HA或者没有使用到Yarn保持默认值即可；如果是单ResourceManager，请配置真实的ResourceManager主机名或者ip
singleYarnIp="yarnIp1"

# 资源上传根路径,主持HDFS和S3,由于hdfs支持本地文件系统，需要确保本地文件夹存在且有读写权限
resourceUploadPath="/data/dolphinscheduler"

# 具备权限创建resourceUploadPath的用户
hdfsRootUser="hdfs"



#在哪些机器上部署DS服务，本机选localhost
ips="192.168.5.72,192.168.5.19,192.168.5.61"

#ssh端口,默认22
sshPort="22"

#master服务部署在哪台机器上
masters="192.168.5.72,192.168.5.19"

#worker服务部署在哪台机器上,并指定此worker属于哪一个worker组,下面示例的default即为组名
workers="192.168.5.72:default,192.168.5.19:default,192.168.5.61:default"

#报警服务部署在哪台机器上
alertServer="192.168.5.72"

#后端api服务部署在在哪台机器上
apiServers="192.168.5.72"

pay attention:

  • If you need to upload resources to the Hadoop cluster function, and the NameNode of the Hadoop cluster is configured with HA, you need to enable HDFS type resource upload, and you need to copy the core-site.xml and hdfs-site.xml under the Hadoop cluster to /opt /dolphinscheduler/conf, skip the second step for non-NameNode HA

8. One-click deployment

Switch to the deployment user dolphinscheduler, and then execute the one-click deployment script

sh install.sh

Note: For
the first deployment, the 3,stop serverfollowing information appears 5 times in step 3 during operation , this information can be ignored

sh: bin/dolphinscheduler-daemon.sh: No such file or directory

After the script is completed, the following 5 services will be started, use the jpscommand to check whether the service is started ( jpsit java JDKcomes with it)

MasterServer         ----- master服务
WorkerServer         ----- worker服务
LoggerServer         ----- logger服务
ApiApplicationServer ----- api服务
AlertServer          ----- alert服务

If the above services are started normally, the automatic deployment is successful

After the deployment is successful, you can view the logs, and the logs are stored in the logs folder.

 logs/
    ├── dolphinscheduler-alert-server.log
    ├── dolphinscheduler-master-server.log
    |—— dolphinscheduler-worker-server.log
    |—— dolphinscheduler-api-server.log
    |—— dolphinscheduler-logger-server.log

9. Log in to the system

Visit the front-end page address, interface ip (modify by yourself) http://192.168.5.xx:12345/dolphinscheduler

Username/Password: admin/dolphinscheduler123

10. Start and stop service

  • Stop all services in the cluster with one click

    sh ./bin/stop-all.sh

  • Open all services of the cluster with one click

    sh ./bin/start-all.sh

  • Start and stop Master

sh ./bin/dolphinscheduler-daemon.sh start master-server
sh ./bin/dolphinscheduler-daemon.sh stop master-server
  • Start and stop Worker
sh ./bin/dolphinscheduler-daemon.sh start worker-server
sh ./bin/dolphinscheduler-daemon.sh stop worker-server
  • Start and stop Api
sh ./bin/dolphinscheduler-daemon.sh start api-server
sh ./bin/dolphinscheduler-daemon.sh stop api-server
  • Start and stop Logger
sh ./bin/dolphinscheduler-daemon.sh start logger-server
sh ./bin/dolphinscheduler-daemon.sh stop logger-server
  • Start and stop Alert
sh ./bin/dolphinscheduler-daemon.sh start alert-server
sh ./bin/dolphinscheduler-daemon.sh stop alert-server
注：服务用途请具体参见《系统架构设计》小节

11. Uninstall and reinstall

  • Stop all services of dolphin scheduler
sh bin/stop-all.sh
  • Clear the database of dolphinscheduler created in Mysql or GP
  • Delete the directory where dolphinscheduler was installed before
  • Modify the relevant configuration files and start the installation from [6. Database initialization]

FAQ

  1. When submitting the spark program, the dag tenant problem, the queue problem The
    tenant is equivalent to the user when submitting the spark task, and has nothing to do with the user of the dolphinscheduler, but the tenant corresponding to the dolphinscheduler user needs to be configured

  2. To enable the resource center, you need to configure HDFS, and you need to pay attention to the HADOOP_HOME configuration address of the CDH version

optimization

Queue: The
DolphinSchedule queue is currently bound to users. Different queues are set for different users. The user binding queue on the tenant does not take effect.

Sub-nodes:
Only the workflow of the current project can be selected, and other workflows can be introduced as sub-nodes in a workflow. The sub-nodes can also run during runtime, and can be centrally scheduled or single-point scheduling.
Only the workflow of the current project can be introduced. It is not possible to introduce the workflow of other projects (you can submit to the community). A compromise approach is to export the workflow of other projects and then import the current workflow. However, if there are problems, the other workflows need to be re-exported and imported after changes.

Dependent nodes:
You can create dependent nodes, and you can have multiple dependent nodes. Only after the execution of the dependent node is completed, can other nodes be executed. After the other nodes can run successfully, the following can be run

Failure retry mechanism: the
running state of the shell task. The
end identification state: exit 0 is all successful. In the case of failure, you can specify exit 1
to set the number of failures and the retry interval of failures. The granularity is minutes.

Complementary number mechanism:
with dependency: string
No dependency: pit: 27 workflow instances, the machine load is extremely high, tamping,

Complement: One day behind, for example, the number 2-4 needs to choose the number 1-3

Optimization: The data selection date does not really follow the date, which is very awkward

Guess you like

Origin blog.csdn.net/qq_43081842/article/details/114208264
Recommended
Ranking
Linux关机和重启详解（shutdown、halt、poweroff、reboot、init）
Netty work notes 0007---NIO's three core component relationships
Knife4j tutorial
2021.10.29，内容:什么时候用接口和抽象类
How to solve the problem that changing the memory frequency causes the computer to become unusable?
SpringMVC Tutorial - Controller
linux learning skills -Linux 25 transport Vega paid special privileges and facl extension
Financial quarterly report evaluation report data automatic generation 1.0
Agile Development Series - The Values of Agile Development
scrapy achieve browsercookie Middleware
Daily
More
2024-05-19(0)
2024-05-18(31)
2024-05-17(6)
2024-05-16(23)
2024-05-15(5)
2024-05-14(9)
2024-05-13(8)
2024-05-12(28)
2024-05-11(32)
2024-05-10(34)

Code World

© 2018 codetd.com