DolphinScheduler-2.0.0 安装部署

2023-12-27 13:59:06

一、环境准备

  • Centos 6.5 三台 (192.168.206.5? ?192.168.206.6? ?192.168.206.7)
  • Zookeeper:?3.4.6?左右
  • JDK:?JDK1.8+ 左右
  • Mysql:5.1.47+

二、准备工作

1、配置host (三台)

[root@master03 ~]# cat >>/etc/hosts<<EOF
192.168.206.5 node1
192.168.206.6 node2
192.168.206.7 node3
EOF

2、安装openjdk

安装此处略过

[root@master01 ~]# which java
/bin/java
[root@master01 ~]#
[root@master01 ~]# ll /bin/java
lrwxrwxrwx 1 root root 23 7月  18 09:40 /bin/java -> /data/java/jre/bin/java
[root@master01 ~]#
[root@master01 ~]# java -version
java version "1.8.0_181"
Java(TM) SE Runtime Environment (build 1.8.0_181-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)

3、Zookeeper集群

(node1操作)

下载 apache-zookeeper-3.6.4-bin

Index of /dist/zookeeper

cd /opt/src
sz apache-zookeeper-3.6.4-bin.tar.gz
tar -zxvf apache-zookeeper-3.6.4-bin.tar.gz
mv apache-zookeeper-3.6.4-bin /opt/
ln -s apache-zookeeper-3.6.4-bin/ zookeeper
cd /opt/zookeeper/conf
cp -r zoo_sample.cfg  zoo.cfg

vi zoo.cfg

dataDir=/opt/zookeeper/var/data
dataLogDir=/opt/zookeeper/var/datalog
clientPort=2181
server.1=node1:2888:3888
server.2=node2:2888:3888
server.3=node3:2888:3888

grep -v "^#\|^$" zoo.cfg

tickTime=2000
initLimit=10
syncLimit=5
dataDir=/opt/zookeeper/var/data
dataLogDir=/opt/zookeeper/var/datalog
clientPort=2181
server.1=node1:2888:3888
server.2=node2:2888:3888
server.3=node3:2888:3888
scp -r apache-zookeeper-3.6.4-bin/ node2:/opt/
scp -r apache-zookeeper-3.6.4-bin/ node3:/opt/

(node2 node3 操作)

cd /opt/src
ln -s apache-zookeeper-3.6.4-bin/ zookeeper

(node1 node2 node3 操作)?

启动

cd /opt/zookeeper/bin
./zkServer.sh start

查看状态

cd /opt/zookeeper/bin
./zkServer.sh status

4、配置免密

DolphinScheduler 集群默认的启动脚本是 start-all.sh,看名字就是启动一个节点,所有的节点都会启动,他是通过 ssh 启动其他节点,故需要配置免密

(node1 node2 node3 )

[root@ jerry7-11 ~]# ssh-keygen -t rsa   //默认指定算法的是rsa,可以没有-t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):   
//保存私钥的路径默认/root/.ssh/id_rsa,可以指定生成地址
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):   //密码
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.  生成私钥
Your public key has been saved in /root/.ssh/id_rsa.pub.  生成公钥
The key fingerprint is:
SHA256:PXTg9KW4P582PWJMEwUstPfyXk6NWdrB518yCORbisc root@hdss7-11
The key's randomart image is:
+---[RSA 2048]----+
|          +...o  |
|         o =.+ . |
|          =.=..  |
|         = o..o  |
|        S * ...+o|
|         o B +oB+|
|        . E * BoB|
|         .   *+B=|
|            ..+o+|
+----[SHA256]-----+

[root@master01 ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub -p22 root@192.168.206.5
[root@master01 ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub -p22 root@192.168.206.6
[root@master01 ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub -p22 root@192.168.206.7
[root@master01 ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub -p22 root@node1
[root@master01 ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub -p22 root@node2
[root@master01 ~]# ssh-copy-id -i /root/.ssh/id_rsa.pub -p22 root@node3

5、部署mysql

此处略,借鉴?Linux(CentOS 7)下安装 MySQL 8.0.31_centos 使用yum安装mysql8.0.31-CSDN博客

6、下载DolphinScheduler

(node1 操作)

此处略,去官网下载 apache-dolphinscheduler-2.0.0-alpha-bin.tar.gz

cd /opt/src
rz apache-dolphinscheduler-2.0.0-alpha-bin.tar.gz
tar -zxf apache-dolphinscheduler-2.0.0-alpha-bin.tar.gz

7、下载mysql-connector-java.jar驱动

去官网?MySQL :: MySQL Downloads,点击下方的Mysql?Community?Downloads。

然后单击选择 Connector/J选项:

点击?Archives

选择数据库版本,随后点击 Platform Independent?选项

?

下载等待压缩包下载完毕即可:

cd /opt/src
rz mysql-connector-j-8.0.31.zip
unzip mysql-connector-j-8.0.31.zip
cd /opt/src/mysql-connector-j-8.0.31
mv mysql-connector-j-8.0.31.jar mysql-connector-java-8.0.31.jar

三、部署DolphinScheduler

3.1、DolphinSchedule源数据库配置

(node1 操作)

1、创建数据库

在安装mysql的节点创建Dolphin的元数据库

mysql> CREATE DATABASE dolphinscheduler DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; -- 创建数据库并指定编码格式

开启权限

create USER 'dolphinscheduler'@'%'  IDENTIFIED WITH mysql_native_password BY '123456';

grant all on dolphinscheduler.* to 'dolphinscheduler'@'%';

flush privileges;

注意事项:要用root账户操作

低版本

-- 赋权
mysql> GRANT ALL PRIVILEGES ON dolphinscheduler.* TO '用户名'@'%' IDENTIFIED BY '密码';
-- 如果提示'Your password does not satisfy the current policy requirements'就说明这个密码不符当前的密码策略,可以执行如下命令
mysql> set global validate_password_policy=LOW; -- 验证密码等级强度
mysql> set global validate_password_length=6; -- 设置密码长度
mysql> flush privileges; -- 刷新权限

3.2、DolphinScheduler配置及安装

3.2.1 驱动配置

cp /opt/src/mysql-connector-j-8.0.31/mysql-connector-java-8.0.31.jar /opt/src/apache-dolphinscheduler-2.0.0-alpha-bin/lib/
3.2.2 配置文件修改
3.2.2.1 修改数据源配置
  • 修改conf目录下的datasource.properties配置文件]

    vi datasource.properties
    
    #修改内容如下
    
    # datasource configuration
    #spring.datasource.driver-class-name=org.postgresql.Driver
    #spring.datasource.url=jdbc:postgresql://127.0.0.1:5432/dolphinscheduler
    #spring.datasource.username=root
    #spring.datasource.password=root
    
    # mysql 因为这里使用的mysql作为元数据库,所以要将上面postgresql的配置注释掉
    spring.datasource.driver-class-name=com.mysql.jdbc.Driver
    spring.datasource.url=jdbc:mysql://192.168.206.5:3306/dolphinscheduler?characterEncoding=UTF-8&allowMultiQueries=true
    spring.datasource.username=dolphinscheduler
    spring.datasource.password=123456
    
    #保存退出即可
    
  • 执行脚本初始化元数据

    dolphin已经为我们准备了初始化元数据库的脚本,执行script目录下的create-dolphinscheduler.sh脚本即可

    ./create-dolphinscheduler.sh
    # 如果在部署完所有的步骤后,访问DolphinScheduler界面创建项目时,发现提示元数据库缺少字段,可以不通过create-dolphinscheduler.sh这个脚本初始化元数据库,可以将/opt/apps/apache-dolphinscheduler-2.0.0-alpha-bin/sql这个目录下的dolphinscheduler_mysql.sql这个文件在mysql中直接执行用来创建元数据表,在mysql中执行命令:source //opt/apps/apache-dolphinscheduler-2.0.0-alpha-bin/dolphinscheduler_mysql.sql
    

    执行完成后移可以去我们已经在mysql建立的dolphinscheduler库中查看一下

    mysql> show tables;
    +----------------------------------------+
    | Tables_in_dolphinscheduler             |
    +----------------------------------------+
    | QRTZ_BLOB_TRIGGERS                     |
    | QRTZ_CALENDARS                         |
    | QRTZ_CRON_TRIGGERS                     |
    | QRTZ_FIRED_TRIGGERS                    |
    | QRTZ_JOB_DETAILS                       |
    | QRTZ_LOCKS                             |
    | QRTZ_PAUSED_TRIGGER_GRPS               |
    | QRTZ_SCHEDULER_STATE                   |
    | QRTZ_SIMPLE_TRIGGERS                   |
    | QRTZ_SIMPROP_TRIGGERS                  |
    | QRTZ_TRIGGERS                          |
    | t_ds_access_token                      |
    | t_ds_alert                             |
    | t_ds_alert_plugin_instance             |
    | t_ds_alertgroup                        |
    | t_ds_command                           |
    | t_ds_datasource                        |
    | t_ds_environment                       |
    | t_ds_environment_worker_group_relation |
    | t_ds_error_command                     |
    | t_ds_master_server                     |
    | t_ds_plugin_define                     |
    | t_ds_process_definition                |
    -- 这里就不全部展示了
    
3.2.2.2 修改各组件环境变量

这里用到什么组件就修改什么就可了,没有使用到的组件可直接注释掉,修改/conf/env目录下的dolphinscheduler_env.sh配置文件

export HADOOP_HOME=/opt/apps/hadoop
export HADOOP_CONF_DIR=/opt/apps/hadoop/etc/hadoop
export SPARK_HOME1=/opt/apps/spark1
export SPARK_HOME2=/opt/apps/spark2
export PYTHON_HOME=/opt/apps/python
export JAVA_HOME=/data/java
export HIVE_HOME=/opt/apps/hive
export FLINK_HOME=/opt/apps/flink
export DATAX_HOME=/opt/apps/datax

export PATH=$HADOOP_HOME/bin:$SPARK_HOME1/bin:$SPARK_HOME2/bin:$PYTHON_HOME/bin:$JAVA_HOME/bin:$HIVE_HOME/bin:$FLINK_HOME/bin:$DATAX_HOME/bin:$PATH

3.2.2.3修改Dolphin安装配置
  • 修改/conf/config目录下的install_config.conf配置文件,案例如下

    vi ./install_config.conf
    
    # 文件内容如下
    #
    # Licensed to the Apache Software Foundation (ASF) under one or more
    # contributor license agreements.  See the NOTICE file distributed with
    # this work for additional information regarding copyright ownership.
    # The ASF licenses this file to You under the Apache License, Version 2.0
    # (the "License"); you may not use this file except in compliance with
    # the License.  You may obtain a copy of the License at
    #
    #     http://www.apache.org/licenses/LICENSE-2.0
    #
    # Unless required by applicable law or agreed to in writing, software
    # distributed under the License is distributed on an "AS IS" BASIS,
    # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    # See the License for the specific language governing permissions and
    # limitations under the License.
    #
    
    # ---------------------------------------------------------
    # INSTALL MACHINE
    # ---------------------------------------------------------
    # Dolphin服务所要部署的节点,ips后面可加主机名或者直接加ip地址
    ips="lx01,lx02,lx03"
    
    # ssh端口号,一般默认为22,不用修改
    sshPort="22"
    
    # master服务所在节点
    masters="lx01,lx02"
    
    # worker服务所在节点,并指定所属worker组,这里指定default组
    workers="lx01:default,lx02:default,lx03:default"
    
    # alter服务(报警服务)所在节点,单节点就可以
    alertServer="lx03"
    
    # api服务(后端api服务)所在节点,单节点就可以
    apiServers="lx01"
    
    # DS安装位置,三个节点都要有这个目录
    installPath="/opt/apps/dolphinscheduler"
    
    # 使用哪个用户进行安装部署,这里就直接使用root,如果使用别的用户进行安装部署需要对该用户进行授权
    deployUser="root"
    
    # DolphinScheduler本地数据的存储目录,三个节点都要有这个目录
    dataBasedirPath="/opt/data/dolphin"
    
    # JAVA环境变量的物理路径
    javaHome="/opt/apps/jdk1.8"
    
    # API后端服务的端口号同时也是web页面访问的端口号,根据实际需求进行配置
    apiServerPort="12345"
    
    # 所使使用的数据库类型,选择mysql或者postgresql,这里使用mysql
    dbtype="mysql"
    
    # mysql服务所在节点及端口号
    dbhost="lx01:3306"
    
    # mysql的用户名
    username="用户名"
    
    # mysql的密码
    password="密码"
    
    # 在mysql中所使用的库名,就是之前已经配置好的dolphin的元数据库
    dbname="dolphinscheduler"
    
    
    # 注册服务插件jar包位置,这个默认即可
    registryPluginDir="lib/plugin/registry/zookeeper"
    
    # 注册服务名称,默认即可
    registryPluginName="zookeeper"
    
    # 注册服务位置,将ip地址换成zoopeeker集群的位置即可,可以写ip地址或者主机名
    registryServers="lx01:2181,lx02:2181,lx03:2181"
    
    # Dolphin在zookeeper中注册的位置,默认即可
    zkRoot="/dolphinscheduler"
    
    
    # 警报服务插件jar包位置,默认即可
    alertPluginDir="lib/plugin/alert"
    
    
    # 任务调度插件jar包位置,默认即可
    taskPluginDir="lib/plugin/task"
    
    # 资源存储类型: HDFS, S3, NONE,这里选择hdfs,如果选择NONE则是使用本地文件系统
    resourceStorageType="HDFS"
    
    # 资源存储位置,因为上面选择的存储类型是HDFS,这个路径就是HDFS上的路径,要确保存在这个路径,并且Dolphin具备读写权限
    resourceUploadPath="/dolphinscheduler"
    
    # 默认文件系统,如果选择了HDFS并且HDFS已经开启了HA那么就要将core-site.xml和hdfs-site.xml放入到conf目录下
    defaultFS="hdfs://lx01:8020"
    
    # 没有使用到s3,这个就可以注释掉,如果使用的s3再另行配置
    # s3Endpoint="http://192.168.xx.xx:9010"
    # s3AccessKey="xxxxxxxxxx"
    # s3SecretKey="xxxxxxxxxx"
    
    # yarn的resourcemanager http端口号
    resourceManagerHttpAddressPort="8088"
    
    # 如resourcemanager HA启用,则配置为ResourceManager节点的主备ip或者hostname,比如"192.168.xx.xx,192.168.xx.xx";如果是单ResourceManager请配置yarnHaIps=""即可
    yarnHaIps="192.168.xx.xx,192.168.xx.xx"
    
    # 如果ResourceManager是HA或者没有使用到Yarn保持默认值"yarnIp1"即可;如果是单ResourceManager,请配置真实的ResourceManager主机名或者ip
    singleYarnIp="lx01"
    
    # 使用哪个用户对hdfs文件系统进行操作,一般我们要选一个权限比较高的
    hdfsRootUser="hdfs"
    
    # 关于kerberos根据实际情况配置即可,如果开启了kerberos就根据实际情况配置,没有开启则不用动
    # kerberos config
    # whether kerberos starts, if kerberos starts, following four items need to config, otherwise please ignore
    kerberosStartUp="false"
    # kdc krb5 config file path
    krb5ConfPath="$installPath/conf/krb5.conf"
    # keytab username,watch out the @ sign should followd by \\
    keytabUserName="hdfs-mycluster\\@ESZ.COM"
    # username keytab path
    keytabPath="$installPath/conf/hdfs.headless.keytab"
    # kerberos expire time, the unit is hour
    kerberosExpireTime="2"
    
    # 是否使用sudo,默认为true就可
    sudoEnable="true"
    
    # 工作租户自动创建,默认为false即可
    workerTenantAutoCreate="false"
    
    

    博主配置如下

    [root@master01 config]# grep -v "^#\|^$" install_config.conf
    ips="node1,node2,node3"
    sshPort="22"
    masters="node1,node2"
    workers="node1:default,node2:default,node3:default"
    alertServer="node3"
    apiServers="node1"
    installPath="/opt/apps/dolphinscheduler"
    deployUser="dolphinscheduler"
    dataBasedirPath="/data/dolphin"
    javaHome="/data/java"
    apiServerPort="12345"
    dbtype="mysql"
    dbhost="192.168.206.5:3306"
    username="dolphinscheduler"
    password="123456"
    dbname="dolphinscheduler"
    registryPluginDir="lib/plugin/registry/zookeeper"
    registryPluginName="zookeeper"
    registryServers="192.168.206.5:2181,192.168.206.6:2181,192.168.206.7:2181"
    zkRoot="/dolphinscheduler"
    alertPluginDir="lib/plugin/alert"
    taskPluginDir="lib/plugin/task"
    resourceStorageType="NONE"
    resourceUploadPath="/dolphinscheduler"
    defaultFS="hdfs://node1:8020"
    resourceManagerHttpAddressPort="8088"
    yarnHaIps=""
    singleYarnIp="node1"
    hdfsRootUser="hdfs"
    kerberosStartUp="false"
    krb5ConfPath="$installPath/conf/krb5.conf"
    keytabUserName="hdfs-mycluster\\@ESZ.COM"
    keytabPath="$installPath/conf/hdfs.headless.keytab"
    kerberosExpireTime="2"
    sudoEnable="true"
    workerTenantAutoCreate="false"

3.2.2.4 安装

安装中问题,提示没有dolphinscheduler用户,三台都操作

groupadd dolphinscheduler
useradd -g dolphinscheduler-m -d /home/dolphinscheduler -s /bin/bash dolphinscheduler

执行apache-dolphinscheduler-2.0.0-alpha-bin目录下的install.sh脚本进行安装部署

./install.sh

安装完成后DolphinScheduler会自动启动,可通过jps命令查看服务是否已经启动

jps
3475 QuorumPeerMain
5571 MasterServer # Master服务
5877 Jps
3703 NameNode
3848 DataNode
4457 NodeManager
5657 LoggerServer # 日志服务
4858 JobHistoryServer
5707 ApiApplicationServer # Api服务
4302 ResourceManager
5615 WorkerServer # Worker服务

查看启动状态

cd /opt/apps/dolphinscheduler/bin
./status-all.sh


node1  Begin status master-server......
master-server  [  RUNNING]
End status master-server.
node2  Begin status master-server......
master-server  [  STOP  ]
End status master-server.
node1  Begin status worker-server......
worker-server  [  RUNNING]
End status worker-server.
node1  Begin status logger-server......
logger-server  [  RUNNING  ]
End status logger-server.
node2  Begin status worker-server......
worker-server  [  RUNNING]
End status worker-server.
node2  Begin status logger-server......
logger-server  [  RUNNING  ]
End status logger-server.
node3  Begin status worker-server......
worker-server  [  RUNNING  ]
End status worker-server.
node3  Begin status logger-server......
logger-server  [  RUNNING  ]
End status logger-server.
node3  Begin status alert-server......
alert-server  [  RUNNING  ]
End status alert-server.
node1  Begin status api-server......
api-server  [  RUNNING  ]
End status api-server.

如何停止和启动

[root@master01 bin]# pwd
/opt/apps/dolphinscheduler/bin

-rwxr-xr-x 1 root root  2110 12月 26 16:30 start-all.sh
-rwxr-xr-x 1 root root  2979 12月 26 16:30 status-all.sh
-rwxr-xr-x 1 root root  2105 12月 26 16:30 stop-all.sh

启动异常查看日志

/opt/apps/dolphinscheduler/logs
3.2.2.5通过web界面访问
URL
  • URL

    ip地址:12345/dolphinscheduler

  • 默认用户

    admin

  • 默认密码

    dolphinscheduler123

  • 登录界面如下

文章来源:https://blog.csdn.net/Jerry00713/article/details/135239260
本文来自互联网用户投稿,该文观点仅代表作者本人,不代表本站立场。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。