CDH 6.3.2集成flink 1.18 zookeeper版本不匹配Flink-yarn启动失败
2023-12-31 12:43:58
CDH 6.3.2集成flink 1.18 zookeeper版本不匹配Flink-yarn不能正常启动，而在CHD Web页面，flink日志报错提示不明确，不能定位具体错误。CM WEB启动失败错误日志如下图所示：
CM查看完成错误日志
[31/Dec/2023 10:45:09 +0000] 260001 MainThread redactor INFO Started launcher: /opt/cloudera/cm-agent/service/csd/csd.sh start
[31/Dec/2023 10:45:09 +0000] 260001 MainThread redactor ERROR Redaction rules file doesn't exist, not redacting logs. file: redaction-rules.json, directory: /run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN
[31/Dec/2023 10:45:09 +0000] 260001 MainThread redactor INFO Re-exec watcher: /opt/cloudera/cm-agent/bin/cm proc_watcher 260009
2023年 12月 31日 星期日 10:45:09 CST
+ locate_java_home
+ locate_java_home_no_verify
+ JAVA11_HOME_CANDIDATES=('/usr/java/jdk-11' '/usr/lib/jvm/jdk-11' '/usr/lib/jvm/java-11-oracle')
+ local JAVA11_HOME_CANDIDATES
+ OPENJAVA11_HOME_CANDIDATES=('/usr/lib/jvm/jdk-11' '/usr/lib64/jvm/jdk-11')
+ local OPENJAVA11_HOME_CANDIDATES
+ JAVA8_HOME_CANDIDATES=('/usr/java/jdk1.8' '/usr/java/jre1.8' '/usr/lib/jvm/j2sdk1.8-oracle' '/usr/lib/jvm/j2sdk1.8-oracle/jre' '/usr/lib/jvm/java-8-oracle')
+ local JAVA8_HOME_CANDIDATES
+ OPENJAVA8_HOME_CANDIDATES=('/usr/lib/jvm/java-1.8.0-openjdk' '/usr/lib/jvm/java-8-openjdk' '/usr/lib64/jvm/java-1.8.0-openjdk' '/usr/lib64/jvm/java-8-openjdk')
+ local OPENJAVA8_HOME_CANDIDATES
+ MISCJAVA_HOME_CANDIDATES=('/Library/Java/Home' '/usr/java/default' '/usr/lib/jvm/default-java' '/usr/lib/jvm/java-openjdk' '/usr/lib/jvm/jre-openjdk')
+ local MISCJAVA_HOME_CANDIDATES
+ case ${BIGTOP_JAVA_MAJOR} in
+ JAVA_HOME_CANDIDATES=(${JAVA8_HOME_CANDIDATES[@]} ${MISCJAVA_HOME_CANDIDATES[@]} ${OPENJAVA8_HOME_CANDIDATES[@]} ${JAVA11_HOME_CANDIDATES[@]} ${OPENJAVA11_HOME_CANDIDATES[@]})
+ '[' -z '' ']'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd /usr/java/jdk1.8.0_181-cloudera
+ for candidate in '`ls -rvd ${candidate_regex}* 2>/dev/null`'
+ '[' -e /usr/java/jdk1.8.0_181-cloudera/bin/java ']'
+ export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
+ JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
+ break 2
+ verify_java_home
+ '[' -z /usr/java/jdk1.8.0_181-cloudera ']'
+ echo JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
+ '[' -n '' ']'
+ source_parcel_environment
+ '[' '!' -z '' ']'
+ echo 'Using /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN as conf dir'
+ echo 'Using scripts/control.sh as process script'
+ replace_conf_dir
+ echo CONF_DIR=/var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN
+ echo CMF_CONF_DIR=
+ EXCLUDE_CMF_FILES=('cloudera-config.sh' 'hue.sh' 'impala.sh' 'sqoop.sh' 'supervisor.conf' 'config.zip' 'proc.json' '*.log' '*.keytab' '*jceks' 'supervisor_status')
++ printf '! -name %s ' cloudera-config.sh hue.sh impala.sh sqoop.sh supervisor.conf config.zip proc.json '*.log' flink_on_yarn.keytab '*jceks' supervisor_status
+ find /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN -type f '!' -path '/var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/logs/*' '!' -name cloudera-config.sh '!' -name hue.sh '!' -name impala.sh '!' -name sqoop.sh '!' -name supervisor.conf '!' -name config.zip '!' -name proc.json '!' -name '*.log' '!' -name flink_on_yarn.keytab '!' -name '*jceks' '!' -name supervisor_status -exec perl -pi -e 's#\{\{CMF_CONF_DIR}}#/var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN#g' '{}' ';'
+ make_scripts_executable
+ find /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN -regex '.*\.\(py\|sh\)$' -exec chmod u+x '{}' ';'
+ RUN_DIR=/var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN
+ '[' '' == true ']'
+ chmod u+x /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/scripts/control.sh
+ export COMMON_SCRIPT=/opt/cloudera/cm-agent/service/common/cloudera-config.sh
+ COMMON_SCRIPT=/opt/cloudera/cm-agent/service/common/cloudera-config.sh
+ exec /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/scripts/control.sh start
+ USAGE='Usage: control.sh (start|stop)'
+ OPERATION=start
+ case $OPERATION in
++ hostname -f
+ NODE_HOST=node01
+ '[' '!' -d /opt/cloudera/parcels/FLINK ']'
+ FLINK_HOME=/opt/cloudera/parcels/FLINK/lib/flink
+ FLINK_CONF_DIR=/var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf
+ '[' '!' -d /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf ']'
+ mkdir /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf
+ cp /opt/cloudera/parcels/FLINK/lib/flink/conf/flink-conf.yaml /opt/cloudera/parcels/FLINK/lib/flink/conf/log4j-cli.properties /opt/cloudera/parcels/FLINK/lib/flink/conf/log4j-console.properties /opt/cloudera/parcels/FLINK/lib/flink/conf/log4j.properties /opt/cloudera/parcels/FLINK/lib/flink/conf/log4j-session.properties /opt/cloudera/parcels/FLINK/lib/flink/conf/logback-console.xml /opt/cloudera/parcels/FLINK/lib/flink/conf/logback-session.xml /opt/cloudera/parcels/FLINK/lib/flink/conf/logback.xml /opt/cloudera/parcels/FLINK/lib/flink/conf/masters /opt/cloudera/parcels/FLINK/lib/flink/conf/workers /opt/cloudera/parcels/FLINK/lib/flink/conf/zoo.cfg /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/
+ sed -i 's#=#: #g' /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf.properties
++ cat /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf.properties
++ grep high-availability:
+ HIGH_MODE=
+ '[' '' = '' ']'
+ echo 'high-availability: zookeeper'
++ cat /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf.properties
++ grep high-availability:
+ HIGH_MODE='high-availability: zookeeper'
+ echo 'HIGH_MODE: high-availability: zookeeper'
++ cat /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf.properties
++ grep high-availability.zookeeper.quorum:
+ HIGH_ZK_QUORUM=
+ '[' '' = '' ']'
+ echo 'high-availability.zookeeper.quorum: node01:2181'
++ cat /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf.properties
++ grep high-availability.zookeeper.quorum:
+ HIGH_ZK_QUORUM='high-availability.zookeeper.quorum: node01:2181'
+ echo 'HIGH_ZK_QUORUM: high-availability.zookeeper.quorum: node01:2181'
+ cp /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf.properties /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/flink-conf.yaml
+ HADOOP_CONF_DIR=/var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/yarn-conf
+ export FLINK_HOME FLINK_CONF_DIR HADOOP_CONF_DIR
+ echo CONF_DIR: /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN
+ echo HADOOP_CONF_DIR: /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/yarn-conf
+ echo ''
++ date
+ echo 'Date: 2023年 12月 31日 星期日 10:45:09 CST'
+ echo 'Host: node01'
+ echo 'NODE_TYPE: '
+ echo 'ZK_QUORUM: node01:2181'
+ echo 'FLINK_HOME: /opt/cloudera/parcels/FLINK/lib/flink'
+ echo 'FLINK_CONF_DIR: /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf'
+ echo ''
+ '[' true = true ']'
+ exec /opt/cloudera/parcels/FLINK/lib/flink/bin/flink-yarn.sh --container 1 --streaming
[31/Dec/2023 10:45:27 +0000] 262506 MainThread redactor INFO Started launcher: /opt/cloudera/cm-agent/service/csd/csd.sh start
[31/Dec/2023 10:45:27 +0000] 262506 MainThread redactor ERROR Redaction rules file doesn't exist, not redacting logs. file: redaction-rules.json, directory: /run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN
[31/Dec/2023 10:45:27 +0000] 262506 MainThread redactor INFO Re-exec watcher: /opt/cloudera/cm-agent/bin/cm proc_watcher 262546
2023年 12月 31日 星期日 10:45:27 CST
+ locate_java_home
+ locate_java_home_no_verify
+ JAVA11_HOME_CANDIDATES=('/usr/java/jdk-11' '/usr/lib/jvm/jdk-11' '/usr/lib/jvm/java-11-oracle')
+ local JAVA11_HOME_CANDIDATES
+ OPENJAVA11_HOME_CANDIDATES=('/usr/lib/jvm/jdk-11' '/usr/lib64/jvm/jdk-11')
+ local OPENJAVA11_HOME_CANDIDATES
+ JAVA8_HOME_CANDIDATES=('/usr/java/jdk1.8' '/usr/java/jre1.8' '/usr/lib/jvm/j2sdk1.8-oracle' '/usr/lib/jvm/j2sdk1.8-oracle/jre' '/usr/lib/jvm/java-8-oracle')
+ local JAVA8_HOME_CANDIDATES
+ OPENJAVA8_HOME_CANDIDATES=('/usr/lib/jvm/java-1.8.0-openjdk' '/usr/lib/jvm/java-8-openjdk' '/usr/lib64/jvm/java-1.8.0-openjdk' '/usr/lib64/jvm/java-8-openjdk')
+ local OPENJAVA8_HOME_CANDIDATES
+ MISCJAVA_HOME_CANDIDATES=('/Library/Java/Home' '/usr/java/default' '/usr/lib/jvm/default-java' '/usr/lib/jvm/java-openjdk' '/usr/lib/jvm/jre-openjdk')
+ local MISCJAVA_HOME_CANDIDATES
+ case ${BIGTOP_JAVA_MAJOR} in
+ JAVA_HOME_CANDIDATES=(${JAVA8_HOME_CANDIDATES[@]} ${MISCJAVA_HOME_CANDIDATES[@]} ${OPENJAVA8_HOME_CANDIDATES[@]} ${JAVA11_HOME_CANDIDATES[@]} ${OPENJAVA11_HOME_CANDIDATES[@]})
+ '[' -z '' ']'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd /usr/java/jdk1.8.0_181-cloudera
+ for candidate in '`ls -rvd ${candidate_regex}* 2>/dev/null`'
+ '[' -e /usr/java/jdk1.8.0_181-cloudera/bin/java ']'
+ export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
+ JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
+ break 2
+ verify_java_home
+ '[' -z /usr/java/jdk1.8.0_181-cloudera ']'
+ echo JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
+ '[' -n '' ']'
+ source_parcel_environment
+ '[' '!' -z '' ']'
+ echo 'Using /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN as conf dir'
+ echo 'Using scripts/control.sh as process script'
+ replace_conf_dir
+ echo CONF_DIR=/var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN
+ echo CMF_CONF_DIR=
+ EXCLUDE_CMF_FILES=('cloudera-config.sh' 'hue.sh' 'impala.sh' 'sqoop.sh' 'supervisor.conf' 'config.zip' 'proc.json' '*.log' '*.keytab' '*jceks' 'supervisor_status')
++ printf '! -name %s ' cloudera-config.sh hue.sh impala.sh sqoop.sh supervisor.conf config.zip proc.json '*.log' flink_on_yarn.keytab '*jceks' supervisor_status
+ find /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN -type f '!' -path '/var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/logs/*' '!' -name cloudera-config.sh '!' -name hue.sh '!' -name impala.sh '!' -name sqoop.sh '!' -name supervisor.conf '!' -name config.zip '!' -name proc.json '!' -name '*.log' '!' -name flink_on_yarn.keytab '!' -name '*jceks' '!' -name supervisor_status -exec perl -pi -e 's#\{\{CMF_CONF_DIR}}#/var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN#g' '{}' ';'
+ make_scripts_executable
+ find /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN -regex '.*\.\(py\|sh\)$' -exec chmod u+x '{}' ';'
+ RUN_DIR=/var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN
+ '[' '' == true ']'
+ chmod u+x /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/scripts/control.sh
+ export COMMON_SCRIPT=/opt/cloudera/cm-agent/service/common/cloudera-config.sh
+ COMMON_SCRIPT=/opt/cloudera/cm-agent/service/common/cloudera-config.sh
+ exec /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/scripts/control.sh start
+ USAGE='Usage: control.sh (start|stop)'
+ OPERATION=start
+ case $OPERATION in
++ hostname -f
+ NODE_HOST=node01
+ '[' '!' -d /opt/cloudera/parcels/FLINK ']'
+ FLINK_HOME=/opt/cloudera/parcels/FLINK/lib/flink
+ FLINK_CONF_DIR=/var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf
+ '[' '!' -d /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf ']'
+ rm -rf /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/flink-conf.yaml /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/log4j-cli.properties /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/log4j-console.properties /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/log4j.properties /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/log4j-session.properties /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/logback-console.xml /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/logback-session.xml /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/logback.xml /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/masters /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/workers /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/zoo.cfg
+ cp /opt/cloudera/parcels/FLINK/lib/flink/conf/flink-conf.yaml /opt/cloudera/parcels/FLINK/lib/flink/conf/log4j-cli.properties /opt/cloudera/parcels/FLINK/lib/flink/conf/log4j-console.properties /opt/cloudera/parcels/FLINK/lib/flink/conf/log4j.properties /opt/cloudera/parcels/FLINK/lib/flink/conf/log4j-session.properties /opt/cloudera/parcels/FLINK/lib/flink/conf/logback-console.xml /opt/cloudera/parcels/FLINK/lib/flink/conf/logback-session.xml /opt/cloudera/parcels/FLINK/lib/flink/conf/logback.xml /opt/cloudera/parcels/FLINK/lib/flink/conf/masters /opt/cloudera/parcels/FLINK/lib/flink/conf/workers /opt/cloudera/parcels/FLINK/lib/flink/conf/zoo.cfg /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/
+ sed -i 's#=#: #g' /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf.properties
++ cat /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf.properties
++ grep high-availability:
+ HIGH_MODE='high-availability: zookeeper'
+ '[' 'high-availability: zookeeper' = '' ']'
++ cat /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf.properties
++ grep high-availability.zookeeper.quorum:
+ HIGH_ZK_QUORUM='high-availability.zookeeper.quorum: node01:2181'
+ '[' 'high-availability.zookeeper.quorum: node01:2181' = '' ']'
+ cp /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf.properties /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/flink-conf.yaml
+ HADOOP_CONF_DIR=/var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/yarn-conf
+ export FLINK_HOME FLINK_CONF_DIR HADOOP_CONF_DIR
+ echo CONF_DIR: /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN
+ echo HADOOP_CONF_DIR: /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/yarn-conf
+ echo ''
++ date
+ echo 'Date: 2023年 12月 31日 星期日 10:45:27 CST'
+ echo 'Host: node01'
+ echo 'NODE_TYPE: '
+ echo 'ZK_QUORUM: node01:2181'
+ echo 'FLINK_HOME: /opt/cloudera/parcels/FLINK/lib/flink'
+ echo 'FLINK_CONF_DIR: /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf'
+ echo ''
+ '[' true = true ']'
+ exec /opt/cloudera/parcels/FLINK/lib/flink/bin/flink-yarn.sh --container 1 --streaming
[31/Dec/2023 10:45:42 +0000] 263981 MainThread redactor INFO Started launcher: /opt/cloudera/cm-agent/service/csd/csd.sh start
[31/Dec/2023 10:45:42 +0000] 263981 MainThread redactor ERROR Redaction rules file doesn't exist, not redacting logs. file: redaction-rules.json, directory: /run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN
[31/Dec/2023 10:45:42 +0000] 263981 MainThread redactor INFO Re-exec watcher: /opt/cloudera/cm-agent/bin/cm proc_watcher 264012
2023年 12月 31日 星期日 10:45:42 CST
+ locate_java_home
+ locate_java_home_no_verify
+ JAVA11_HOME_CANDIDATES=('/usr/java/jdk-11' '/usr/lib/jvm/jdk-11' '/usr/lib/jvm/java-11-oracle')
+ local JAVA11_HOME_CANDIDATES
+ OPENJAVA11_HOME_CANDIDATES=('/usr/lib/jvm/jdk-11' '/usr/lib64/jvm/jdk-11')
+ local OPENJAVA11_HOME_CANDIDATES
+ JAVA8_HOME_CANDIDATES=('/usr/java/jdk1.8' '/usr/java/jre1.8' '/usr/lib/jvm/j2sdk1.8-oracle' '/usr/lib/jvm/j2sdk1.8-oracle/jre' '/usr/lib/jvm/java-8-oracle')
+ local JAVA8_HOME_CANDIDATES
+ OPENJAVA8_HOME_CANDIDATES=('/usr/lib/jvm/java-1.8.0-openjdk' '/usr/lib/jvm/java-8-openjdk' '/usr/lib64/jvm/java-1.8.0-openjdk' '/usr/lib64/jvm/java-8-openjdk')
+ local OPENJAVA8_HOME_CANDIDATES
+ MISCJAVA_HOME_CANDIDATES=('/Library/Java/Home' '/usr/java/default' '/usr/lib/jvm/default-java' '/usr/lib/jvm/java-openjdk' '/usr/lib/jvm/jre-openjdk')
+ local MISCJAVA_HOME_CANDIDATES
+ case ${BIGTOP_JAVA_MAJOR} in
+ JAVA_HOME_CANDIDATES=(${JAVA8_HOME_CANDIDATES[@]} ${MISCJAVA_HOME_CANDIDATES[@]} ${OPENJAVA8_HOME_CANDIDATES[@]} ${JAVA11_HOME_CANDIDATES[@]} ${OPENJAVA11_HOME_CANDIDATES[@]})
+ '[' -z '' ']'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd /usr/java/jdk1.8.0_181-cloudera
+ for candidate in '`ls -rvd ${candidate_regex}* 2>/dev/null`'
+ '[' -e /usr/java/jdk1.8.0_181-cloudera/bin/java ']'
+ export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
+ JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
+ break 2
+ verify_java_home
+ '[' -z /usr/java/jdk1.8.0_181-cloudera ']'
+ echo JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
+ '[' -n '' ']'
+ source_parcel_environment
+ '[' '!' -z '' ']'
+ echo 'Using /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN as conf dir'
+ echo 'Using scripts/control.sh as process script'
+ replace_conf_dir
+ echo CONF_DIR=/var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN
+ echo CMF_CONF_DIR=
+ EXCLUDE_CMF_FILES=('cloudera-config.sh' 'hue.sh' 'impala.sh' 'sqoop.sh' 'supervisor.conf' 'config.zip' 'proc.json' '*.log' '*.keytab' '*jceks' 'supervisor_status')
++ printf '! -name %s ' cloudera-config.sh hue.sh impala.sh sqoop.sh supervisor.conf config.zip proc.json '*.log' flink_on_yarn.keytab '*jceks' supervisor_status
+ find /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN -type f '!' -path '/var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/logs/*' '!' -name cloudera-config.sh '!' -name hue.sh '!' -name impala.sh '!' -name sqoop.sh '!' -name supervisor.conf '!' -name config.zip '!' -name proc.json '!' -name '*.log' '!' -name flink_on_yarn.keytab '!' -name '*jceks' '!' -name supervisor_status -exec perl -pi -e 's#\{\{CMF_CONF_DIR}}#/var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN#g' '{}' ';'
+ make_scripts_executable
+ find /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN -regex '.*\.\(py\|sh\)$' -exec chmod u+x '{}' ';'
+ RUN_DIR=/var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN
+ '[' '' == true ']'
+ chmod u+x /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/scripts/control.sh
+ export COMMON_SCRIPT=/opt/cloudera/cm-agent/service/common/cloudera-config.sh
+ COMMON_SCRIPT=/opt/cloudera/cm-agent/service/common/cloudera-config.sh
+ exec /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/scripts/control.sh start
+ USAGE='Usage: control.sh (start|stop)'
+ OPERATION=start
+ case $OPERATION in
++ hostname -f
+ NODE_HOST=node01
+ '[' '!' -d /opt/cloudera/parcels/FLINK ']'
+ FLINK_HOME=/opt/cloudera/parcels/FLINK/lib/flink
+ FLINK_CONF_DIR=/var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf
+ '[' '!' -d /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf ']'
+ rm -rf /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/flink-conf.yaml /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/log4j-cli.properties /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/log4j-console.properties /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/log4j.properties /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/log4j-session.properties /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/logback-console.xml /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/logback-session.xml /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/logback.xml /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/masters /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/workers /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/zoo.cfg
+ cp /opt/cloudera/parcels/FLINK/lib/flink/conf/flink-conf.yaml /opt/cloudera/parcels/FLINK/lib/flink/conf/log4j-cli.properties /opt/cloudera/parcels/FLINK/lib/flink/conf/log4j-console.properties /opt/cloudera/parcels/FLINK/lib/flink/conf/log4j.properties /opt/cloudera/parcels/FLINK/lib/flink/conf/log4j-session.properties /opt/cloudera/parcels/FLINK/lib/flink/conf/logback-console.xml /opt/cloudera/parcels/FLINK/lib/flink/conf/logback-session.xml /opt/cloudera/parcels/FLINK/lib/flink/conf/logback.xml /opt/cloudera/parcels/FLINK/lib/flink/conf/masters /opt/cloudera/parcels/FLINK/lib/flink/conf/workers /opt/cloudera/parcels/FLINK/lib/flink/conf/zoo.cfg /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/
+ sed -i 's#=#: #g' /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf.properties
++ cat /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf.properties
++ grep high-availability:
+ HIGH_MODE='high-availability: zookeeper'
+ '[' 'high-availability: zookeeper' = '' ']'
++ cat /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf.properties
++ grep high-availability.zookeeper.quorum:
+ HIGH_ZK_QUORUM='high-availability.zookeeper.quorum: node01:2181'
+ '[' 'high-availability.zookeeper.quorum: node01:2181' = '' ']'
+ cp /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf.properties /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/flink-conf.yaml
+ HADOOP_CONF_DIR=/var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/yarn-conf
+ export FLINK_HOME FLINK_CONF_DIR HADOOP_CONF_DIR
+ echo CONF_DIR: /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN
+ echo HADOOP_CONF_DIR: /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/yarn-conf
+ echo ''
++ date
+ echo 'Date: 2023年 12月 31日 星期日 10:45:42 CST'
+ echo 'Host: node01'
+ echo 'NODE_TYPE: '
+ echo 'ZK_QUORUM: node01:2181'
+ echo 'FLINK_HOME: /opt/cloudera/parcels/FLINK/lib/flink'
+ echo 'FLINK_CONF_DIR: /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf'
+ echo ''
+ '[' true = true ']'
+ exec /opt/cloudera/parcels/FLINK/lib/flink/bin/flink-yarn.sh --container 1 --streaming
[31/Dec/2023 10:45:59 +0000] 265973 MainThread redactor INFO Started launcher: /opt/cloudera/cm-agent/service/csd/csd.sh start
[31/Dec/2023 10:45:59 +0000] 265973 MainThread redactor ERROR Redaction rules file doesn't exist, not redacting logs. file: redaction-rules.json, directory: /run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN
[31/Dec/2023 10:45:59 +0000] 265973 MainThread redactor INFO Re-exec watcher: /opt/cloudera/cm-agent/bin/cm proc_watcher 265980
2023年 12月 31日 星期日 10:45:59 CST
+ locate_java_home
+ locate_java_home_no_verify
+ JAVA11_HOME_CANDIDATES=('/usr/java/jdk-11' '/usr/lib/jvm/jdk-11' '/usr/lib/jvm/java-11-oracle')
+ local JAVA11_HOME_CANDIDATES
+ OPENJAVA11_HOME_CANDIDATES=('/usr/lib/jvm/jdk-11' '/usr/lib64/jvm/jdk-11')
+ local OPENJAVA11_HOME_CANDIDATES
+ JAVA8_HOME_CANDIDATES=('/usr/java/jdk1.8' '/usr/java/jre1.8' '/usr/lib/jvm/j2sdk1.8-oracle' '/usr/lib/jvm/j2sdk1.8-oracle/jre' '/usr/lib/jvm/java-8-oracle')
+ local JAVA8_HOME_CANDIDATES
+ OPENJAVA8_HOME_CANDIDATES=('/usr/lib/jvm/java-1.8.0-openjdk' '/usr/lib/jvm/java-8-openjdk' '/usr/lib64/jvm/java-1.8.0-openjdk' '/usr/lib64/jvm/java-8-openjdk')
+ local OPENJAVA8_HOME_CANDIDATES
+ MISCJAVA_HOME_CANDIDATES=('/Library/Java/Home' '/usr/java/default' '/usr/lib/jvm/default-java' '/usr/lib/jvm/java-openjdk' '/usr/lib/jvm/jre-openjdk')
+ local MISCJAVA_HOME_CANDIDATES
+ case ${BIGTOP_JAVA_MAJOR} in
+ JAVA_HOME_CANDIDATES=(${JAVA8_HOME_CANDIDATES[@]} ${MISCJAVA_HOME_CANDIDATES[@]} ${OPENJAVA8_HOME_CANDIDATES[@]} ${JAVA11_HOME_CANDIDATES[@]} ${OPENJAVA11_HOME_CANDIDATES[@]})
+ '[' -z '' ']'
+ for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}'
++ ls -rvd /usr/java/jdk1.8.0_181-cloudera
+ for candidate in '`ls -rvd ${candidate_regex}* 2>/dev/null`'
+ '[' -e /usr/java/jdk1.8.0_181-cloudera/bin/java ']'
+ export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
+ JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
+ break 2
+ verify_java_home
+ '[' -z /usr/java/jdk1.8.0_181-cloudera ']'
+ echo JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
+ '[' -n '' ']'
+ source_parcel_environment
+ '[' '!' -z '' ']'
+ echo 'Using /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN as conf dir'
+ echo 'Using scripts/control.sh as process script'
+ replace_conf_dir
+ echo CONF_DIR=/var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN
+ echo CMF_CONF_DIR=
+ EXCLUDE_CMF_FILES=('cloudera-config.sh' 'hue.sh' 'impala.sh' 'sqoop.sh' 'supervisor.conf' 'config.zip' 'proc.json' '*.log' '*.keytab' '*jceks' 'supervisor_status')
++ printf '! -name %s ' cloudera-config.sh hue.sh impala.sh sqoop.sh supervisor.conf config.zip proc.json '*.log' flink_on_yarn.keytab '*jceks' supervisor_status
+ find /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN -type f '!' -path '/var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/logs/*' '!' -name cloudera-config.sh '!' -name hue.sh '!' -name impala.sh '!' -name sqoop.sh '!' -name supervisor.conf '!' -name config.zip '!' -name proc.json '!' -name '*.log' '!' -name flink_on_yarn.keytab '!' -name '*jceks' '!' -name supervisor_status -exec perl -pi -e 's#\{\{CMF_CONF_DIR}}#/var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN#g' '{}' ';'
+ make_scripts_executable
+ find /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN -regex '.*\.\(py\|sh\)$' -exec chmod u+x '{}' ';'
+ RUN_DIR=/var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN
+ '[' '' == true ']'
+ chmod u+x /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/scripts/control.sh
+ export COMMON_SCRIPT=/opt/cloudera/cm-agent/service/common/cloudera-config.sh
+ COMMON_SCRIPT=/opt/cloudera/cm-agent/service/common/cloudera-config.sh
+ exec /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/scripts/control.sh start
+ USAGE='Usage: control.sh (start|stop)'
+ OPERATION=start
+ case $OPERATION in
++ hostname -f
+ NODE_HOST=node01
+ '[' '!' -d /opt/cloudera/parcels/FLINK ']'
+ FLINK_HOME=/opt/cloudera/parcels/FLINK/lib/flink
+ FLINK_CONF_DIR=/var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf
+ '[' '!' -d /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf ']'
+ rm -rf /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/flink-conf.yaml /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/log4j-cli.properties /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/log4j-console.properties /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/log4j.properties /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/log4j-session.properties /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/logback-console.xml /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/logback-session.xml /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/logback.xml /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/masters /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/workers /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/zoo.cfg
+ cp /opt/cloudera/parcels/FLINK/lib/flink/conf/flink-conf.yaml /opt/cloudera/parcels/FLINK/lib/flink/conf/log4j-cli.properties /opt/cloudera/parcels/FLINK/lib/flink/conf/log4j-console.properties /opt/cloudera/parcels/FLINK/lib/flink/conf/log4j.properties /opt/cloudera/parcels/FLINK/lib/flink/conf/log4j-session.properties /opt/cloudera/parcels/FLINK/lib/flink/conf/logback-console.xml /opt/cloudera/parcels/FLINK/lib/flink/conf/logback-session.xml /opt/cloudera/parcels/FLINK/lib/flink/conf/logback.xml /opt/cloudera/parcels/FLINK/lib/flink/conf/masters /opt/cloudera/parcels/FLINK/lib/flink/conf/workers /opt/cloudera/parcels/FLINK/lib/flink/conf/zoo.cfg /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/
+ sed -i 's#=#: #g' /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf.properties
++ grep high-availability:
++ cat /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf.properties
+ HIGH_MODE='high-availability: zookeeper'
+ '[' 'high-availability: zookeeper' = '' ']'
++ grep high-availability.zookeeper.quorum:
++ cat /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf.properties
+ HIGH_ZK_QUORUM='high-availability.zookeeper.quorum: node01:2181'
+ '[' 'high-availability.zookeeper.quorum: node01:2181' = '' ']'
+ cp /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf.properties /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf/flink-conf.yaml
+ HADOOP_CONF_DIR=/var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/yarn-conf
+ export FLINK_HOME FLINK_CONF_DIR HADOOP_CONF_DIR
+ echo CONF_DIR: /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN
+ echo HADOOP_CONF_DIR: /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/yarn-conf
+ echo ''
++ date
+ echo 'Date: 2023年 12月 31日 星期日 10:45:59 CST'
+ echo 'Host: node01'
+ echo 'NODE_TYPE: '
+ echo 'ZK_QUORUM: node01:2181'
+ echo 'FLINK_HOME: /opt/cloudera/parcels/FLINK/lib/flink'
+ echo 'FLINK_CONF_DIR: /var/run/cloudera-scm-agent/process/603-flink_on_yarn-FLINK_YARN/flink-conf'
+ echo ''
+ '[' true = true ']'
+ exec /opt/cloudera/parcels/FLINK/lib/flink/bin/flink-yarn.sh --container 1 --streaming
CM管理页面查看异常日志，始终不能定位启动异常原因，此时查看flink详情启动日志，日志文件/var/log/flink/flink-yarn.out，详细错误日志如下图所示：
完整错误日志信息：
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/FLINK-1.18.0-BIN-SCALA_2.12/lib/flink/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/jars/hive-jdbc-2.1.1-cdh6.3.2-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/jars/avro-tools-1.8.2-cdh6.3.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/jars/slf4j-simple-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/jars/parquet-tools-1.9.0-cdh6.3.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
ERROR StatusLogger Reconfiguration failed: No configuration found for '7ea987ac' at 'null' in 'null'
22:57:15.334 [main] ERROR org.apache.flink.yarn.cli.FlinkYarnSessionCli - Error while running the Flink session.
org.apache.flink.client.deployment.ClusterDeploymentException: Couldn't deploy Yarn session cluster
at org.apache.flink.yarn.YarnClusterDescriptor.deploySessionCluster(YarnClusterDescriptor.java:444) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:604) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.lambda$main$4(FlinkYarnSessionCli.java:865) ~[flink-dist-1.18.0.jar:1.18.0]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_181]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_181]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875) ~[avro-tools-1.8.2-cdh6.3.2.jar:1.8.2-cdh6.3.2]
at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:865) [flink-dist-1.18.0.jar:1.18.0]
Caused by: org.apache.flink.yarn.YarnClusterDescriptor$YarnDeploymentException: The YARN application unexpectedly switched to state FAILED during deployment.
Diagnostics from YARN: Application application_1703945118355_0021 failed 1 times in previous 10000 milliseconds (global limit =2; local limit is =1) due to AM Container for appattempt_1703945118355_0021_000001 exited with exitCode: 239
Failing this attempt.Diagnostics: [2023-12-30 22:57:14.939]Exception from container-launch.
Container id: container_1703945118355_0021_01_000001
Exit code: 239
[2023-12-30 22:57:14.942]Container exited with a non-zero exit code 239. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
[2023-12-30 22:57:14.943]Container exited with a non-zero exit code 239. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
For more detailed output, check the application tracking page: http://node01:8088/cluster/app/application_1703945118355_0021 Then click on links to logs of each attempt.
. Failing the application.
If log aggregation is enabled on your cluster, use this command to further investigate the issue:
yarn logs -applicationId application_1703945118355_0021
at org.apache.flink.yarn.YarnClusterDescriptor.startAppMaster(YarnClusterDescriptor.java:1262) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.yarn.YarnClusterDescriptor.deployInternal(YarnClusterDescriptor.java:633) ~[flink-dist-1.18.0.jar:1.18.0]
at org.apache.flink.yarn.YarnClusterDescriptor.deploySessionCluster(YarnClusterDescriptor.java:437) ~[flink-dist-1.18.0.jar:1.18.0]
... 7 more
------------------------------------------------------------
The program finished with the following exception:
org.apache.flink.client.deployment.ClusterDeploymentException: Couldn't deploy Yarn session cluster
at org.apache.flink.yarn.YarnClusterDescriptor.deploySessionCluster(YarnClusterDescriptor.java:444)
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:604)
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.lambda$main$4(FlinkYarnSessionCli.java:865)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
at org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:865)
Caused by: org.apache.flink.yarn.YarnClusterDescriptor$YarnDeploymentException: The YARN application unexpectedly switched to state FAILED during deployment.
Diagnostics from YARN: Application application_1703945118355_0021 failed 1 times in previous 10000 milliseconds (global limit =2; local limit is =1) due to AM Container for appattempt_1703945118355_0021_000001 exited with exitCode: 239
Failing this attempt.Diagnostics: [2023-12-30 22:57:14.939]Exception from container-launch.
Container id: container_1703945118355_0021_01_000001
Exit code: 239
[2023-12-30 22:57:14.942]Container exited with a non-zero exit code 239. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
[2023-12-30 22:57:14.943]Container exited with a non-zero exit code 239. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
For more detailed output, check the application tracking page: http://node01:8088/cluster/app/application_1703945118355_0021 Then click on links to logs of each attempt.
. Failing the application.
If log aggregation is enabled on your cluster, use this command to further investigate the issue:
yarn logs -applicationId application_1703945118355_0021
at org.apache.flink.yarn.YarnClusterDescriptor.startAppMaster(YarnClusterDescriptor.java:1262)
at org.apache.flink.yarn.YarnClusterDescriptor.deployInternal(YarnClusterDescriptor.java:633)
at org.apache.flink.yarn.YarnClusterDescriptor.deploySessionCluster(YarnClusterDescriptor.java:437)
... 7 more
分析日志信息，猜测可能是flink 1.18.0和zookeeper版本不兼容，因为CDH默认zookeeper版本是zookeeper-3.4.6，而flink1.18可能因为版本较高，所以需要更高版本的zookeeper，因此考虑升级CDH的zookeeper版本到zookeeper 3.8。
升级步骤
1.maven仓库下载Cloudeara版本的zookeeper jar包，仓库地址：https://mvnrepository.com/artifact/org.apache.zookeeper/zookeeper?repo=cloudera-repos，如下图所示：
2.下载zookeeper 3.8
下载地址：https://www.apache.org/dyn/closer.lua/zookeeper/zookeeper-3.8.3/apache-zookeeper-3.8.3-bin.tar.gz
3.上传Cloudeara版本的zookeeper jar到CM下，即把zookeeper-3.8.1.7.1.9.0-387.jar包上传到/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/jars/下，并备份旧版本zookeeper即可。
cd /opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/jars/
mv zookeeper-3.4.5-cdh6.3.2.jar zookeeper-3.4.5-cdh6.3.2.jar.bak
mv zookeeper-3.4.5.jar zookeeper-3.4.5.jar.bak
如下图所示：
4.备份旧版本zookeeper的lib，上传下载apache-zookeeper-3.8.3-bin的lib目录下的jar包到/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/zookeeper目录下
如下图所示：
cd /opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/zookeeper
备份zookeeper的lib，mv lib lib.bak
上传apache-zookeeper-3.8.3-bin下的lib
如下图所示：
重新启动Flink-yarn，启动成功。
5.相关大数据学习demo地址:
https://github.com/carteryh/big-data
文章来源:https://blog.csdn.net/qq_21480329/article/details/135313722
本文来自互联网用户投稿，该文观点仅代表作者本人，不代表本站立场。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。 如若内容造成侵权/违法违规/事实不符，请联系我的编程经验分享网邮箱：veading@qq.com进行投诉反馈，一经查实，立即删除！
本文来自互联网用户投稿，该文观点仅代表作者本人，不代表本站立场。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。 如若内容造成侵权/违法违规/事实不符，请联系我的编程经验分享网邮箱：veading@qq.com进行投诉反馈，一经查实，立即删除！