1、问题描述

在ds上执行运行一个spark任务，yarn显示成功运行并有正常输出，ds日志也没有错误信息输出，但是ds的页面反馈为失败。

页面显示：

worker节点日志显示：

[ERROR] 2021-11-01 10:31:53.418 org.apache.dolphinscheduler.common.utils.HttpUtils:[73] - Connect to 192.168.xxx.xx3:8088 [/192.168.xxx.xx3] failed: 拒绝连接 (Connection refused)
org.apache.http.conn.HttpHostConnectException: Connect to 192.168.xxx.xx3:8088 [/192.168.210.203] failed: 拒绝连接 (Connection refused)at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151)at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)at org.apache.dolphinscheduler.common.utils.HttpUtils.get(HttpUtils.java:60)at org.apache.dolphinscheduler.common.utils.HadoopUtils$YarnHAAdminUtils.getRMState(HadoopUtils.java:656)at org.apache.dolphinscheduler.common.utils.HadoopUtils$YarnHAAdminUtils.getAcitveRMName(HadoopUtils.java:626)at org.apache.dolphinscheduler.common.utils.HadoopUtils.getAppAddress(HadoopUtils.java:568)at org.apache.dolphinscheduler.common.utils.HadoopUtils.getApplicationUrl(HadoopUtils.java:215)at org.apache.dolphinscheduler.common.utils.HadoopUtils.getApplicationStatus(HadoopUtils.java:421)at org.apache.dolphinscheduler.server.worker.task.AbstractCommandExecutor.isSuccessOfYarnState(AbstractCommandExecutor.java:403)at org.apache.dolphinscheduler.server.worker.task.AbstractCommandExecutor.run(AbstractCommandExecutor.java:233)at org.apache.dolphinscheduler.server.worker.task.AbstractYarnTask.handle(AbstractYarnTask.java:57)at org.apache.dolphinscheduler.server.worker.runner.TaskExecuteThread.run(TaskExecuteThread.java:133)at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)at java.util.concurrent.FutureTask.run(FutureTask.java:266)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: 拒绝连接 (Connection refused)at java.net.PlainSocketImpl.socketConnect(Native Method)at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)at java.net.Socket.connect(Socket.java:589)at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:74)at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134)... 24 common frames omitted
[INFO] 2021-11-01 10:31:53.419 org.apache.dolphinscheduler.common.utils.HadoopUtils:[217] - application url : http://null:8088/ws/v1/cluster/apps/%s
[INFO] 2021-11-01 10:31:53.852  - [taskAppId=TASK-11-43-90]:[129] -  -> 21/11/01 10:31:53 INFO Client: Application report for application_1633943332832_0156 (state: FINISHED)21/11/01 10:31:53 INFO Client: client token: N/Adiagnostics: N/AApplicationMaster host: 192.168.210.204ApplicationMaster RPC port: 0queue: defaultstart time: 1635733894208final status: SUCCEEDEDtracking URL: http://ambari4.slave.com:8088/proxy/application_1633943332832_0156/user: hyj21/11/01 10:31:53 INFO Client: Deleted staging directory hdfs://ambari1.master.com:8020/user/hyj/.sparkStaging/application_1633943332832_015621/11/01 10:31:53 INFO ShutdownHookManager: Shutdown hook called21/11/01 10:31:53 INFO ShutdownHookManager: Deleting directory /tmp/spark-35a3a881-d301-4c12-ba15-d7318ca0f153
[INFO] 2021-11-01 10:31:53.853  - [taskAppId=TASK-11-43-90]:[127] - FINALIZE_SESSION
[ERROR] 2021-11-01 10:31:58.440 org.apache.dolphinscheduler.common.utils.HttpUtils:[73] - null: 未知的名称或服务
java.net.UnknownHostException: null: 未知的名称或服务at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929)at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324)at java.net.InetAddress.getAllByName0(InetAddress.java:1277)at java.net.InetAddress.getAllByName(InetAddress.java:1193)at java.net.InetAddress.getAllByName(InetAddress.java:1127)at org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45)at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:111)at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353)at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:380)at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)at org.apache.dolphinscheduler.common.utils.HttpUtils.get(HttpUtils.java:60)at org.apache.dolphinscheduler.common.utils.HadoopUtils.getApplicationStatus(HadoopUtils.java:423)at org.apache.dolphinscheduler.server.worker.task.AbstractCommandExecutor.isSuccessOfYarnState(AbstractCommandExecutor.java:403)at org.apache.dolphinscheduler.server.worker.task.AbstractCommandExecutor.run(AbstractCommandExecutor.java:233)at org.apache.dolphinscheduler.server.worker.task.AbstractYarnTask.handle(AbstractYarnTask.java:57)at org.apache.dolphinscheduler.server.worker.runner.TaskExecuteThread.run(TaskExecuteThread.java:133)at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)at java.util.concurrent.FutureTask.run(FutureTask.java:266)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)
[ERROR] 2021-11-01 10:31:58.441  - [taskAppId=TASK-11-43-90]:[417] - yarn applications: [application_1633943332832_0156]  status failed 
java.lang.NullPointerException: nullat org.apache.dolphinscheduler.common.utils.HadoopUtils.getApplicationStatus(HadoopUtils.java:426)at org.apache.dolphinscheduler.server.worker.task.AbstractCommandExecutor.isSuccessOfYarnState(AbstractCommandExecutor.java:403)at org.apache.dolphinscheduler.server.worker.task.AbstractCommandExecutor.run(AbstractCommandExecutor.java:233)at org.apache.dolphinscheduler.server.worker.task.AbstractYarnTask.handle(AbstractYarnTask.java:57)at org.apache.dolphinscheduler.server.worker.runner.TaskExecuteThread.run(TaskExecuteThread.java:133)at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)at java.util.concurrent.FutureTask.run(FutureTask.java:266)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)

2、解决办法

经过排查还是配置文件install_config.conf的配置出现了问题，经查看集群的yarn-site.xml中的yarn并没有配置HA,yarn-site.xml的主要配置如下：

    <property><name>yarn.resourcemanager.ha.enabled</name><value>false</value></property><property><name>yarn.resourcemanager.hostname</name><value>192.168.xxx.xx6</value></property>

而我则在install_config.conf中错误的配置了下面的HA:

# if resourcemanager HA is enabled, please set the HA IPs; if resourcemanager is single, keep this value empty
yarnHaIps="192.168.xxx.xx3,192.168.xxx.xx4,192.168.xxx.xx5,192.168.xxx.xx6"# if resourcemanager HA is enabled or not use resourcemanager, please keep the default value; If resourcemanager is single, you only need to replace ds1 to actual resourcemanager hostname
singleYarnIp="192.168.xxx.xx6"

install_config.conf中正确的配置如下：

# if resourcemanager HA is enabled, please set the HA IPs; if resourcemanager is single, keep this value empty
yarnHaIps=""# if resourcemanager HA is enabled or not use resourcemanager, please keep the default value; If resourcemanager is single, you only need to replace ds1 to actual resourcemanager hostname
singleYarnIp="192.168.xxx.xx6"

正确配置之后，重新install.sh，集群恢复正常。