【doris踩坑记录】docker启动doris，重启后be无法注册到fe中

问题描述

使用docker搭建doris集群，使用docker-compose官方例子搭建：
，搭建完成后，重启docker 发现be就无法注册到fe中了。

搭建例子：https://doris.apache.org/zh-CN/docs/dev/install/construct-docker/run-docker-cluster

fe报错

doris-fe  | 2023-04-27 07:52:56,232 WARN (heartbeat mgr|34) [HeartbeatMgr.runAfterCatalogReady():139] get bad heartbeat response: type: BACKEND, status: BAD, msg: java.net.ConnectException: Connection refused (Connection refused), beId: 10003, beHost: 172.30.80.3, bePort: 0, httpPort: 0, brpcPort: 0
doris-fe  | 2023-04-27 07:53:01,247 WARN (heartbeat-mgr-pool-4|220) [HeartbeatMgr$BackendHeartbeatHandler.call():268] backend heartbeat got exception
doris-fe  | org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
doris-fe  |     at org.apache.thrift.transport.TSocket.open(TSocket.java:226) ~[libthrift-0.13.0.jar:0.13.0]
doris-fe  |     at org.apache.doris.common.GenericPool$ThriftClientFactory.create(GenericPool.java:143) ~[doris-fe.jar:1.2-SNAPSHOT]
doris-fe  |     at org.apache.doris.common.GenericPool$ThriftClientFactory.create(GenericPool.java:126) ~[doris-fe.jar:1.2-SNAPSHOT]
doris-fe  |     at org.apache.commons.pool2.BaseKeyedPooledObjectFactory.makeObject(BaseKeyedPooledObjectFactory.java:62) ~[commons-pool2-2.2.jar:2.2]
doris-fe  |     at org.apache.commons.pool2.impl.GenericKeyedObjectPool.create(GenericKeyedObjectPool.java:1012) ~[commons-pool2-2.2.jar:2.2]
doris-fe  |     at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:356) ~[commons-pool2-2.2.jar:2.2]
doris-fe  |     at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:277) ~[commons-pool2-2.2.jar:2.2]
doris-fe  |     at org.apache.doris.common.GenericPool.borrowObject(GenericPool.java:95) ~[doris-fe.jar:1.2-SNAPSHOT]
doris-fe  |     at org.apache.doris.system.HeartbeatMgr$BackendHeartbeatHandler.call(HeartbeatMgr.java:225) ~[doris-fe.jar:1.2-SNAPSHOT]
doris-fe  |     at org.apache.doris.system.HeartbeatMgr$BackendHeartbeatHandler.call(HeartbeatMgr.java:203) ~[doris-fe.jar:1.2-SNAPSHOT]
doris-fe  |     at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_342]
doris-fe  |     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_342]
doris-fe  |     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_342]
doris-fe  |     at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_342]
doris-fe  | Caused by: java.net.ConnectException: Connection refused (Connection refused)
doris-fe  |     at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_342]
doris-fe  |     at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_342]
doris-fe  |     at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_342]
doris-fe  |     at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_342]
doris-fe  |     at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_342]
doris-fe  |     at java.net.Socket.connect(Socket.java:607) ~[?:1.8.0_342]
doris-fe  |     at org.apache.thrift.transport.TSocket.open(TSocket.java:221) ~[libthrift-0.13.0.jar:0.13.0]
doris-fe  |     ... 13 more
doris-fe  | 2023-04-27 07:53:01,247 WARN (heartbeat mgr|34) [HeartbeatMgr.runAfterCatalogReady():139] get bad heartbeat response: type: BACKEND, status: BAD, msg: java.net.ConnectException: Connection refused (Connection refused), beId: 10003, beHost: 172.30.80.3, bePort: 0, httpPort: 0, brpcPort: 0

be报错

doris-be  | 2023-04-27T07:30:46+00:00 [Note] [Entrypoint]: masterFe = 172.30.80.2
doris-be  | 2023-04-27T07:30:46+00:00 [Note] [Entrypoint]: be_addr = 172.30.80.3:9050
doris-be  | 2023-04-27T07:30:46+00:00 [Warn] [Entrypoint]: start check be status~
doris-be  | 2023-04-27T07:30:46+00:00 [Note] [Entrypoint]: register is failed, wait next~
doris-be  | 2023-04-27T07:30:46+00:00 [Note] [Entrypoint]: feIpArray = 172.30.80.2
doris-be  | 2023-04-27T07:30:46+00:00 [Note] [Entrypoint]: feEditLogPortArray = 9010
doris-be  | 2023-04-27T07:30:46+00:00 [Note] [Entrypoint]: masterFe = 172.30.80.2:9010
doris-be  | 2023-04-27T07:30:46+00:00 [Note] [Entrypoint]: be_addr = 172.30.80.3:9050
doris-be  | 2023-04-27T07:30:46+00:00 [Note] [Entrypoint]: priority_networks = 172.30.80.0/24
doris-be  | 2023-04-27T07:30:46+00:00 [Note] [Entrypoint]: MASTER FE is not started. retry.
doris-be  | 2023-04-27T07:31:07+00:00 [Warn] [Entrypoint]: start check be status~
doris-be  | 2023-04-27T07:31:07+00:00 [Note] [Entrypoint]: register is failed, wait next~
doris-be  | 2023-04-27T07:31:07+00:00 [Note] [Entrypoint]: MASTER FE is not started. retry.
doris-be  | 2023-04-27T07:31:27+00:00 [Warn] [Entrypoint]: start check be status~
doris-be  | 2023-04-27T07:31:27+00:00 [Note] [Entrypoint]: register is failed, wait next~
doris-be  | 2023-04-27T07:31:27+00:00 [Note] [Entrypoint]: MASTER FE is not started. retry.
doris-be  | 2023-04-27T07:31:47+00:00 [Warn] [Entrypoint]: start check be status~
doris-be  | 2023-04-27T07:31:47+00:00 [Note] [Entrypoint]: MASTER FE is not started. retry.
doris-be  | 2023-04-27T07:31:47+00:00 [Note] [Entrypoint]: register is failed, wait next~
doris-be  | 2023-04-27T07:32:08+00:00 [Warn] [Entrypoint]: start check be status~
doris-be  | 2023-04-27T07:32:08+00:00 [Note] [Entrypoint]: MASTER FE is not started. retry.
doris-be  | 2023-04-27T07:32:08+00:00 [Note] [Entrypoint]: register is failed, wait next~
doris-be  | 2023-04-27T07:32:28+00:00 [Warn] [Entrypoint]: start check be status~
doris-be  | 2023-04-27T07:32:28+00:00 [Note] [Entrypoint]: MASTER FE is not started. retry.
doris-be  | 2023-04-27T07:32:28+00:00 [Note] [Entrypoint]: register is failed, wait next~
doris-be  | 2023-04-27T07:32:49+00:00 [Warn] [Entrypoint]: start check be status~
doris-be  | 2023-04-27T07:32:49+00:00 [Note] [Entrypoint]: MASTER FE is not started. retry.
doris-be  | 2023-04-27T07:32:49+00:00 [Note] [Entrypoint]: register is failed, wait next~
doris-be  | 2023-04-27T07:33:09+00:00 [Note] [Entrypoint]: MASTER FE is not started. retry.
doris-be  | 2023-04-27T07:33:09+00:00 [Warn] [Entrypoint]: start check be status~
doris-be  | 2023-04-27T07:33:09+00:00 [Note] [Entrypoint]: register is failed, wait next~
doris-be  | 2023-04-27T07:33:29+00:00 [Warn] [Entrypoint]: start check be status~
doris-be  | 2023-04-27T07:33:29+00:00 [Note] [Entrypoint]: MASTER FE is not started. retry.
doris-be  | 2023-04-27T07:33:29+00:00 [Note] [Entrypoint]: register is failed, wait next~
doris-be  | 2023-04-27T07:33:50+00:00 [Warn] [Entrypoint]: start check be status~
doris-be  | 2023-04-27T07:33:50+00:00 [Note] [Entrypoint]: MASTER FE is not started. retry.
doris-be  | 2023-04-27T07:33:50+00:00 [Note] [Entrypoint]: register is failed, wait next~
doris-be  | 2023-04-27T07:34:10+00:00 [Warn] [Entrypoint]: start check be status~
doris-be  | 2023-04-27T07:34:10+00:00 [Note] [Entrypoint]: MASTER FE is not started. retry.
doris-be  | 2023-04-27T07:34:10+00:00 [Note] [Entrypoint]: register is failed, wait next~
doris-be  | 2023-04-27T07:34:31+00:00 [Warn] [Entrypoint]: start check be status~
doris-be  | 2023-04-27T07:34:31+00:00 [Note] [Entrypoint]: MASTER FE is not started. retry.
doris-be  | 2023-04-27T07:34:31+00:00 [Note] [Entrypoint]: register is failed, wait next~
doris-be  | 2023-04-27T07:34:51+00:00 [Warn] [Entrypoint]: start check be status~
doris-be  | 2023-04-27T07:34:51+00:00 [Note] [Entrypoint]: MASTER FE is not started. retry.
doris-be  | 2023-04-27T07:34:51+00:00 [Note] [Entrypoint]: register is failed, wait next~
doris-be  | 2023-04-27T07:35:11+00:00 [Warn] [Entrypoint]: start check be status~
doris-be  | 2023-04-27T07:35:11+00:00 [Note] [Entrypoint]: MASTER FE is not started. retry.
doris-be  | 2023-04-27T07:35:11+00:00 [Note] [Entrypoint]: register is failed, wait next~
doris-be  | 2023-04-27T07:35:32+00:00 [Warn] [Entrypoint]: start check be status~
doris-be  | 2023-04-27T07:35:32+00:00 [Note] [Entrypoint]: MASTER FE is not started. retry.
doris-be  | 2023-04-27T07:35:32+00:00 [Note] [Entrypoint]: register is failed, wait next~
doris-be  | 2023-04-27T07:35:52+00:00 [Note] [Entrypoint]: BE is not register. retry.
doris-be  | 2023-04-27T07:36:13+00:00 [Note] [Entrypoint]: BE is not register. retry.
doris-be  | 2023-04-27T07:36:33+00:00 [Note] [Entrypoint]: BE is not register. retry.
doris-be  | 2023-04-27T07:36:53+00:00 [Note] [Entrypoint]: BE is not register. retry.
doris-be  | 2023-04-27T07:37:14+00:00 [Note] [Entrypoint]: BE is not register. retry.
doris-be  | 2023-04-27T07:37:34+00:00 [Note] [Entrypoint]: BE is not register. retry.
doris-be  | 2023-04-27T07:37:55+00:00 [Note] [Entrypoint]: BE is not register. retry.
doris-be  | 2023-04-27T07:38:15+00:00 [Note] [Entrypoint]: BE is not register. retry.
doris-be  | 2023-04-27T07:38:36+00:00 [Note] [Entrypoint]: BE is not register. retry.
doris-be  | 2023-04-27T07:38:56+00:00 [Note] [Entrypoint]: BE is not register. retry.
doris-be  | 2023-04-27T07:39:16+00:00 [Note] [Entrypoint]: BE is not register. retry.
doris-be  | 2023-04-27T07:39:37+00:00 [Note] [Entrypoint]: BE is not register. retry.
doris-be  | 2023-04-27T07:39:57+00:00 [Note] [Entrypoint]: BE is not register. retry.
doris-be  | 2023-04-27T07:40:18+00:00 [Note] [Entrypoint]: BE is not register. retry.
doris-be  | 2023-04-27T07:40:38+00:00 [Note] [Entrypoint]: BE is not register. retr

最后解决

发现是官方的一个bug，有位大神回答：
You can’t change password in this Docker IMAGE. Because in BE initiation process, its will execute show frontends using mysql-client. Therefore, if you change the password, the initiation bash script couldn’t execute sql. So an error occurred and BE couldn’t startup normally.

最后解决办法

可以试着给be启动的mysql命令里面加上自己新的密码, 在be容器里面, /usr/local/bin/中的entry_point.sh和init_be.sh脚本中, 使用mysql命令的部分, 加上-p[新密码]