GBASE南大通用8a MPP 数据库memlog使用方法及问题分析示例

2023-12-28 06:47:38

GBASE南大通用8a MPP集群在遇到疑难问题，特别是性能严重异常、现象特别异常时，还有一种关键的故障排查方法，可以跟踪内存的处理日志——memlog；memlog配置方法如下，需要手动配置或下发memlog的配置参数，其中_gbase_inmem_log_level和_gbase_inmem_log_source可以热切，无需重启gbased服务

_gbase_inmem_log_level 参数说明：

0: Fatal

1: Error

2: Warn

3: Info

4: Debug

_gbase_inmem_log_source 参数说明：

1: GENERIC

2: REP

4: CONN

8: SESSION

16: PACKETS

32: EVENTS

64: LOCK

128: CTQ

256: WINS

如果要同时显示多种类型，则set时，可以将数位相加，如显示REP和CONN，则set [global] _gbase_inmem_log_source=6，如果要写入配置文件，则需要在配置文件中，将具体显示类型写入配置文件:_gbase_inmem_log_source=REP,CONN

GBase南大通用8a MPP集群排查问题举例：网络通信异常排查

当trc中出现某个通信阶段消耗的时间较长时，可以通过memlog协助判断gbase 内部通信的问题。

1、memlog打开参数样例

set global _gbase_inmem_log_level=4;

set global _gbase_inmem_log_source=2;

2、memlog使用方法：

1）列出memlog清单

[gbase@localhost ~]$ memlog -l

memlog: find 3 log files, as bellow:

shm_gclusterd_5258

shm_gbased_192.168.80.129_5050

shm_gbased_192.168.80.128_5050

2）对某个实例的memlog进行解析

memlog shm_gbased_192.168.80.129_5050

3、通信的角色和流程

一般内部数据复制（REP）的通信流程：

LINK_INIT ——> OPEN_TABLE ——> GET_TABLE_INFO ——> SendData（发送数据的日志在memlog中不显示） ——> Finish ——> Destroy

4、memlog示例

[gbase@localhost ~]$ memlog shm_gbased_192.168.80.128_5050 【注释】memlog用法

TIME TID LEVEL SOURCE Log

211208 15:39:08 140562791978752 [Info] [REP] GBaseRep.h:0169: Creating Rep instance: 0x5392300?【注释】实例编号

211208 15:39:08 140562791978752 [Info] [REP] GBaseRep.cpp:4644: Rep 0x5392300 opened.

211208 15:39:09 140562790913792 [Info] [REP] GBaseRep.h:0169: Creating Rep instance: 0x5392d00

211208 15:39:09 140562790913792 [Info] [REP] GBaseRep.cpp:4644: Rep 0x5392d00 opened.

211208 15:39:09 140562790913792 [Info] [REP] RepActor.cpp:0131: New REP opened: 0x5392d00, thd: 0x5432000

211208 15:39:09 140562791978752 [Info] [REP] GBaseRep.cpp:1179: GBaseRep(0x5392300): role(Initiator), state(Opened), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Sending Cmd: LINK_INIT to 192.168.80.128:gctmpdb._tmp_2152769728_7_t1_1_1638949066_s, group_id(0), host_gr?【注释】LINK_INIT 发起通信

211208 15:39:09 140562791978752 [Info] [REP] GBaseRep.cpp:1194: GBaseRep(0x5392300): role(Initiator), state(Opened), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Sent LINK_INIT to 192.168.80.128:5050, gctmpdb._tmp_2152769728_7_t1_1_1638949066_s

211208 15:39:09 140562790647552 [Info] [REP] GBaseRep.h:0169: Creating Rep instance: 0x5393200

211208 15:39:09 140562790647552 [Info] [REP] GBaseRep.cpp:4644: Rep 0x5393200 opened.

211208 15:39:09 140562790647552 [Info] [REP] RepActor.cpp:0131: New REP opened: 0x5393200, thd: 0x544a000

211208 15:39:09 140562504722176 [Info] [REP] GBaseRep.cpp:3541: GBaseRep(0x5392d00): role(Unknown), state(Opened), scn(-1), src_id() -- Cmd: 0(LINK_INIT) from: 92300 group_id: 0

211208 15:39:09 140562504722176 [Info] [REP] GBaseRep.cpp:4255: GBaseRep(0x5392d00): role(Unknown), state(Opened), scn(-1), src_id() -- GBaseRep::ReceiveLinkInit <<<<<<<<

211208 15:39:09 140562504722176 [Info] [REP] GBaseRep.cpp:4271: GBaseRep(0x5392d00): role(Unknown), state(Opened), scn(-1), src_id() -- GBaseRep::SaveTableName <<<<<<<<

211208 15:39:09 140562504722176 [Info] [REP] GBaseRep.cpp:4342: GBaseRep(0x5392d00): role(Unknown), state(Opened), scn(0), src_id(localhost.localdomain_8_0x5392300) -- ReadRepTable: host(192.168.80.129), table(gctmpdb._tmp_2152769728_7_t1_1_1638949066_s)

211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:3541: GBaseRep(0x5393200): role(Unknown), state(Opened), scn(-1), src_id() -- Cmd: 0(LINK_INIT) from: e4300 group_id: 0

211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:4255: GBaseRep(0x5393200): role(Unknown), state(Opened), scn(-1), src_id() -- GBaseRep::ReceiveLinkInit <<<<<<<<

211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:4271: GBaseRep(0x5393200): role(Unknown), state(Opened), scn(-1), src_id() -- GBaseRep::SaveTableName <<<<<<<<

211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:4342: GBaseRep(0x5393200): role(Unknown), state(Opened), scn(0), src_id(localhost.localdomain_8_0x50e4300) -- ReadRepTable: host(192.168.80.129), table(gctmpdb._tmp_2152769728_7_t1_1_1638949066_s)

211208 15:39:09 140562504722176 [Info] [REP] GBaseRep.cpp:1179: GBaseRep(0x5392d00): role(Repeater), state(Opened), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Sending Cmd: LINK_INIT to 192.168.80.129:gctmpdb._tmp_2152769728_7_t1_1_1638949066_s, group_id(0), host_gro

211208 15:39:09 140562504722176 [Info] [REP] GBaseRep.cpp:1194: GBaseRep(0x5392d00): role(Repeater), state(Opened), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Sent LINK_INIT to 192.168.80.129:5050, gctmpdb._tmp_2152769728_7_t1_1_1638949066_s

211208 15:39:09 140562504722176 [Info] [REP] GBaseRep.cpp:4586: GBaseRep(0x5392d00): role(Repeater), state(LinkReady), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Sending Cmd: LINK_INIT_ACK to 192.168.80.128, group_id: 0

211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:1179: GBaseRep(0x5393200): role(Repeater), state(Opened), scn(0), src_id(localhost.localdomain_8_0x50e4300) -- Sending Cmd: LINK_INIT to 192.168.80.129:gctmpdb._tmp_2152769728_7_t1_1_1638949066_s, group_id(0), host_gro

211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:1194: GBaseRep(0x5393200): role(Repeater), state(Opened), scn(0), src_id(localhost.localdomain_8_0x50e4300) -- Sent LINK_INIT to 192.168.80.129:5050, gctmpdb._tmp_2152769728_7_t1_1_1638949066_s

211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:4586: GBaseRep(0x5393200): role(Repeater), state(LinkReady), scn(0), src_id(localhost.localdomain_8_0x50e4300) -- Sending Cmd: LINK_INIT_ACK to 192.168.80.129, group_id: 0

211208 15:39:09 140562791978752 [Info] [REP] GBaseRep.cpp:3351: GBaseRep(0x5392300): role(Initiator), state(SettingLink), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Cmd: 1(LINK_INIT_ACK) from: 92d00 group_id: 0?【注释】收到LINK_INIT_ACK

211208 15:39:09 140562791978752 [Info] [REP] GBaseRep.cpp:1214: GBaseRep(0x5392300): role(Initiator), state(LinkReady), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Sending Cmd: OPEN_TABLE to 192.168.80.128, group_id: 0?【注释】发起OPEN_TABLE

211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:3541: GBaseRep(0x5392d00): role(Repeater), state(LinkReady), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Cmd: 2(OPEN_TABLE) from: 92300 group_id: 0

211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:4265: GBaseRep(0x5392d00): role(Repeater), state(LinkReady), scn(0), src_id(localhost.localdomain_8_0x5392300) -- GBaseRep::ReceiveOpenTable <<<<<<<<

211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:4608: GBaseRep(0x5392d00): role(Repeater), state(LinkReady), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Sending Cmd: OPEN_TABLE_ACK to 192.168.80.128, group_id: 0

211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:1214: GBaseRep(0x5392d00): role(Repeater), state(LinkReady), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Sending Cmd: OPEN_TABLE to 192.168.80.129, group_id: 0

211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:1226: GBaseRep(0x5392d00): role(Repeater), state(LinkReady), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Sending Cmd: GET_TABLE_INFO

211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:1277: GBaseRep(0x5392d00): role(Repeater), state(LinkReady), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Sending Cmd: GET_TABLE_INFO to 192.168.80.129:gctmpdb._tmp_2152769728_7_t1_1_1638949066_s, group_id(0)

211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:1045: GBaseRep(0x5392d00): role(Repeater), state(WaitingTableInfo), scn(0), src_id(localhost.localdomain_8_0x5392300) -- GBaseRep::PrepareAllReceiveTable, table opened & locked...

211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:3541: GBaseRep(0x5393200): role(Repeater), state(LinkReady), scn(0), src_id(localhost.localdomain_8_0x50e4300) -- Cmd: 2(OPEN_TABLE) from: e4300 group_id: 0

211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:4265: GBaseRep(0x5393200): role(Repeater), state(LinkReady), scn(0), src_id(localhost.localdomain_8_0x50e4300) -- GBaseRep::ReceiveOpenTable <<<<<<<<

211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:4608: GBaseRep(0x5393200): role(Repeater), state(LinkReady), scn(0), src_id(localhost.localdomain_8_0x50e4300) -- Sending Cmd: OPEN_TABLE_ACK to 192.168.80.129, group_id: 0

211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:1214: GBaseRep(0x5393200): role(Repeater), state(LinkReady), scn(0), src_id(localhost.localdomain_8_0x50e4300) -- Sending Cmd: OPEN_TABLE to 192.168.80.129, group_id: 0

211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:1226: GBaseRep(0x5393200): role(Repeater), state(LinkReady), scn(0), src_id(localhost.localdomain_8_0x50e4300) -- Sending Cmd: GET_TABLE_INFO

211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:1277: GBaseRep(0x5393200): role(Repeater), state(LinkReady), scn(0), src_id(localhost.localdomain_8_0x50e4300) -- Sending Cmd: GET_TABLE_INFO to 192.168.80.129:gctmpdb._tmp_2152769728_7_t1_1_1638949066_s, group_id(0)

211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:1045: GBaseRep(0x5393200): role(Repeater), state(WaitingTableInfo), scn(0), src_id(localhost.localdomain_8_0x50e4300) -- GBaseRep::PrepareAllReceiveTable, table opened & locked...

211208 15:39:09 140562791978752 [Info] [REP] GBaseRep.cpp:3351: GBaseRep(0x5392300): role(Initiator), state(LinkReady), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Cmd: 3(OPEN_TABLE_ACK) from: 92d00 group_id: 0?【注释】收到OPEN_TABLE_ACK

211208 15:39:09 140562791978752 [Info] [REP] GBaseRep.cpp:1226: GBaseRep(0x5392300): role(Initiator), state(LinkReady), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Sending Cmd: GET_TABLE_INFO?【注释】获取表信息

211208 15:39:09 140562791978752 [Info] [REP] GBaseRep.cpp:1277: GBaseRep(0x5392300): role(Initiator), state(LinkReady), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Sending Cmd: GET_TABLE_INFO to 192.168.80.128:gctmpdb._tmp_2152769728_7_t1_1_1638949066_s, group_id(0)

211208 15:39:09 140562504722176 [Info] [REP] GBaseRep.cpp:3541: GBaseRep(0x5392d00): role(Repeater), state(WaitingTableInfo), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Cmd: 4(GET_TABLE_INFO) from: 92300 group_id: 0

211208 15:39:09 140562504722176 [Info] [REP] GBaseRep.cpp:3582: GBaseRep(0x5392d00): role(Repeater), state(WaitingTableInfo), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Skipped Cmd: SISC_GET_TABLE_INFO in state: WaitingTableInfo

211208 15:39:09 140562504722176 [Info] [REP] GBaseRep.cpp:3541: GBaseRep(0x5393200): role(Repeater), state(WaitingTableInfo), scn(0), src_id(localhost.localdomain_8_0x50e4300) -- Cmd: 4(GET_TABLE_INFO) from: e4300 group_id: 0

211208 15:39:09 140562504722176 [Info] [REP] GBaseRep.cpp:3582: GBaseRep(0x5393200): role(Repeater), state(WaitingTableInfo), scn(0), src_id(localhost.localdomain_8_0x50e4300) -- Skipped Cmd: SISC_GET_TABLE_INFO in state: WaitingTableInfo

211208 15:39:09 140562504722176 [Info] [REP] GBaseRep.cpp:3541: GBaseRep(0x5392d00): role(Repeater), state(TableInfoReplyPending), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Cmd: 1(LINK_INIT_ACK) from: e4d00 group_id: 0

211208 15:39:09 140562504722176 [Info] [REP] GBaseRep.cpp:3541: GBaseRep(0x5393200): role(Repeater), state(TableInfoReplyPending), scn(0), src_id(localhost.localdomain_8_0x50e4300) -- Cmd: 1(LINK_INIT_ACK) from: e5200 group_id: 0

211208 15:39:09 140562504722176 [Info] [REP] GBaseRep.cpp:1314: GBaseRep(0x5392d00): role(Repeater), state(TableInfoReplyPending), scn(0), src_id(localhost.localdomain_8_0x5392300) -- table: 0x3335180, table_info: 0x32a5560

211208 15:39:09 140562504722176 [Info] [REP] GBaseRep.cpp:4383: GBaseRep(0x5392d00): role(Repeater), state(TableInfoReceived), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Sending Cmd: BACK_TABLE_INFO to 192.168.80.128, group_id(0)

211208 15:39:09 140562504722176 [Info] [REP] GBaseRep.cpp:1314: GBaseRep(0x5393200): role(Repeater), state(TableInfoReplyPending), scn(0), src_id(localhost.localdomain_8_0x50e4300) -- table: 0x3335360, table_info: 0x32a55f0

211208 15:39:09 140562504722176 [Info] [REP] GBaseRep.cpp:4383: GBaseRep(0x5393200): role(Repeater), state(TableInfoReceived), scn(0), src_id(localhost.localdomain_8_0x50e4300) -- Sending Cmd: BACK_TABLE_INFO to 192.168.80.129, group_id(0)

211208 15:39:09 140562791978752 [Info] [REP] GBaseRep.cpp:3351: GBaseRep(0x5392300): role(Initiator), state(WaitingTableInfo), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Cmd: 5(BACK_TABLE_INFO) from: 92d00 group_id: 0?【注释】返回表信息

211208 15:39:09 140562791978752 [Info] [REP] GBaseRep.cpp:1314: GBaseRep(0x5392300): role(Initiator), state(WaitingTableInfo), scn(0), src_id(localhost.localdomain_8_0x5392300) -- table: 0x46b4b40, table_info: 0x32a4d80?【注释】收到具体表地址

211208 15:39:09 140562791978752 [Info] [REP] GBaseRep.cpp:0507: GBaseRep(0x5392300): role(Initiator), state(WaitingTableInfo), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Sending Cmd: FINISH to 192.168.80.128, group_id: -1?【注释】FINISH

211208 15:39:09 140562791978752 [Info] [REP] GBaseRep.cpp:0514: GBaseRep(0x5392300): role(Initiator), state(Closing), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Receiving Cmd ACK: FINISH from 192.168.80.128, group_id: -1

211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:3541: GBaseRep(0x5392d00): role(Repeater), state(TableDataReceived), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Cmd: 20(FINISH) from: 92300 group_id: -1

211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:0507: GBaseRep(0x5392d00): role(Repeater), state(TableDataReceived), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Sending Cmd: FINISH to 192.168.80.129, group_id: -1

211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:3541: GBaseRep(0x5393200): role(Repeater), state(TableDataReceived), scn(0), src_id(localhost.localdomain_8_0x50e4300) -- Cmd: 20(FINISH) from: e4300 group_id: -1

211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:0507: GBaseRep(0x5393200): role(Repeater), state(TableDataReceived), scn(0), src_id(localhost.localdomain_8_0x50e4300) -- Sending Cmd: FINISH to 192.168.80.129, group_id: -1

211208 15:39:09 140562504722176 [Info] [REP] GBaseRep.cpp:3541: GBaseRep(0x5392d00): role(Repeater), state(Closing), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Cmd: 21(FINISH_ACK) from: e4d00 group_id: -1

211208 15:39:09 140562504722176 [Info] [REP] GBaseRep.cpp:3755: GBaseRep(0x5392d00): role(Repeater), state(Closing), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Streams still open: 0

211208 15:39:09 140562504722176 [Info] [REP] GBaseRep.cpp:4617: GBaseRep(0x5392d00): role(Repeater), state(Closing), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Sending Cmd: FINISH_ACK to 192.168.80.128, group_id: -1

211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:3541: GBaseRep(0x5393200): role(Repeater), state(Closing), scn(0), src_id(localhost.localdomain_8_0x50e4300) -- Cmd: 21(FINISH_ACK) from: e5200 group_id: -1

211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:3755: GBaseRep(0x5393200): role(Repeater), state(Closing), scn(0), src_id(localhost.localdomain_8_0x50e4300) -- Streams still open: 0

211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:4617: GBaseRep(0x5393200): role(Repeater), state(Closing), scn(0), src_id(localhost.localdomain_8_0x50e4300) -- Sending Cmd: FINISH_ACK to 192.168.80.129, group_id: -1

211208 15:39:09 140562504722176 [Info] [REP] GBaseRep.cpp:4699: Closed 0x5392d00, thd: 0x5432000, opened: 0, role: Repeater

211208 15:39:09 140562504722176 [Info] [REP] GBaseRep.cpp:5173: Destroying Rep instance: 0x5392d00

211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:4699: Closed 0x5393200, thd: 0x544a000, opened: 0, role: Repeater

211208 15:39:09 140562496329472 [Info] [REP] GBaseRep.cpp:5173: Destroying Rep instance: 0x5393200

211208 15:39:09 140562791978752 [Info] [REP] GBaseRep.cpp:3351: GBaseRep(0x5392300): role(Initiator), state(Closing), scn(0), src_id(localhost.localdomain_8_0x5392300) -- Cmd: 21(FINISH_ACK) from: 92d00 group_id: -1?【注释】收到FINISH_ACK

211208 15:39:09 140562791978752 [Info] [REP] GBaseRep.cpp:4699: Closed 0x5392300, thd: 0x5378000, opened: 0, role: Initiator

211208 15:39:09 140562791978752 [Info] [REP] GBaseRep.cpp:5173: Destroying Rep instance: 0x5392300?【注释】销毁通信实例

Total logs: 78

以上示例记录了正常数据复制的详细记录，当数据复制的过程出现问题时，可以通过menlong做一个深入诊断，判断当前处理机制是否存在问题；

memlog可以显示多种类型的日志，比如CONN、SESSION、PACKETS等，可以通过配置参数进行设置，方便用户分析排查；

最后，memlog需要结合专业的产品知识做为基础，才能帮助排查分析问题，在一定条件下，也可以通过该手段协助专业工程师进行信息收集，提高排查问题的效率。

文章来源:https://blog.csdn.net/weixin_47390342/article/details/135253530
本文来自互联网用户投稿，该文观点仅代表作者本人，不代表本站立场。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如若内容造成侵权/违法违规/事实不符，请联系我的编程经验分享网邮箱：veading@qq.com进行投诉反馈，一经查实，立即删除！