手动修复 rabbitmq 报错 “Crash dump is being written to“

rabbitmq 报错:

2023-11-07 16:38:52.682 [error] emulator Error in process <0.368.0> on node '[e-mail]' with exit value:
{shutdown,[{mnesia_loader,handle_exit,2,[{file,"mnesia_loader.erl"},{line,963}]},{mnesia_loader,tab_receiver,5,[{file,"mnesia_loader.erl"},{line,440}]},{mnesia_loader,spawned_receiver,8,[{file,"mnesia_loader.erl"},{line,343}]}]}
2023-11-07 16:38:52.683 [error] emulator Error in process <0.367.0> on node '[e-mail]' with exit value:
{badarg,[{ets,insert,[mnesia_gvar,{last_error,{{shutdown,[{mnesia_loader,handle_exit,2,[{file,"mnesia_loader.erl"},{line,963}]},{mnesia_loader,tab_receiver,5,[{file,"mnesia_loader.erl"},{line,440}]},{mnesia_loader,spawned_receiver,8,[{file,"mnesia_loader.erl"},{line,343}]}]},[{mnesia_loader,wait_on_load_complete,1,[{file,"mnesia_loader.erl"},{line,359}]},{mnesia_tm,apply_fun,3,[{file,"mnesia_tm.erl"},{line,840}]},{mnesia_tm,execute_transaction,5,[{file,"mnesia_tm.erl"},{line,816}]},{mnesia_loader,init_receiver,5,[{file,"mnesia_loader.erl"},{line,285}]},{mnesia_loader,do_get_network_copy,5,[{file,"mnesia_loader.erl"},{line,221}]},{mnesia_controller,'-load_table_fun/1-fun-4-',5,[{file,"mnesia_controller.erl"},{line,2186}]},{mnesia_controller,'-load_and_reply/2-fun-0-',2,[{file,"mnesia_controller.erl"},{line,2133}]}]}}],[]},{mnesia_lib,set,2,[{file,"mnesia_lib.erl"},{line,443}]},{mnesia_lib,fix_error,1,[{file,"mnesia_lib.erl"},{line,906}]},{mnesia_tm,return_abort,3,[{file,"mnesia_tm.erl"},{line,962}]},{mnesia_loader,init_receiver,5,[{file,"mnesia_loader.erl"},{line,285}]},{mnesia_loader,do_get_network_copy,5,[{file,"mnesia_loader.erl"},{line,221}]},{mnesia_controller,'-load_table_fun/1-fun-4-',5,[{file,"mnesia_controller.erl"},{line,2186}]},{mnesia_controller,'-load_and_reply/2-fun-0-',2,[{file,"mnesia_controller.erl"},{line,2133}]}]}
2023-11-07 16:38:52.685 [info] <0.43.0> Application mnesia exited with reason: stopped
2023-11-07 16:38:52.685 [info] <0.43.0> Application tools exited with reason: stopped
2023-11-07 16:38:52.685 [error] <0.8.0> 
Error description:init:do_boot/3init:start_em/1rabbit:start_it/1 line 465rabbit:broker_start/1 line 341rabbit:start_loaded_apps/2 line 586app_utils:manage_applications/6 line 126lists:foldl/3 line 1263rabbit:'-handle_app_error/1-fun-0-'/3 line 709
throw:{could_not_start,ra,{ra,{{shutdown,{failed_to_start_child,ra_system_sup,{shutdown,{failed_to_start_child,ra_log_sup,{shutdown,{failed_to_start_child,ra_log_wal_sup,{shutdown,{failed_to_start_child,ra_log_wal,{{case_clause,{ok,<<>>}},[{ra_log_wal,open_existing,1,[{file,"src/ra_log_wal.erl"},{line,556}]},{ra_log_wal,'-recover_wal/2-lc$^0/1-0-',1,[{file,"src/ra_log_wal.erl"},{line,240}]},{ra_log_wal,recover_wal,2,[{file,"src/ra_log_wal.erl"},{line,243}]},{ra_log_wal,init,1,[{file,"src/ra_log_wal.erl"},{line,186}]},{gen_batch_server,init_it,6,[{file,"src/gen_batch_server.erl"},{line,125}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}}}}}}}}},{ra_app,start,[normal,[]]}}}}
Log file(s) (may contain more information):<stdout>BOOT FAILED
===========Error description:init:do_boot/3init:start_em/1rabbit:start_it/1 line 465rabbit:broker_start/1 line 341rabbit:start_loaded_apps/2 line 586app_utils:manage_applications/6 line 126lists:foldl/3 line 1263rabbit:'-handle_app_error/1-fun-0-'/3 line 709
throw:{could_not_start,ra,{ra,{{shutdown,{failed_to_start_child,ra_system_sup,{shutdown,{failed_to_start_child,ra_log_sup,{shutdown,{failed_to_start_child,ra_log_wal_sup,{shutdown,{failed_to_start_child,ra_log_wal,{{case_clause,{ok,<<>>}},[{ra_log_wal,open_existing,1,[{file,"src/ra_log_wal.erl"},{line,556}]},{ra_log_wal,'-recover_wal/2-lc$^0/1-0-',1,[{file,"src/ra_log_wal.erl"},{line,240}]},{ra_log_wal,recover_wal,2,[{file,"src/ra_log_wal.erl"},{line,243}]},{ra_log_wal,init,1,[{file,"src/ra_log_wal.erl"},{line,186}]},{gen_batch_server,init_it,6,[{file,"src/gen_batch_server.erl"},{line,125}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}}}}}}}}},{ra_app,start,[normal,[]]}}}}
Log file(s) (may contain more information):<stdout>{"init terminating in do_boot",{could_not_start,ra,{ra,{{shutdown,{failed_to_start_child,ra_system_sup,{shutdown,{failed_to_start_child,ra_log_sup,{shutdown,{failed_to_start_child,ra_log_wal_sup,{shutdown,{failed_to_start_child,ra_log_wal,{{case_clause,{ok,<<>>}},[{ra_log_wal,open_existing,1,[{file,"src/ra_log_wal.erl"},{line,556}]},{ra_log_wal,'-recover_wal/2-lc$^0/1-0-',1,[{file,"src/ra_log_wal.erl"},{line,240}]},{ra_log_wal,recover_wal,2,[{file,"src/ra_log_wal.erl"},{line,243}]},{ra_log_wal,init,1,[{file,"src/ra_log_wal.erl"},{line,186}]},{gen_batch_server,init_it,6,[{file,"src/gen_batch_server.erl"},{line,125}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]}}}}}}}}},{ra_app,start,[normal,[]]}}}}}
init terminating in do_boot ({could_not_start,ra,{ra,{{shutdown,{_}},{ra_app,start,[_]}}}})Crash dump is being written to: /var/log/rabbitmq/erl_crash.dump...done

修复方法:
(1) 找到 rabbitmq 使用的 pv，例如: rabbitmq-0 的 pod:

# kubectl get pv | grep rabbitmq-0
pvc-70ed48bf-bef8-4658-b530-1fd3a6ef5937   200Gi      RWO            Delete           Bound    openstack/rabbitmq-data-rabbitmq-0                                    ceph-ssd                6d17h

(2) 找到 pv 使用的信息:

# kubectl get pv pvc-70ed48bf-bef8-4658-b530-1fd3a6ef5937 -o yaml
apiVersion: v1
kind: PersistentVolume
metadata:annotations:kubernetes.io/createdby: rbd-dynamic-provisionerpv.kubernetes.io/bound-by-controller: "yes"pv.kubernetes.io/provisioned-by: kubernetes.io/rbdcreationTimestamp: "2023-10-31T15:40:59Z"finalizers:- kubernetes.io/pv-protectionname: pvc-70ed48bf-bef8-4658-b530-1fd3a6ef5937resourceVersion: "7552"uid: 6848417a-dd4f-430c-85e5-f3234a1ac6bf
spec:accessModes:- ReadWriteOncecapacity:storage: 200GiclaimRef:apiVersion: v1kind: PersistentVolumeClaimname: rabbitmq-data-rabbitmq-0namespace: openstackresourceVersion: "4704"uid: 70ed48bf-bef8-4658-b530-1fd3a6ef5937persistentVolumeReclaimPolicy: Deleterbd:image: kubernetes-dynamic-pvc-c8a3585f-dc7b-438c-a22e-cca9d84c341fkeyring: /etc/ceph/keyringmonitors:- ceph-mon.ceph.svc.cluster.local:6789pool: ssdpoolsecretRef:name: pvc-ceph-client-keyuser: adminstorageClassName: ceph-ssdvolumeMode: Filesystem
status:phase: Bound

需要的信息:

    image: kubernetes-dynamic-pvc-c8a3585f-dc7b-438c-a22e-cca9d84c341f

(3) 在 pod 节点上查看对应的物理设备

# ssh node-2 rbd showmapped | grep kubernetes-dynamic-pvc-c8a3585f-dc7b-438c-a22e-cca9d84c341f
0  ssdpool           kubernetes-dynamic-pvc-c8a3585f-dc7b-438c-a22e-cca9d84c341f -    /dev/rbd0  

(4) 查看设备挂载目录

# ssh node-2 mount | grep rbd0
/dev/rbd0 on /var/lib/kubelet/plugins/kubernetes.io/rbd/mounts/ssdpool-image-kubernetes-dynamic-pvc-c8a3585f-dc7b-438c-a22e-cca9d84c341f type ext4 (rw,relatime,stripe=1024)
/dev/rbd0 on /var/lib/kubelet/pods/3a37e264-4fd5-4cb8-844b-6b6cd4a6859c/volumes/kubernetes.io~rbd/pvc-70ed48bf-bef8-4658-b530-1fd3a6ef5937 type ext4 (rw,relatime,stripe=1024)

(5) 查找 wal 文件路径，查找的路径来自步骤 (4)

# ssh node-2 find /var/lib/kubelet/pods/3a37e264-4fd5-4cb8-844b-6b6cd4a6859c/volumes/kubernetes.io~rbd/pvc-70ed48bf-bef8-4658-b530-1fd3a6ef5937 -name "*.wal"
/var/lib/kubelet/pods/3a37e264-4fd5-4cb8-844b-6b6cd4a6859c/volumes/kubernetes.io~rbd/pvc-70ed48bf-bef8-4658-b530-1fd3a6ef5937/mnesia/[e-mail]/quorum/[e-mail]/00000025.wal

(6) 删除 wal 文件
此步骤请慎重操作，建议将文件备份后再操作。

# ssh node-2 rm -rf /var/lib/kubelet/pods/3a37e264-4fd5-4cb8-844b-6b6cd4a6859c/volumes/kubernetes.io~rbd/pvc-70ed48bf-bef8-4658-b530-1fd3a6ef5937/mnesia/[e-mail]/quorum/[e-mail]/00000025.wal
Warning: Permanently added 'node-2' (ED25519) to the list of known hosts.

(7) 删除 pod，重新启动 pod

# kubectl delete pods rabbitmq-0 -n openstack 
pod "rabbitmq-0" deleted

等待 pod 再次启动，过一会重新数据同步恢复。