当前位置：首页 > news >正文

K8s环境下rook-v1.13.3部署Ceph-v18.2.1集群

news 2026/2/7 12:11:01

文章目录

1.K8s环境搭建
2.Ceph集群部署
- 2.1 部署Rook Operator
- 2.2 镜像准备
- 2.3 配置节点角色
- 2.4 部署operator
- 2.5 部署Ceph集群
- 2.6 强制删除命名空间
- 2.7 验证集群
3.Ceph界面

1.K8s环境搭建

参考：CentOS7搭建k8s-v1.28.6集群详情，把K8s集群完成搭建，再进行Ceph集群搭建

2.Ceph集群部署

2.1 部署Rook Operator

# 下载 rook 项目，相关源码（2024-02-06最新版本）
git clone --single-branch --branch v1.13.3 https://github.com/rook/rook.git
# 如果网络不好，可以下载：rook-1.13.3.tar.gz
# https://gh.api.99988866.xyz/https://github.com/rook/rook/archive/refs/tags/v1.13.3.tar.gz
# 解压：tar -zxvf rook-1.13.3.tar.gz[root@ceph61 ~]# git clone --single-branch --branch v1.13.3 https://github.com/rook/rook.git
Cloning into 'rook'...
remote: Enumerating objects: 95134, done.
remote: Counting objects: 100% (5514/5514), done.
remote: Compressing objects: 100% (348/348), done.
remote: Total 95134 (delta 5299), reused 5203 (delta 5166), pack-reused 89620
Receiving objects: 100% (95134/95134), 50.70 MiB | 10.43 MiB/s, done.
Resolving deltas: 100% (66720/66720), done.
Note: checking out '54663a4333bb72ba2114f387048775a619f2344d'.You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:git checkout -b new_branch_name

2.2 镜像准备

# 第一步：检查需要下载的镜像
[root@ceph61 examples]# cat images.txtgcr.io/k8s-staging-sig-storage/objectstorage-sidecar/objectstorage-sidecar:v20230130-v0.1.0-24-gc0cf995quay.io/ceph/ceph:v18.2.1quay.io/ceph/cosi:v0.1.1quay.io/cephcsi/cephcsi:v3.10.1quay.io/csiaddons/k8s-sidecar:v0.8.0registry.k8s.io/sig-storage/csi-attacher:v4.4.2registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.9.1registry.k8s.io/sig-storage/csi-provisioner:v3.6.3registry.k8s.io/sig-storage/csi-resizer:v1.9.2registry.k8s.io/sig-storage/csi-snapshotter:v6.3.2rook/ceph:v1.13.3
# 修改为：gcr.io/k8s-staging-sig-storage/objectstorage-sidecar/objectstorage-sidecar:v20230130-v0.1.0-24-gc0cf995quay.io/ceph/ceph:v18.2.1quay.io/ceph/cosi:v0.1.1quay.io/cephcsi/cephcsi:v3.10.1quay.io/csiaddons/k8s-sidecar:v0.8.0registry.cn-hangzhou.aliyuncs.com/google_containers/csi-attacher:v4.4.2registry.cn-hangzhou.aliyuncs.com/google_containers/csi-node-driver-registrar:v2.9.1registry.cn-hangzhou.aliyuncs.com/google_containers/csi-provisioner:v3.6.3registry.cn-hangzhou.aliyuncs.com/google_containers/csi-resizer:v1.9.2registry.cn-hangzhou.aliyuncs.com/google_containers/csi-snapshotter:v6.3.2rook/ceph:v1.13.3# 第2步：登录阿里镜像源
docker login --username=你的阿里云账号 registry.cn-beijing.aliyuncs.com
输入你的阿里云账号密码# 第2步：手动下载镜像
# 不知道如何下载（此次部署集群，该镜像不下载也没有影响）：gcr.io/k8s-staging-sig-storage/objectstorage-sidecar/objectstorage-sidecar:v20230130-v0.1.0-24-gc0cf995
docker pull quay.io/ceph/ceph:v18.2.1
docker pull quay.io/ceph/cosi:v0.1.1
docker pull quay.io/cephcsi/cephcsi:v3.10.1
docker pull quay.io/csiaddons/k8s-sidecar:v0.8.0
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/csi-attacher:v4.4.2
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/csi-node-driver-registrar:v2.9.1
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/csi-provisioner:v3.6.3
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/csi-resizer:v1.9.2
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/csi-snapshotter:v6.3.2
docker pull rook/ceph:v1.13.3# 第4步：重新打tag
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/csi-attacher:v4.4.2 registry.k8s.io/sig-storage/csi-attacher:v4.4.2
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/csi-node-driver-registrar:v2.9.1 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.9.1
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/csi-provisioner:v3.6.3 registry.k8s.io/sig-storage/csi-provisioner:v3.6.3
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/csi-resizer:v1.9.2 registry.k8s.io/sig-storage/csi-resizer:v1.9.2
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/csi-snapshotter:v6.3.2 registry.k8s.io/sig-storage/csi-snapshotter:v6.3.2

2.3 配置节点角色

# 查看节点标签
kubectl get nodes --show-labels
kubectl get nodes <node-name> --show-labels[root@ceph61 examples]# kubectl get nodes --show-labels
NAME     STATUS   ROLES           AGE   VERSION   LABELS
ceph61   Ready    control-plane   47h   v1.28.2   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=ceph61,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=,node.kubernetes.io/exclude-from-external-load-balancers=
ceph62   Ready    <none>          47h   v1.28.2   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=ceph62,kubernetes.io/os=linux
ceph63   Ready    <none>          47h   v1.28.2   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=ceph63,kubernetes.io/os=linux# 设置ceph61、ceph62、ceph63节点标签
kubectl label nodes {ceph61,ceph62,ceph63} ceph-osd=enabled
kubectl label nodes {ceph61,ceph62,ceph63} ceph-mon=enabled
kubectl label nodes ceph61 ceph-mgr=enabled[root@ceph61 examples]# kubectl get nodes --show-labels
NAME     STATUS   ROLES           AGE   VERSION   LABELS
ceph61   Ready    control-plane   47h   v1.28.2   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,ceph-mgr=enabled,ceph-mon=enabled,ceph-osd=enabled,kubernetes.io/arch=amd64,kubernetes.io/hostname=ceph61,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=,node.kubernetes.io/exclude-from-external-load-balancers=
ceph62   Ready    <none>          47h   v1.28.2   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,ceph-mon=enabled,ceph-osd=enabled,kubernetes.io/arch=amd64,kubernetes.io/hostname=ceph62,kubernetes.io/os=linux
ceph63   Ready    <none>          47h   v1.28.2   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,ceph-mon=enabled,ceph-osd=enabled,kubernetes.io/arch=amd64,kubernetes.io/hostname=ceph63,kubernetes.io/os=linux

2.4 部署operator

# 第1步：修改去掉注释 vim /root/rook-1.13.3/deploy/examples/operator.yaml
ROOK_CSI_CEPH_IMAGE: "quay.io/cephcsi/cephcsi:v3.10.1"
ROOK_CSI_REGISTRAR_IMAGE: "registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.9.1"
ROOK_CSI_RESIZER_IMAGE: "registry.k8s.io/sig-storage/csi-resizer:v1.9.2"
ROOK_CSI_PROVISIONER_IMAGE: "registry.k8s.io/sig-storage/csi-provisioner:v3.6.3"
ROOK_CSI_SNAPSHOTTER_IMAGE: "registry.k8s.io/sig-storage/csi-snapshotter:v6.3.2"
ROOK_CSI_ATTACHER_IMAGE: "registry.k8s.io/sig-storage/csi-attacher:v4.4.2"
# 修改为：
ROOK_CSI_CEPH_IMAGE: "quay.io/cephcsi/cephcsi:v3.10.1"
ROOK_CSI_REGISTRAR_IMAGE: "registry.cn-hangzhou.aliyuncs.com/google_containers/csi-node-driver-registrar:v2.9.1"
ROOK_CSI_RESIZER_IMAGE: "registry.cn-hangzhou.aliyuncs.com/google_containers/csi-resizer:v1.9.2"
ROOK_CSI_PROVISIONER_IMAGE: "registry.cn-hangzhou.aliyuncs.com/google_containers/csi-provisioner:v3.6.3"
ROOK_CSI_SNAPSHOTTER_IMAGE: "registry.cn-hangzhou.aliyuncs.com/google_containers/csi-snapshotter:v6.3.2"
ROOK_CSI_ATTACHER_IMAGE: "registry.cn-hangzhou.aliyuncs.com/google_containers/csi-attacher:v4.4.2"
# To indicate the image pull policy to be applied to all the containers in the csi driver pods.
ROOK_CSI_IMAGE_PULL_POLICY: "IfNotPresent"# 检查：crds.yaml、common.yaml、operator.yamlcd /root/rook-1.13.3/deploy/examples
kubectl create -f crds.yaml -f common.yaml -f operator.yaml
# kubectl apply -f crds.yaml -f common.yaml -f operator.yaml
# kubectl delete -f crds.yaml -f common.yaml -f operator.yaml[root@ceph61 examples]# kubectl create -f crds.yaml -f common.yaml -f operator.yaml
customresourcedefinition.apiextensions.k8s.io/cephblockpoolradosnamespaces.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephblockpools.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephbucketnotifications.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephbuckettopics.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephclients.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephclusters.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephcosidrivers.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephfilesystemmirrors.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephfilesystems.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephfilesystemsubvolumegroups.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephnfses.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephobjectrealms.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephobjectstores.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephobjectstoreusers.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephobjectzonegroups.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephobjectzones.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephrbdmirrors.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/objectbucketclaims.objectbucket.io created
customresourcedefinition.apiextensions.k8s.io/objectbuckets.objectbucket.io created
namespace/rook-ceph created
clusterrole.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created
clusterrole.rbac.authorization.k8s.io/cephfs-external-provisioner-runner created
clusterrole.rbac.authorization.k8s.io/objectstorage-provisioner-role created
clusterrole.rbac.authorization.k8s.io/rbd-csi-nodeplugin created
clusterrole.rbac.authorization.k8s.io/rbd-external-provisioner-runner created
clusterrole.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created
clusterrole.rbac.authorization.k8s.io/rook-ceph-global created
clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created
clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-system created
clusterrole.rbac.authorization.k8s.io/rook-ceph-object-bucket created
clusterrole.rbac.authorization.k8s.io/rook-ceph-osd created
clusterrole.rbac.authorization.k8s.io/rook-ceph-system created
clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-nodeplugin-role created
clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role created
clusterrolebinding.rbac.authorization.k8s.io/objectstorage-provisioner-role-binding created
clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin created
clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role created
clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-global created
clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created
clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-object-bucket created
clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-osd created
clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system created
role.rbac.authorization.k8s.io/cephfs-external-provisioner-cfg created
role.rbac.authorization.k8s.io/rbd-csi-nodeplugin created
role.rbac.authorization.k8s.io/rbd-external-provisioner-cfg created
role.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created
role.rbac.authorization.k8s.io/rook-ceph-mgr created
role.rbac.authorization.k8s.io/rook-ceph-osd created
role.rbac.authorization.k8s.io/rook-ceph-purge-osd created
role.rbac.authorization.k8s.io/rook-ceph-rgw created
role.rbac.authorization.k8s.io/rook-ceph-system created
rolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role-cfg created
rolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin-role-cfg created
rolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role-cfg created
rolebinding.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created
rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created
rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr created
rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-system created
rolebinding.rbac.authorization.k8s.io/rook-ceph-osd created
rolebinding.rbac.authorization.k8s.io/rook-ceph-purge-osd created
rolebinding.rbac.authorization.k8s.io/rook-ceph-rgw created
rolebinding.rbac.authorization.k8s.io/rook-ceph-system created
serviceaccount/objectstorage-provisioner created
serviceaccount/rook-ceph-cmd-reporter created
serviceaccount/rook-ceph-mgr created
serviceaccount/rook-ceph-osd created
serviceaccount/rook-ceph-purge-osd created
serviceaccount/rook-ceph-rgw created
serviceaccount/rook-ceph-system created
serviceaccount/rook-csi-cephfs-plugin-sa created
serviceaccount/rook-csi-cephfs-provisioner-sa created
serviceaccount/rook-csi-rbd-plugin-sa created
serviceaccount/rook-csi-rbd-provisioner-sa created
configmap/rook-ceph-operator-config created
deployment.apps/rook-ceph-operator created# 在继续操作之前，验证 rook-ceph-operator 是否正常运行，时间稍微长一些
kubectl get deployment -n rook-ceph
kubectl get pod -n rook-ceph[root@ceph61 examples]# kubectl get deployment -n rook-ceph
NAME                 READY   UP-TO-DATE   AVAILABLE   AGE
rook-ceph-operator   1/1     1            1           4m52s
[root@ceph61 examples]# kubectl get pod -n rook-ceph
NAME                                  READY   STATUS    RESTARTS   AGE
rook-ceph-operator-5bfd9fd4fb-gg2g2   1/1     Running   0          4m56s

2.5 部署Ceph集群

# apply可以重复执行，create不能重复执行
kubectl create -f cluster.yaml
# kubectl apply -f cluster.yaml# 删除资源对象
kubectl delete -f cluster.yaml# 强制删除命名空间
kubectl delete ns rook-ceph --grace-period=0 --force
kubectl delete namespace rook-ceph --force --grace-period=0
# 如果强制删除，还是无法成功的话，执行如下命令：
yum install jq -y
NAMESPACE=rook-ceph
kubectl proxy & kubectl get namespace $NAMESPACE -o json |jq '.spec = {"finalizers":[]}' >temp.json
curl -k -H "Content-Type: application/json" -X PUT --data-binary @temp.json 127.0.0.1:8001/api/v1/namespaces/$NAMESPACE/finalize# 指定Pod某个容器执行命令
kubectl exec <pod-name> -c <container-name> date# 查看日志
kubectl logs <pod-name>
kubectl logs -f <pod-name> # 实时查看日志
kubectl logs -f <pod-name>  -c <container-name># 查看所有资源
kubectl get all -A[root@ceph61 examples]# kubectl apply -f cluster.yaml
cephcluster.ceph.rook.io/rook-ceph created# 实时查看进度，确保所有Pod处于Running，OSD prepare POD处于Completed，时间稍微长一些
kubectl get pod -n rook-ceph -w
# 实时查看集群创建进度
kubectl get cephcluster -n rook-ceph rook-ceph -w
# 详细描述
kubectl describe cephcluster -n rook-ceph rook-ceph[root@ceph61 examples]# kubectl get pod -n rook-ceph
NAME                                               READY   STATUS      RESTARTS      AGE
csi-cephfsplugin-6jcsf                             2/2     Running     1 (14m ago)   25m
csi-cephfsplugin-8pj67                             2/2     Running     1 (24m ago)   25m
csi-cephfsplugin-lgxb8                             2/2     Running     0             25m
csi-cephfsplugin-provisioner-7b94748884-6st2v      5/5     Running     1 (14m ago)   25m
csi-cephfsplugin-provisioner-7b94748884-lbb5q      5/5     Running     0             25m
csi-rbdplugin-7h4fd                                2/2     Running     1 (24m ago)   25m
csi-rbdplugin-dflzl                                2/2     Running     1 (14m ago)   25m
csi-rbdplugin-jz9fq                                2/2     Running     0             25m
csi-rbdplugin-provisioner-5d78f964b7-h4tbd         5/5     Running     3 (13m ago)   25m
csi-rbdplugin-provisioner-5d78f964b7-xxqzp         5/5     Running     1 (14m ago)   25m
rook-ceph-crashcollector-ceph61-5cbdcd57f4-7xwgv   1/1     Running     0             2m10s
rook-ceph-crashcollector-ceph62-745dc4c977-vdctc   1/1     Running     0             2m22s
rook-ceph-crashcollector-ceph63-b8f9b54d6-j2784    1/1     Running     0             2m23s
rook-ceph-exporter-ceph61-74c796f9b4-2g8bv         1/1     Running     0             2m10s
rook-ceph-exporter-ceph62-687b86d6f5-dv7x6         1/1     Running     0             2m22s
rook-ceph-exporter-ceph63-5d86c955fb-nc2q4         1/1     Running     0             2m23s
rook-ceph-mgr-a-5d5ccb978d-ss77f                   3/3     Running     0             2m23s
rook-ceph-mgr-b-5f5fcfcb5f-slzv5                   3/3     Running     0             2m22s
rook-ceph-mon-a-57dd87b4f-bnm5t                    2/2     Running     0             14m
rook-ceph-mon-b-6bdf4c8d65-r7ncw                   2/2     Running     0             14m
rook-ceph-mon-c-6ffb957cb7-77fcb                   2/2     Running     0             9m10s
rook-ceph-operator-5bfd9fd4fb-k2jr4                1/1     Running     0             30m
rook-ceph-osd-prepare-ceph61-96lgt                 0/1     Completed   0             2m
rook-ceph-osd-prepare-ceph62-xbzmm                 0/1     Completed   0             2m
rook-ceph-osd-prepare-ceph63-hj5m7                 0/1     Completed   0             119s

2.6 强制删除命名空间

# 强制删除命名空间
[root@ceph61 examples]# kubectget namespace
NAME              STATUS        AGE
cattle-system     Active        40h
default           Active        42h
kube-node-lease   Active        42h
kube-public       Active        42h
kube-system       Active        42h
kuboard           Active        41h
rook-ceph         Terminating   23h# rook-ceph命名空间 Terminating
kubectl delete ns rook-ceph --grace-period=0 --force
kubectl delete namespace rook-ceph --force --grace-period=0# 如果强制删除，还是无法成功的话，执行如下命令：
yum install jq -y
NAMESPACE=rook-ceph
kubectl proxy & kubectl get namespace $NAMESPACE -o json |jq '.spec = {"finalizers":[]}' >temp.json
curl -k -H "Content-Type: application/json" -X PUT --data-binary @temp.json 127.0.0.1:8001/api/v1/namespaces/$NAMESPACE/finalize[root@ceph61 ~]# NAMESPACE=rook-ceph
[root@ceph61 ~]# kubectl proxy & kubectl get namespace $NAMESPACE -o json |jq '.spec = {"finalizers":[]}' >temp.json
[1] 29753
Starting to serve on 127.0.0.1:8001
[root@ceph61 ~]# curl -k -H "Content-Type: application/json" -X PUT --data-binary @temp.json 127.0.0.1:8001/api/v1/namespaces/$NAMESPACE/finalize
{"kind": "Namespace","apiVersion": "v1","metadata": {"name": "rook-ceph","uid": "92a3a816-1464-4ecd-8d1c-c13cc1f8a345","resourceVersion": "70192","creationTimestamp": "2024-02-06T02:51:20Z","deletionTimestamp": "2024-02-06T11:03:41Z","labels": {"kubernetes.io/metadata.name": "rook-ceph"},"managedFields": [{"manager": "kubectl-create","operation": "Update","apiVersion": "v1","time": "2024-02-06T02:51:20Z","fieldsType": "FieldsV1","fieldsV1": {"f:metadata": {"f:labels": {".": {},"f:kubernetes.io/metadata.name": {}}}}},{"manager": "kube-controller-manager","operation": "Update","apiVersion": "v1","time": "2024-02-06T11:04:08Z","fieldsType": "FieldsV1","fieldsV1": {"f:status": {"f:conditions": {".": {},"k:{\"type\":\"NamespaceContentRemaining\"}": {".": {},"f:lastTransitionTime": {},"f:message": {},"f:reason": {},"f:status": {},"f:type": {}},"k:{\"type\":\"NamespaceDeletionContentFailure\"}": {".": {},"f:lastTransitionTime": {},"f:message": {},"f:reason": {},"f:status": {},"f:type": {}},"k:{\"type\":\"NamespaceDeletionDiscoveryFailure\"}": {".": {},"f:lastTransitionTime": {},"f:message": {},"f:reason": {},"f:status": {},"f:type": {}},"k:{\"type\":\"NamespaceDeletionGroupVersionParsingFailure\"}": {".": {},"f:lastTransitionTime": {},"f:message": {},"f:reason": {},"f:status": {},"f:type": {}},"k:{\"type\":\"NamespaceFinalizersRemaining\"}": {".": {},"f:lastTransitionTime": {},"f:message": {},"f:reason": {},"f:status": {},"f:type": {}}}}},"subresource": "status"}]},"spec": {},"status": {"phase": "Terminating","conditions": [{"type": "NamespaceDeletionDiscoveryFailure","status": "True","lastTransitionTime": "2024-02-06T11:03:46Z","reason": "DiscoveryFailed","message": "Discovery failed for some groups, 1 failing: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: stale GroupVersion discovery: metrics.k8s.io/v1beta1"},{"type": "NamespaceDeletionGroupVersionParsingFailure","status": "False","lastTransitionTime": "2024-02-06T11:03:46Z","reason": "ParsedGroupVersions","message": "All legacy kube types successfully parsed"},{"type": "NamespaceDeletionContentFailure","status": "False","lastTransitionTime": "2024-02-06T11:03:46Z","reason": "ContentDeleted","message": "All content successfully deleted, may be waiting on finalization"},{"type": "NamespaceContentRemaining","status": "False","lastTransitionTime": "2024-02-06T11:04:08Z","reason": "ContentRemoved","message": "All content successfully removed"},{"type": "NamespaceFinalizersRemaining","status": "False","lastTransitionTime": "2024-02-06T11:03:46Z","reason": "ContentHasNoFinalizers","message": "All content-preserving finalizers finished"}]}
}# 查看
[root@ceph61 ~]# kubectl get namespace
NAME              STATUS   AGE
cattle-system     Active   40h
default           Active   42h
kube-node-lease   Active   42h
kube-public       Active   42h
kube-system       Active   42h
kuboard           Active   41h

2.7 验证集群

# 要验证群集是否处于正常状态，可以连接到 toolbox 工具箱并运行命令
cd /root/rook-1.13.3/deploy/examples
kubectl create -f toolbox.yaml
[root@ceph61 examples]# kubectl create -f toolbox.yaml
deployment.apps/rook-ceph-tools created# 查看Pod
[root@ceph61 examples]# kubectl get pod -n rook-ceph
NAME                                               READY   STATUS      RESTARTS      AGE
csi-cephfsplugin-6jcsf                             2/2     Running     1 (21m ago)   32m
csi-cephfsplugin-8pj67                             2/2     Running     1 (32m ago)   32m
csi-cephfsplugin-lgxb8                             2/2     Running     0             32m
csi-cephfsplugin-provisioner-7b94748884-6st2v      5/5     Running     1 (21m ago)   32m
csi-cephfsplugin-provisioner-7b94748884-lbb5q      5/5     Running     0             32m
csi-rbdplugin-7h4fd                                2/2     Running     1 (32m ago)   32m
csi-rbdplugin-dflzl                                2/2     Running     1 (21m ago)   32m
csi-rbdplugin-jz9fq                                2/2     Running     0             32m
csi-rbdplugin-provisioner-5d78f964b7-h4tbd         5/5     Running     3 (20m ago)   32m
csi-rbdplugin-provisioner-5d78f964b7-xxqzp         5/5     Running     1 (21m ago)   32m
rook-ceph-crashcollector-ceph61-5cbdcd57f4-7xwgv   1/1     Running     0             9m40s
rook-ceph-crashcollector-ceph62-6cbdf99cff-vrw44   1/1     Running     0             7m15s
rook-ceph-crashcollector-ceph63-855974fbd8-8w5ql   1/1     Running     0             7m16s
rook-ceph-exporter-ceph61-74c796f9b4-2g8bv         1/1     Running     0             9m40s
rook-ceph-exporter-ceph62-7f845dcd9c-fjr87         1/1     Running     0             7m11s
rook-ceph-exporter-ceph63-5dcf677d86-ct776         1/1     Running     0             7m11s
rook-ceph-mgr-a-5d5ccb978d-ss77f                   3/3     Running     0             9m53s
rook-ceph-mgr-b-5f5fcfcb5f-slzv5                   3/3     Running     0             9m52s
rook-ceph-mon-a-57dd87b4f-bnm5t                    2/2     Running     0             22m
rook-ceph-mon-b-6bdf4c8d65-r7ncw                   2/2     Running     0             21m
rook-ceph-mon-c-6ffb957cb7-77fcb                   2/2     Running     0             16m
rook-ceph-operator-5bfd9fd4fb-k2jr4                1/1     Running     0             38m
rook-ceph-osd-0-7cbd59747f-5sl7v                   2/2     Running     0             7m16s
rook-ceph-osd-1-f487664fb-gdwlx                    2/2     Running     0             7m16s
rook-ceph-osd-2-77b849c5b-lc7p2                    2/2     Running     0             7m16s
rook-ceph-osd-3-5469748766-tj256                   2/2     Running     0             7m16s
rook-ceph-osd-4-77d4bd4b7c-2gqdz                   2/2     Running     0             7m16s
rook-ceph-osd-5-694548dbcc-jr9bg                   2/2     Running     0             7m16s
rook-ceph-osd-6-844ff54957-d84xs                   2/2     Running     0             7m16s
rook-ceph-osd-7-5c7b9d4b76-fsh46                   2/2     Running     0             7m16s
rook-ceph-osd-8-6dd768d7fb-9smbs                   2/2     Running     0             7m16s
rook-ceph-osd-prepare-ceph61-czhcp                 0/1     Completed   0             2m53s
rook-ceph-osd-prepare-ceph62-2754w                 0/1     Completed   0             2m49s
rook-ceph-osd-prepare-ceph63-fhqfl                 0/1     Completed   0             2m45s
rook-ceph-tools-66b77b8df5-gj256                   1/1     Running     0             3m26s# 进入Pod容器
kubectl exec -it rook-ceph-tools-66b77b8df5-gj256 -n rook-ceph -- bash# 进去之后就可以执行各种命令了：
ceph status
ceph osd status
ceph df
rados df[root@ceph61 examples]# kubectl exec -it rook-ceph-tools-66b77b8df5-gj256 -n rook-ceph -- bash
bash-4.4$ ceph -v
ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)
bash-4.4$ ceph -scluster:id:     7793dc7d-fb3a-48dd-9297-8f2d37fb9901health: HEALTH_WARNSlow OSD heartbeats on back (longest 69426.357ms)Slow OSD heartbeats on front (longest 65492.636ms)services:mon: 3 daemons, quorum a,c,b (age 18s)mgr: b(active, since 9m), standbys: aosd: 9 osds: 9 up (since 76s), 9 in (since 15m)data:pools:   1 pools, 1 pgsobjects: 2 objects, 449 KiBusage:   240 MiB used, 270 GiB / 270 GiB availpgs:     1 active+cleanbash-4.4$ ceph osd status
ID  HOST     USED  AVAIL  WR OPS  WR DATA  RD OPS  RD DATA  STATE0  ceph62  26.5M  29.9G      0        0       0        0   exists,up1  ceph63  26.9M  29.9G      0        0       0        0   exists,up2  ceph62  26.9M  29.9G      0        0       0        0   exists,up3  ceph63  26.5M  29.9G      0        0       0        0   exists,up4  ceph62  26.5M  29.9G      0        0       0        0   exists,up5  ceph63  26.5M  29.9G      0        0       0        0   exists,up6  ceph61  26.5M  29.9G      0        0       0        0   exists,up7  ceph61  26.9M  29.9G      0        0       0        0   exists,up8  ceph61  26.5M  29.9G      0        0       0        0   exists,up
bash-4.4$ ceph df
--- RAW STORAGE ---
CLASS     SIZE    AVAIL     USED  RAW USED  %RAW USED
hdd    270 GiB  270 GiB  240 MiB   240 MiB       0.09
TOTAL  270 GiB  270 GiB  240 MiB   240 MiB       0.09--- POOLS ---
POOL  ID  PGS   STORED  OBJECTS     USED  %USED  MAX AVAIL
.mgr   1    1  449 KiB        2  1.3 MiB      0     85 GiB
bash-4.4$ rados df
POOL_NAME     USED  OBJECTS  CLONES  COPIES  MISSING_ON_PRIMARY  UNFOUND  DEGRADED  RD_OPS       RD  WR_OPS       WR  USED COMPR  UNDER COMPR
.mgr       1.3 MiB        2       0       6                   0        0         0     288  494 KiB     153  1.3 MiB         0 B          0 Btotal_objects    2
total_used       240 MiB
total_avail      270 GiB
total_space      270 GiB

3.Ceph界面

# Dashboard，查看dashboard的svc，Rook将启用端口 8443 以进行 https 访问
kubectl get svc -n rook-ceph
[root@ceph61 examples]# kubectl get svc -n rook-ceph
NAME                      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
rook-ceph-exporter        ClusterIP   10.104.154.115   <none>        9926/TCP            22m
rook-ceph-mgr             ClusterIP   10.109.137.46    <none>        9283/TCP            22m
rook-ceph-mgr-dashboard   ClusterIP   10.101.248.83    <none>        8443/TCP            22m
rook-ceph-mon-a           ClusterIP   10.100.67.82     <none>        6789/TCP,3300/TCP   35m
rook-ceph-mon-b           ClusterIP   10.97.62.97      <none>        6789/TCP,3300/TCP   34m
rook-ceph-mon-c           ClusterIP   10.99.101.215    <none>        6789/TCP,3300/TCP   29m# rook-ceph-mgr：是 Ceph 的管理进程（Manager），负责集群的监控、状态报告、数据分析、调度等功能，它默认监听 9283 端口，并提供了 Prometheus 格式的监控指标，可以被 Prometheus 拉取并用于集群监控。
# rook-ceph-mgr-dashboard：是 Rook 提供的一个 Web 界面，用于方便地查看 Ceph 集群的监控信息、状态、性能指标等。
# rook-ceph-mon：是 Ceph Monitor 进程的 Kubernetes 服务。Ceph Monitor 是 Ceph 集群的核心组件之一，负责维护 Ceph 集群的状态、拓扑结构、数据分布等信息，是 Ceph 集群的管理节点。# 查看默认账号admin的密码，这个密码就是等会用来登录界面的
# kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath="{['data']['password']}" | base64 --decode && echo
[root@ceph61 examples]# kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath="{['data']['password']}" | base64 --decode && echo
2<rI.]qNXqGX#F2c-M@E# 使用 NodePort 类型的Service暴露dashboard：dashboard-external-https.yaml
[root@ceph61 examples]# cat dashboard-external-https.yaml
apiVersion: v1
kind: Service
metadata:name: rook-ceph-mgr-dashboard-external-httpsnamespace: rook-ceph # namespace:clusterlabels:app: rook-ceph-mgrrook_cluster: rook-ceph # namespace:cluster
spec:ports:- name: dashboardport: 8443protocol: TCPtargetPort: 8443selector:app: rook-ceph-mgrmgr_role: activerook_cluster: rook-ceph # namespace:clustersessionAffinity: Nonetype: NodePort# 创建
kubectl create -f dashboard-external-https.yaml
[root@ceph61 examples]# kubectl create -f dashboard-external-https.yaml
service/rook-ceph-mgr-dashboard-external-https created[root@ceph61 examples]# kubectl get svc -n rook-ceph
NAME                                     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
rook-ceph-exporter                       ClusterIP   10.104.154.115   <none>        9926/TCP            28m
rook-ceph-mgr                            ClusterIP   10.109.137.46    <none>        9283/TCP            28m
rook-ceph-mgr-dashboard                  ClusterIP   10.101.248.83    <none>        8443/TCP            28m
rook-ceph-mgr-dashboard-external-https   NodePort    10.97.245.134    <none>        8443:31222/TCP      20s
rook-ceph-mon-a                          ClusterIP   10.100.67.82     <none>        6789/TCP,3300/TCP   40m
rook-ceph-mon-b                          ClusterIP   10.97.62.97      <none>        6789/TCP,3300/TCP   40m
rook-ceph-mon-c                          ClusterIP   10.99.101.215    <none>        6789/TCP,3300/TCP   35m# 访问
https://192.168.120.61:31222/
用户名：admin
密码：2<rI.]qNXqGX#F2c-M@E

在这里插入图片描述

K8s环境下rook-v1.13.3部署Ceph-v18.2.1集群

文章目录 1.K8s环境搭建2.Ceph集群部署2.1 部署Rook Operator2.2 镜像准备2.3 配置节点角色2.4 部署operator2.5 部署Ceph集群2.6 强制删除命名空间2.7 验证集群 3.Ceph界面 1.K8s环境搭建参考：CentOS7搭建k8s-v1.28.6集群详情，把K8s集群完成搭建&…...

编程日记 2024/2/12 3:53:48

【JavaEE】传输层网络协议

传输层网络协议 1. UDP协议 1.1 特点面向数据报（DatagramSocket）数据报大小限制为64k全双工不可靠传输有接收缓冲区，无发送缓冲区 UDP的特点，我理解起来就是工人组成的**“人工传送带”**： 面向数据报（…...

编程日记 2024/2/12 3:47:43

08-Java过滤器模式 ( Filter Pattern )

Java过滤器模式实现范例过滤器模式（Filter Pattern）或允许开发人员使用不同的标准来过滤一组对象，通过逻辑运算以解耦的方式把它们连接起来过滤器模式（Filter Pattern） 又称标准模式（Criteria Pattern…...

编程日记 2024/2/12 3:46:41

ChatGPT高效提问—prompt常见用法（续篇八）

ChatGPT高效提问—prompt常见用法（续篇八） 1.1 对抗对抗是一个重要主题，深入探讨了大型语言模型（LLM）的安全风险。它不仅反映了人们对LLM可能出现的风险和安全问题的理解，而且能够帮助我们识别这些潜在的风险，并通过切实可行的技术手段来规避。截至目前，网络…...

编程日记 2024/2/12 3:42:34

微软.NET6开发的C#特性——接口和属性

我是荔园微风，作为一名在IT界整整25年的老兵，看到不少初学者在学习编程语言的过程中如此的痛苦，我决定做点什么，下面我就重点讲讲微软.NET6开发人员需要知道的C#特性，然后比较其他各种语言进行认识。 C#经历了多年发展…...

编程日记 2024/2/12 3:40:32

容器基础知识：容器和虚拟化的区别

虚拟化与容器化对比容器化和虚拟化都是用于优化资源利用率并实现高效应用程序部署的技术。然而，它们在方法和关键特征上存在差异： 虚拟化: 可以理解为创建虚拟机 (VM)。虚拟机模拟一台拥有自己硬件（CPU、内存、存储）和操作系统…...

编程日记 2024/2/12 3:38:31

【Linux】vim的基本操作与配置（下）

Hello everybody!今天我们继续讲解vim的操作与配置，希望大家在看过这篇文章与上篇文章后都能够轻松上手vim! 1.补充在上一篇文章中我们说过了，在底行模式下set nu可以显示行号。今天补充一条：set nonu可以取消行号。这两条命令大家看看就可…...

编程日记 2024/2/12 3:35:28

[office] 图文演示excel怎样给单元格添加下拉列表 #知识分享#经验分享

图文演示excel怎样给单元格添加下拉列表在Excel表格中输入数据的时候，为了简便快捷的输入，经常需要给Excel单元格添加一个下拉菜单，这样在输入数据时不必按键盘，只是用鼠标选择选项就可以了。比的位置。 4、可以看到一个预览的…...

编程日记 2024/2/12 3:34:26

【RT-DETR有效改进】利用SENetV2重构化网络结构 (ILSVRC冠军得主，全网独家首发)

👑欢迎大家订阅本专栏，一起学习RT-DETR👑 一、本文介绍本文给大家带来的改进机制是SENetV2，其是2023.11月的最新机制(所以大家想要发论文的可以在上面下点功夫)，其是一种通过调整卷积网络中的通道关系来提升性能的网络结构。SENet并不是一个独立的网络模型，而…...

编程日记 2024/2/12 3:33:25

【Linux】学习-进程间通信

进程间通信介绍进程间通信的本质进程间通信的前提，首先需要让不同的进程看到同一块“内存”此“内存”一定不属于任何进程，而应该强调共享二字进程间通信的目的数据传输：一个进程需要将它的数据发送给另一个进程资源共享：…...

编程日记 2024/2/12 3:32:24

45. C++ 字符指针和字符串

一、字符指针可以指向一个字符串。我们可以用字符串常量对字符指针进行初始化。例如，有说明语句： char *str "This is a string.";是对字符指针进行初始化。此时，字符指针指向的是一个字符串常量的首地址，即指向字符…...

编程日记 2024/2/12 3:31:24

如何制作一款3D FPS游戏

制作一款3D FPS游戏是一个复杂的过程，需要涵盖多个方面，包括游戏设计、游戏引擎选择、模型制作、音效制作、关卡设计等。下面是一个关于如何制作一款3D FPS游戏的超长文章。游戏设计首先，你需要确定游戏的整体设计和核心玩法。这包括游戏的…...

编程日记 2024/2/12 3:29:21

人工智能|深度学习——使用多层级注意力机制和keras实现问题分类

代码下载使用多层级注意力机制和keras实现问题分类资源-CSDN文库 1 准备工作 1.1 什么是词向量? ”词向量”（词嵌入）是将一类将词的语义映射到向量空间中去的自然语言处理技术。即将一个词用特定的向量来表示，向量之间的距离（例…...

编程日记 2024/2/12 3:25:17

C语言常见面试题：C语言中如何进行网页开发编程？

在C语言中进行网页开发通常不是一个直接的过程，因为C语言主要用于系统级编程，而不是Web开发。传统的Web开发主要使用高级语言如JavaScript、Python、Ruby、PHP等，以及与Web相关的技术，如HTML、CSS和数据库。然而，如果…...

编程日记 2024/2/12 3:21:13

DevOps落地笔记-20|软件质量：决定系统成功的关键

上一课时介绍通过提高工程效率来提高价值交付效率，从而提高企业对市场的响应速度。在提高响应速度的同时，也不能降低软件的质量，这就是所谓的“保质保量”。具备高质量软件，高效率的企业走得更快更远。相反，低劣的软件…...

编程日记 2024/2/12 3:20:12

政安晨：梯度与导数～示例演绎《机器学习·神经网络》的高阶理解

这篇文章确实需要一定的数学基础，第一次接触的小伙伴可以先看一下我示例演绎这个主题的前两篇文章： 示例演绎机器学习中（深度学习）神经网络的数学基础——快速理解核心概念（一）： 政安晨&#…...

编程日记 2024/2/12 3:19:11

CTFSHOW命令执行web入门29-54

description: >- 这里就记录一下ctfshow的刷题记录是web入门的命令执行专题里面的题目，他是有分类，并且覆盖也很广泛，所以就通过刷这个来，不过里面有一些脚本的题目发现我自己根本不会笑死。如果还不怎么知道写题的话，可以去看我的gitbook，当然csdn我也转载了我自己的…...

编程日记 2024/2/12 3:15:06

探索ChatGPT4：新一代人工智能语言模型的突破

ChatGPT4，作为最新一代的语言处理模型，代表了人工智能在自然语言理解和生成方面的最新突破。本文将深入介绍ChatGPT4的新特性，探讨其在各个领域的潜在应用。 ChatGPT4概述在继承了前一代模型的强大基础之上，ChatGPT4引入了多项…...

编程日记 2024/2/12 3:14:05

PVST详解

PVST（Per-VLAN Spanning Tree）是Cisco公司的一种扩展的Spanning Tree协议，允许在每个VLAN中独立运行一个Spanning Tree实例，从而提高网络的可靠性和性能。 PVST协议在每个交换机中维护多个Spanning Tree实例，每个实例…...

编程日记 2024/2/12 3:10:00

c++ 子进程交互逻辑

目录一、主进程逻辑 1、创建子进程时候，写入自己的HWND 2、响应子进程消息...

编程日记 2024/2/12 2:58:49

在软件开发中正确使用MySQL日期时间类型的深度解析

在日常软件开发场景中，时间信息的存储是底层且核心的需求。从金融交易的精确记账时间、用户操作的行为日志，到供应链系统的物流节点时间戳，时间数据的准确性直接决定业务逻辑的可靠性。MySQL作为主流关系型数据库，其日期时间类型的…...

编程新知 2025/10/16 13:22:06

synchronized 学习

学习源： https://www.bilibili.com/video/BV1aJ411V763?spm_id_from333.788.videopod.episodes&vd_source32e1c41a9370911ab06d12fbc36c4ebc 1.应用场景不超卖，也要考虑性能问题（场景） 2.常见面试问题： sync出…...

编程新知 2025/12/14 17:38:17

shell脚本--常见案例

1、自动备份文件或目录 2、批量重命名文件 3、查找并删除指定名称的文件： 4、批量删除文件 5、查找并替换文件内容 6、批量创建文件 7、创建文件夹并移动文件 8、在文件夹中查找文件...

编程新知 2026/1/21 19:11:31

visual studio 2022更改主题为深色

visual studio 2022更改主题为深色点击visual studio 上方的工具-> 选项在选项窗口中，选择环境 -> 常规 ，将其中的颜色主题改成深色点击确定，更改完成...

编程新知 2026/1/25 11:13:37

全球首个30米分辨率湿地数据集(2000—2022)

数据简介今天我们分享的数据是全球30米分辨率湿地数据集，包含8种湿地亚类，该数据以0.5X0.5的瓦片存储，我们整理了所有属于中国的瓦片名称与其对应省份，方便大家研究使用。该数据集作为全球首个30米分辨率、覆盖2000–2022年时间…...

编程新知 2026/1/31 17:15:25

【AI学习】三、AI算法中的向量

在人工智能（AI）算法中，向量（Vector）是一种将现实世界中的数据（如图像、文本、音频等）转化为计算机可处理的数值型特征表示的工具。它是连接人类认知（如语义、视觉特征）与…...

编程新知 2026/1/1 14:01:45

现代密码学 | 椭圆曲线密码学—附py代码

Elliptic Curve Cryptography 椭圆曲线密码学（ECC）是一种基于有限域上椭圆曲线数学特性的公钥加密技术。其核心原理涉及椭圆曲线的代数性质、离散对数问题以及有限域上的运算。椭圆曲线密码学是多种数字签名算法的基础，例如椭圆曲线数字签…...

编程新知 2026/2/6 7:30:33

vue3+vite项目中使用.env文件环境变量方法

vue3vite项目中使用.env文件环境变量方法 .env文件作用命名规则常用的配置项示例使用方法注意事项在vite.config.js文件中读取环境变量方法 .env文件作用 .env 文件用于定义环境变量，这些变量可以在项目中通过 import.meta.env 进行访问。Vite 会自动加载这些环境变…...

编程新知 2026/1/26 19:12:01