K8s环境下rook-v1.13.3部署Ceph-v18.2.1集群
文章目录
- 1.K8s环境搭建
- 2.Ceph集群部署
- 2.1 部署Rook Operator
- 2.2 镜像准备
- 2.3 配置节点角色
- 2.4 部署operator
- 2.5 部署Ceph集群
- 2.6 强制删除命名空间
- 2.7 验证集群
- 3.Ceph界面
1.K8s环境搭建
参考：CentOS7搭建k8s-v1.28.6集群详情，把K8s集群完成搭建，再进行Ceph集群搭建
2.Ceph集群部署
2.1 部署Rook Operator
# 下载 rook 项目，相关源码（2024-02-06最新版本）
git clone --single-branch --branch v1.13.3 https://github.com/rook/rook.git
# 如果网络不好，可以下载：rook-1.13.3.tar.gz
# https://gh.api.99988866.xyz/https://github.com/rook/rook/archive/refs/tags/v1.13.3.tar.gz
# 解压：tar -zxvf rook-1.13.3.tar.gz[root@ceph61 ~]# git clone --single-branch --branch v1.13.3 https://github.com/rook/rook.git
Cloning into 'rook'...
remote: Enumerating objects: 95134, done.
remote: Counting objects: 100% (5514/5514), done.
remote: Compressing objects: 100% (348/348), done.
remote: Total 95134 (delta 5299), reused 5203 (delta 5166), pack-reused 89620
Receiving objects: 100% (95134/95134), 50.70 MiB | 10.43 MiB/s, done.
Resolving deltas: 100% (66720/66720), done.
Note: checking out '54663a4333bb72ba2114f387048775a619f2344d'.You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:git checkout -b new_branch_name2.2 镜像准备
# 第一步：检查需要下载的镜像
[root@ceph61 examples]# cat images.txtgcr.io/k8s-staging-sig-storage/objectstorage-sidecar/objectstorage-sidecar:v20230130-v0.1.0-24-gc0cf995quay.io/ceph/ceph:v18.2.1quay.io/ceph/cosi:v0.1.1quay.io/cephcsi/cephcsi:v3.10.1quay.io/csiaddons/k8s-sidecar:v0.8.0registry.k8s.io/sig-storage/csi-attacher:v4.4.2registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.9.1registry.k8s.io/sig-storage/csi-provisioner:v3.6.3registry.k8s.io/sig-storage/csi-resizer:v1.9.2registry.k8s.io/sig-storage/csi-snapshotter:v6.3.2rook/ceph:v1.13.3
# 修改为：gcr.io/k8s-staging-sig-storage/objectstorage-sidecar/objectstorage-sidecar:v20230130-v0.1.0-24-gc0cf995quay.io/ceph/ceph:v18.2.1quay.io/ceph/cosi:v0.1.1quay.io/cephcsi/cephcsi:v3.10.1quay.io/csiaddons/k8s-sidecar:v0.8.0registry.cn-hangzhou.aliyuncs.com/google_containers/csi-attacher:v4.4.2registry.cn-hangzhou.aliyuncs.com/google_containers/csi-node-driver-registrar:v2.9.1registry.cn-hangzhou.aliyuncs.com/google_containers/csi-provisioner:v3.6.3registry.cn-hangzhou.aliyuncs.com/google_containers/csi-resizer:v1.9.2registry.cn-hangzhou.aliyuncs.com/google_containers/csi-snapshotter:v6.3.2rook/ceph:v1.13.3# 第2步：登录阿里镜像源
docker login --username=你的阿里云账号 registry.cn-beijing.aliyuncs.com
输入你的阿里云账号密码# 第2步：手动下载镜像
# 不知道如何下载（此次部署集群，该镜像不下载也没有影响）：gcr.io/k8s-staging-sig-storage/objectstorage-sidecar/objectstorage-sidecar:v20230130-v0.1.0-24-gc0cf995
docker pull quay.io/ceph/ceph:v18.2.1
docker pull quay.io/ceph/cosi:v0.1.1
docker pull quay.io/cephcsi/cephcsi:v3.10.1
docker pull quay.io/csiaddons/k8s-sidecar:v0.8.0
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/csi-attacher:v4.4.2
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/csi-node-driver-registrar:v2.9.1
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/csi-provisioner:v3.6.3
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/csi-resizer:v1.9.2
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/csi-snapshotter:v6.3.2
docker pull rook/ceph:v1.13.3# 第4步：重新打tag
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/csi-attacher:v4.4.2 registry.k8s.io/sig-storage/csi-attacher:v4.4.2
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/csi-node-driver-registrar:v2.9.1 registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.9.1
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/csi-provisioner:v3.6.3 registry.k8s.io/sig-storage/csi-provisioner:v3.6.3
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/csi-resizer:v1.9.2 registry.k8s.io/sig-storage/csi-resizer:v1.9.2
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/csi-snapshotter:v6.3.2 registry.k8s.io/sig-storage/csi-snapshotter:v6.3.22.3 配置节点角色
# 查看节点标签
kubectl get nodes --show-labels
kubectl get nodes <node-name> --show-labels[root@ceph61 examples]# kubectl get nodes --show-labels
NAME STATUS ROLES AGE VERSION LABELS
ceph61 Ready control-plane 47h v1.28.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=ceph61,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=,node.kubernetes.io/exclude-from-external-load-balancers=
ceph62 Ready <none> 47h v1.28.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=ceph62,kubernetes.io/os=linux
ceph63 Ready <none> 47h v1.28.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=ceph63,kubernetes.io/os=linux# 设置ceph61、ceph62、ceph63节点标签
kubectl label nodes {ceph61,ceph62,ceph63} ceph-osd=enabled
kubectl label nodes {ceph61,ceph62,ceph63} ceph-mon=enabled
kubectl label nodes ceph61 ceph-mgr=enabled[root@ceph61 examples]# kubectl get nodes --show-labels
NAME STATUS ROLES AGE VERSION LABELS
ceph61 Ready control-plane 47h v1.28.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,ceph-mgr=enabled,ceph-mon=enabled,ceph-osd=enabled,kubernetes.io/arch=amd64,kubernetes.io/hostname=ceph61,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=,node.kubernetes.io/exclude-from-external-load-balancers=
ceph62 Ready <none> 47h v1.28.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,ceph-mon=enabled,ceph-osd=enabled,kubernetes.io/arch=amd64,kubernetes.io/hostname=ceph62,kubernetes.io/os=linux
ceph63 Ready <none> 47h v1.28.2 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,ceph-mon=enabled,ceph-osd=enabled,kubernetes.io/arch=amd64,kubernetes.io/hostname=ceph63,kubernetes.io/os=linux2.4 部署operator
# 第1步：修改去掉注释 vim /root/rook-1.13.3/deploy/examples/operator.yaml
ROOK_CSI_CEPH_IMAGE: "quay.io/cephcsi/cephcsi:v3.10.1"
ROOK_CSI_REGISTRAR_IMAGE: "registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.9.1"
ROOK_CSI_RESIZER_IMAGE: "registry.k8s.io/sig-storage/csi-resizer:v1.9.2"
ROOK_CSI_PROVISIONER_IMAGE: "registry.k8s.io/sig-storage/csi-provisioner:v3.6.3"
ROOK_CSI_SNAPSHOTTER_IMAGE: "registry.k8s.io/sig-storage/csi-snapshotter:v6.3.2"
ROOK_CSI_ATTACHER_IMAGE: "registry.k8s.io/sig-storage/csi-attacher:v4.4.2"
# 修改为：
ROOK_CSI_CEPH_IMAGE: "quay.io/cephcsi/cephcsi:v3.10.1"
ROOK_CSI_REGISTRAR_IMAGE: "registry.cn-hangzhou.aliyuncs.com/google_containers/csi-node-driver-registrar:v2.9.1"
ROOK_CSI_RESIZER_IMAGE: "registry.cn-hangzhou.aliyuncs.com/google_containers/csi-resizer:v1.9.2"
ROOK_CSI_PROVISIONER_IMAGE: "registry.cn-hangzhou.aliyuncs.com/google_containers/csi-provisioner:v3.6.3"
ROOK_CSI_SNAPSHOTTER_IMAGE: "registry.cn-hangzhou.aliyuncs.com/google_containers/csi-snapshotter:v6.3.2"
ROOK_CSI_ATTACHER_IMAGE: "registry.cn-hangzhou.aliyuncs.com/google_containers/csi-attacher:v4.4.2"
# To indicate the image pull policy to be applied to all the containers in the csi driver pods.
ROOK_CSI_IMAGE_PULL_POLICY: "IfNotPresent"# 检查：crds.yaml、common.yaml、operator.yamlcd /root/rook-1.13.3/deploy/examples
kubectl create -f crds.yaml -f common.yaml -f operator.yaml
# kubectl apply -f crds.yaml -f common.yaml -f operator.yaml
# kubectl delete -f crds.yaml -f common.yaml -f operator.yaml[root@ceph61 examples]# kubectl create -f crds.yaml -f common.yaml -f operator.yaml
customresourcedefinition.apiextensions.k8s.io/cephblockpoolradosnamespaces.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephblockpools.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephbucketnotifications.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephbuckettopics.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephclients.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephclusters.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephcosidrivers.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephfilesystemmirrors.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephfilesystems.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephfilesystemsubvolumegroups.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephnfses.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephobjectrealms.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephobjectstores.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephobjectstoreusers.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephobjectzonegroups.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephobjectzones.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/cephrbdmirrors.ceph.rook.io created
customresourcedefinition.apiextensions.k8s.io/objectbucketclaims.objectbucket.io created
customresourcedefinition.apiextensions.k8s.io/objectbuckets.objectbucket.io created
namespace/rook-ceph created
clusterrole.rbac.authorization.k8s.io/cephfs-csi-nodeplugin created
clusterrole.rbac.authorization.k8s.io/cephfs-external-provisioner-runner created
clusterrole.rbac.authorization.k8s.io/objectstorage-provisioner-role created
clusterrole.rbac.authorization.k8s.io/rbd-csi-nodeplugin created
clusterrole.rbac.authorization.k8s.io/rbd-external-provisioner-runner created
clusterrole.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created
clusterrole.rbac.authorization.k8s.io/rook-ceph-global created
clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created
clusterrole.rbac.authorization.k8s.io/rook-ceph-mgr-system created
clusterrole.rbac.authorization.k8s.io/rook-ceph-object-bucket created
clusterrole.rbac.authorization.k8s.io/rook-ceph-osd created
clusterrole.rbac.authorization.k8s.io/rook-ceph-system created
clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-nodeplugin-role created
clusterrolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role created
clusterrolebinding.rbac.authorization.k8s.io/objectstorage-provisioner-role-binding created
clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin created
clusterrolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role created
clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-global created
clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-cluster created
clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-object-bucket created
clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-osd created
clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system created
role.rbac.authorization.k8s.io/cephfs-external-provisioner-cfg created
role.rbac.authorization.k8s.io/rbd-csi-nodeplugin created
role.rbac.authorization.k8s.io/rbd-external-provisioner-cfg created
role.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created
role.rbac.authorization.k8s.io/rook-ceph-mgr created
role.rbac.authorization.k8s.io/rook-ceph-osd created
role.rbac.authorization.k8s.io/rook-ceph-purge-osd created
role.rbac.authorization.k8s.io/rook-ceph-rgw created
role.rbac.authorization.k8s.io/rook-ceph-system created
rolebinding.rbac.authorization.k8s.io/cephfs-csi-provisioner-role-cfg created
rolebinding.rbac.authorization.k8s.io/rbd-csi-nodeplugin-role-cfg created
rolebinding.rbac.authorization.k8s.io/rbd-csi-provisioner-role-cfg created
rolebinding.rbac.authorization.k8s.io/rook-ceph-cluster-mgmt created
rolebinding.rbac.authorization.k8s.io/rook-ceph-cmd-reporter created
rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr created
rolebinding.rbac.authorization.k8s.io/rook-ceph-mgr-system created
rolebinding.rbac.authorization.k8s.io/rook-ceph-osd created
rolebinding.rbac.authorization.k8s.io/rook-ceph-purge-osd created
rolebinding.rbac.authorization.k8s.io/rook-ceph-rgw created
rolebinding.rbac.authorization.k8s.io/rook-ceph-system created
serviceaccount/objectstorage-provisioner created
serviceaccount/rook-ceph-cmd-reporter created
serviceaccount/rook-ceph-mgr created
serviceaccount/rook-ceph-osd created
serviceaccount/rook-ceph-purge-osd created
serviceaccount/rook-ceph-rgw created
serviceaccount/rook-ceph-system created
serviceaccount/rook-csi-cephfs-plugin-sa created
serviceaccount/rook-csi-cephfs-provisioner-sa created
serviceaccount/rook-csi-rbd-plugin-sa created
serviceaccount/rook-csi-rbd-provisioner-sa created
configmap/rook-ceph-operator-config created
deployment.apps/rook-ceph-operator created# 在继续操作之前，验证 rook-ceph-operator 是否正常运行，时间稍微长一些
kubectl get deployment -n rook-ceph
kubectl get pod -n rook-ceph[root@ceph61 examples]# kubectl get deployment -n rook-ceph
NAME READY UP-TO-DATE AVAILABLE AGE
rook-ceph-operator 1/1 1 1 4m52s
[root@ceph61 examples]# kubectl get pod -n rook-ceph
NAME READY STATUS RESTARTS AGE
rook-ceph-operator-5bfd9fd4fb-gg2g2 1/1 Running 0 4m56s2.5 部署Ceph集群
# apply可以重复执行，create不能重复执行
kubectl create -f cluster.yaml
# kubectl apply -f cluster.yaml# 删除资源对象
kubectl delete -f cluster.yaml# 强制删除命名空间
kubectl delete ns rook-ceph --grace-period=0 --force
kubectl delete namespace rook-ceph --force --grace-period=0
# 如果强制删除，还是无法成功的话，执行如下命令：
yum install jq -y
NAMESPACE=rook-ceph
kubectl proxy & kubectl get namespace $NAMESPACE -o json |jq '.spec = {"finalizers":[]}' >temp.json
curl -k -H "Content-Type: application/json" -X PUT --data-binary @temp.json 127.0.0.1:8001/api/v1/namespaces/$NAMESPACE/finalize# 指定Pod某个容器执行命令
kubectl exec <pod-name> -c <container-name> date# 查看日志
kubectl logs <pod-name>
kubectl logs -f <pod-name> # 实时查看日志
kubectl logs -f <pod-name> -c <container-name># 查看所有资源
kubectl get all -A[root@ceph61 examples]# kubectl apply -f cluster.yaml
cephcluster.ceph.rook.io/rook-ceph created# 实时查看进度，确保所有Pod处于Running，OSD prepare POD处于Completed，时间稍微长一些
kubectl get pod -n rook-ceph -w
# 实时查看集群创建进度
kubectl get cephcluster -n rook-ceph rook-ceph -w
# 详细描述
kubectl describe cephcluster -n rook-ceph rook-ceph[root@ceph61 examples]# kubectl get pod -n rook-ceph
NAME READY STATUS RESTARTS AGE
csi-cephfsplugin-6jcsf 2/2 Running 1 (14m ago) 25m
csi-cephfsplugin-8pj67 2/2 Running 1 (24m ago) 25m
csi-cephfsplugin-lgxb8 2/2 Running 0 25m
csi-cephfsplugin-provisioner-7b94748884-6st2v 5/5 Running 1 (14m ago) 25m
csi-cephfsplugin-provisioner-7b94748884-lbb5q 5/5 Running 0 25m
csi-rbdplugin-7h4fd 2/2 Running 1 (24m ago) 25m
csi-rbdplugin-dflzl 2/2 Running 1 (14m ago) 25m
csi-rbdplugin-jz9fq 2/2 Running 0 25m
csi-rbdplugin-provisioner-5d78f964b7-h4tbd 5/5 Running 3 (13m ago) 25m
csi-rbdplugin-provisioner-5d78f964b7-xxqzp 5/5 Running 1 (14m ago) 25m
rook-ceph-crashcollector-ceph61-5cbdcd57f4-7xwgv 1/1 Running 0 2m10s
rook-ceph-crashcollector-ceph62-745dc4c977-vdctc 1/1 Running 0 2m22s
rook-ceph-crashcollector-ceph63-b8f9b54d6-j2784 1/1 Running 0 2m23s
rook-ceph-exporter-ceph61-74c796f9b4-2g8bv 1/1 Running 0 2m10s
rook-ceph-exporter-ceph62-687b86d6f5-dv7x6 1/1 Running 0 2m22s
rook-ceph-exporter-ceph63-5d86c955fb-nc2q4 1/1 Running 0 2m23s
rook-ceph-mgr-a-5d5ccb978d-ss77f 3/3 Running 0 2m23s
rook-ceph-mgr-b-5f5fcfcb5f-slzv5 3/3 Running 0 2m22s
rook-ceph-mon-a-57dd87b4f-bnm5t 2/2 Running 0 14m
rook-ceph-mon-b-6bdf4c8d65-r7ncw 2/2 Running 0 14m
rook-ceph-mon-c-6ffb957cb7-77fcb 2/2 Running 0 9m10s
rook-ceph-operator-5bfd9fd4fb-k2jr4 1/1 Running 0 30m
rook-ceph-osd-prepare-ceph61-96lgt 0/1 Completed 0 2m
rook-ceph-osd-prepare-ceph62-xbzmm 0/1 Completed 0 2m
rook-ceph-osd-prepare-ceph63-hj5m7 0/1 Completed 0 119s2.6 强制删除命名空间
# 强制删除命名空间
[root@ceph61 examples]# kubectget namespace
NAME STATUS AGE
cattle-system Active 40h
default Active 42h
kube-node-lease Active 42h
kube-public Active 42h
kube-system Active 42h
kuboard Active 41h
rook-ceph Terminating 23h# rook-ceph命名空间 Terminating
kubectl delete ns rook-ceph --grace-period=0 --force
kubectl delete namespace rook-ceph --force --grace-period=0# 如果强制删除，还是无法成功的话，执行如下命令：
yum install jq -y
NAMESPACE=rook-ceph
kubectl proxy & kubectl get namespace $NAMESPACE -o json |jq '.spec = {"finalizers":[]}' >temp.json
curl -k -H "Content-Type: application/json" -X PUT --data-binary @temp.json 127.0.0.1:8001/api/v1/namespaces/$NAMESPACE/finalize[root@ceph61 ~]# NAMESPACE=rook-ceph
[root@ceph61 ~]# kubectl proxy & kubectl get namespace $NAMESPACE -o json |jq '.spec = {"finalizers":[]}' >temp.json
[1] 29753
Starting to serve on 127.0.0.1:8001
[root@ceph61 ~]# curl -k -H "Content-Type: application/json" -X PUT --data-binary @temp.json 127.0.0.1:8001/api/v1/namespaces/$NAMESPACE/finalize
{"kind": "Namespace","apiVersion": "v1","metadata": {"name": "rook-ceph","uid": "92a3a816-1464-4ecd-8d1c-c13cc1f8a345","resourceVersion": "70192","creationTimestamp": "2024-02-06T02:51:20Z","deletionTimestamp": "2024-02-06T11:03:41Z","labels": {"kubernetes.io/metadata.name": "rook-ceph"},"managedFields": [{"manager": "kubectl-create","operation": "Update","apiVersion": "v1","time": "2024-02-06T02:51:20Z","fieldsType": "FieldsV1","fieldsV1": {"f:metadata": {"f:labels": {".": {},"f:kubernetes.io/metadata.name": {}}}}},{"manager": "kube-controller-manager","operation": "Update","apiVersion": "v1","time": "2024-02-06T11:04:08Z","fieldsType": "FieldsV1","fieldsV1": {"f:status": {"f:conditions": {".": {},"k:{\"type\":\"NamespaceContentRemaining\"}": {".": {},"f:lastTransitionTime": {},"f:message": {},"f:reason": {},"f:status": {},"f:type": {}},"k:{\"type\":\"NamespaceDeletionContentFailure\"}": {".": {},"f:lastTransitionTime": {},"f:message": {},"f:reason": {},"f:status": {},"f:type": {}},"k:{\"type\":\"NamespaceDeletionDiscoveryFailure\"}": {".": {},"f:lastTransitionTime": {},"f:message": {},"f:reason": {},"f:status": {},"f:type": {}},"k:{\"type\":\"NamespaceDeletionGroupVersionParsingFailure\"}": {".": {},"f:lastTransitionTime": {},"f:message": {},"f:reason": {},"f:status": {},"f:type": {}},"k:{\"type\":\"NamespaceFinalizersRemaining\"}": {".": {},"f:lastTransitionTime": {},"f:message": {},"f:reason": {},"f:status": {},"f:type": {}}}}},"subresource": "status"}]},"spec": {},"status": {"phase": "Terminating","conditions": [{"type": "NamespaceDeletionDiscoveryFailure","status": "True","lastTransitionTime": "2024-02-06T11:03:46Z","reason": "DiscoveryFailed","message": "Discovery failed for some groups, 1 failing: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: stale GroupVersion discovery: metrics.k8s.io/v1beta1"},{"type": "NamespaceDeletionGroupVersionParsingFailure","status": "False","lastTransitionTime": "2024-02-06T11:03:46Z","reason": "ParsedGroupVersions","message": "All legacy kube types successfully parsed"},{"type": "NamespaceDeletionContentFailure","status": "False","lastTransitionTime": "2024-02-06T11:03:46Z","reason": "ContentDeleted","message": "All content successfully deleted, may be waiting on finalization"},{"type": "NamespaceContentRemaining","status": "False","lastTransitionTime": "2024-02-06T11:04:08Z","reason": "ContentRemoved","message": "All content successfully removed"},{"type": "NamespaceFinalizersRemaining","status": "False","lastTransitionTime": "2024-02-06T11:03:46Z","reason": "ContentHasNoFinalizers","message": "All content-preserving finalizers finished"}]}
}# 查看
[root@ceph61 ~]# kubectl get namespace
NAME STATUS AGE
cattle-system Active 40h
default Active 42h
kube-node-lease Active 42h
kube-public Active 42h
kube-system Active 42h
kuboard Active 41h2.7 验证集群
# 要验证群集是否处于正常状态，可以连接到 toolbox 工具箱并运行命令
cd /root/rook-1.13.3/deploy/examples
kubectl create -f toolbox.yaml
[root@ceph61 examples]# kubectl create -f toolbox.yaml
deployment.apps/rook-ceph-tools created# 查看Pod
[root@ceph61 examples]# kubectl get pod -n rook-ceph
NAME READY STATUS RESTARTS AGE
csi-cephfsplugin-6jcsf 2/2 Running 1 (21m ago) 32m
csi-cephfsplugin-8pj67 2/2 Running 1 (32m ago) 32m
csi-cephfsplugin-lgxb8 2/2 Running 0 32m
csi-cephfsplugin-provisioner-7b94748884-6st2v 5/5 Running 1 (21m ago) 32m
csi-cephfsplugin-provisioner-7b94748884-lbb5q 5/5 Running 0 32m
csi-rbdplugin-7h4fd 2/2 Running 1 (32m ago) 32m
csi-rbdplugin-dflzl 2/2 Running 1 (21m ago) 32m
csi-rbdplugin-jz9fq 2/2 Running 0 32m
csi-rbdplugin-provisioner-5d78f964b7-h4tbd 5/5 Running 3 (20m ago) 32m
csi-rbdplugin-provisioner-5d78f964b7-xxqzp 5/5 Running 1 (21m ago) 32m
rook-ceph-crashcollector-ceph61-5cbdcd57f4-7xwgv 1/1 Running 0 9m40s
rook-ceph-crashcollector-ceph62-6cbdf99cff-vrw44 1/1 Running 0 7m15s
rook-ceph-crashcollector-ceph63-855974fbd8-8w5ql 1/1 Running 0 7m16s
rook-ceph-exporter-ceph61-74c796f9b4-2g8bv 1/1 Running 0 9m40s
rook-ceph-exporter-ceph62-7f845dcd9c-fjr87 1/1 Running 0 7m11s
rook-ceph-exporter-ceph63-5dcf677d86-ct776 1/1 Running 0 7m11s
rook-ceph-mgr-a-5d5ccb978d-ss77f 3/3 Running 0 9m53s
rook-ceph-mgr-b-5f5fcfcb5f-slzv5 3/3 Running 0 9m52s
rook-ceph-mon-a-57dd87b4f-bnm5t 2/2 Running 0 22m
rook-ceph-mon-b-6bdf4c8d65-r7ncw 2/2 Running 0 21m
rook-ceph-mon-c-6ffb957cb7-77fcb 2/2 Running 0 16m
rook-ceph-operator-5bfd9fd4fb-k2jr4 1/1 Running 0 38m
rook-ceph-osd-0-7cbd59747f-5sl7v 2/2 Running 0 7m16s
rook-ceph-osd-1-f487664fb-gdwlx 2/2 Running 0 7m16s
rook-ceph-osd-2-77b849c5b-lc7p2 2/2 Running 0 7m16s
rook-ceph-osd-3-5469748766-tj256 2/2 Running 0 7m16s
rook-ceph-osd-4-77d4bd4b7c-2gqdz 2/2 Running 0 7m16s
rook-ceph-osd-5-694548dbcc-jr9bg 2/2 Running 0 7m16s
rook-ceph-osd-6-844ff54957-d84xs 2/2 Running 0 7m16s
rook-ceph-osd-7-5c7b9d4b76-fsh46 2/2 Running 0 7m16s
rook-ceph-osd-8-6dd768d7fb-9smbs 2/2 Running 0 7m16s
rook-ceph-osd-prepare-ceph61-czhcp 0/1 Completed 0 2m53s
rook-ceph-osd-prepare-ceph62-2754w 0/1 Completed 0 2m49s
rook-ceph-osd-prepare-ceph63-fhqfl 0/1 Completed 0 2m45s
rook-ceph-tools-66b77b8df5-gj256 1/1 Running 0 3m26s# 进入Pod容器
kubectl exec -it rook-ceph-tools-66b77b8df5-gj256 -n rook-ceph -- bash# 进去之后就可以执行各种命令了：
ceph status
ceph osd status
ceph df
rados df[root@ceph61 examples]# kubectl exec -it rook-ceph-tools-66b77b8df5-gj256 -n rook-ceph -- bash
bash-4.4$ ceph -v
ceph version 18.2.1 (7fe91d5d5842e04be3b4f514d6dd990c54b29c76) reef (stable)
bash-4.4$ ceph -scluster:id: 7793dc7d-fb3a-48dd-9297-8f2d37fb9901health: HEALTH_WARNSlow OSD heartbeats on back (longest 69426.357ms)Slow OSD heartbeats on front (longest 65492.636ms)services:mon: 3 daemons, quorum a,c,b (age 18s)mgr: b(active, since 9m), standbys: aosd: 9 osds: 9 up (since 76s), 9 in (since 15m)data:pools: 1 pools, 1 pgsobjects: 2 objects, 449 KiBusage: 240 MiB used, 270 GiB / 270 GiB availpgs: 1 active+cleanbash-4.4$ ceph osd status
ID HOST USED AVAIL WR OPS WR DATA RD OPS RD DATA STATE0 ceph62 26.5M 29.9G 0 0 0 0 exists,up1 ceph63 26.9M 29.9G 0 0 0 0 exists,up2 ceph62 26.9M 29.9G 0 0 0 0 exists,up3 ceph63 26.5M 29.9G 0 0 0 0 exists,up4 ceph62 26.5M 29.9G 0 0 0 0 exists,up5 ceph63 26.5M 29.9G 0 0 0 0 exists,up6 ceph61 26.5M 29.9G 0 0 0 0 exists,up7 ceph61 26.9M 29.9G 0 0 0 0 exists,up8 ceph61 26.5M 29.9G 0 0 0 0 exists,up
bash-4.4$ ceph df
--- RAW STORAGE ---
CLASS SIZE AVAIL USED RAW USED %RAW USED
hdd 270 GiB 270 GiB 240 MiB 240 MiB 0.09
TOTAL 270 GiB 270 GiB 240 MiB 240 MiB 0.09--- POOLS ---
POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL
.mgr 1 1 449 KiB 2 1.3 MiB 0 85 GiB
bash-4.4$ rados df
POOL_NAME USED OBJECTS CLONES COPIES MISSING_ON_PRIMARY UNFOUND DEGRADED RD_OPS RD WR_OPS WR USED COMPR UNDER COMPR
.mgr 1.3 MiB 2 0 6 0 0 0 288 494 KiB 153 1.3 MiB 0 B 0 Btotal_objects 2
total_used 240 MiB
total_avail 270 GiB
total_space 270 GiB3.Ceph界面
# Dashboard，查看dashboard的svc，Rook将启用端口 8443 以进行 https 访问
kubectl get svc -n rook-ceph
[root@ceph61 examples]# kubectl get svc -n rook-ceph
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
rook-ceph-exporter ClusterIP 10.104.154.115 <none> 9926/TCP 22m
rook-ceph-mgr ClusterIP 10.109.137.46 <none> 9283/TCP 22m
rook-ceph-mgr-dashboard ClusterIP 10.101.248.83 <none> 8443/TCP 22m
rook-ceph-mon-a ClusterIP 10.100.67.82 <none> 6789/TCP,3300/TCP 35m
rook-ceph-mon-b ClusterIP 10.97.62.97 <none> 6789/TCP,3300/TCP 34m
rook-ceph-mon-c ClusterIP 10.99.101.215 <none> 6789/TCP,3300/TCP 29m# rook-ceph-mgr：是 Ceph 的管理进程（Manager），负责集群的监控、状态报告、数据分析、调度等功能，它默认监听 9283 端口，并提供了 Prometheus 格式的监控指标，可以被 Prometheus 拉取并用于集群监控。
# rook-ceph-mgr-dashboard：是 Rook 提供的一个 Web 界面，用于方便地查看 Ceph 集群的监控信息、状态、性能指标等。
# rook-ceph-mon：是 Ceph Monitor 进程的 Kubernetes 服务。Ceph Monitor 是 Ceph 集群的核心组件之一，负责维护 Ceph 集群的状态、拓扑结构、数据分布等信息，是 Ceph 集群的管理节点。# 查看默认账号admin的密码，这个密码就是等会用来登录界面的
# kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath="{['data']['password']}" | base64 --decode && echo
[root@ceph61 examples]# kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath="{['data']['password']}" | base64 --decode && echo
2<rI.]qNXqGX#F2c-M@E# 使用 NodePort 类型的Service暴露dashboard：dashboard-external-https.yaml
[root@ceph61 examples]# cat dashboard-external-https.yaml
apiVersion: v1
kind: Service
metadata:name: rook-ceph-mgr-dashboard-external-httpsnamespace: rook-ceph # namespace:clusterlabels:app: rook-ceph-mgrrook_cluster: rook-ceph # namespace:cluster
spec:ports:- name: dashboardport: 8443protocol: TCPtargetPort: 8443selector:app: rook-ceph-mgrmgr_role: activerook_cluster: rook-ceph # namespace:clustersessionAffinity: Nonetype: NodePort# 创建
kubectl create -f dashboard-external-https.yaml
[root@ceph61 examples]# kubectl create -f dashboard-external-https.yaml
service/rook-ceph-mgr-dashboard-external-https created[root@ceph61 examples]# kubectl get svc -n rook-ceph
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
rook-ceph-exporter ClusterIP 10.104.154.115 <none> 9926/TCP 28m
rook-ceph-mgr ClusterIP 10.109.137.46 <none> 9283/TCP 28m
rook-ceph-mgr-dashboard ClusterIP 10.101.248.83 <none> 8443/TCP 28m
rook-ceph-mgr-dashboard-external-https NodePort 10.97.245.134 <none> 8443:31222/TCP 20s
rook-ceph-mon-a ClusterIP 10.100.67.82 <none> 6789/TCP,3300/TCP 40m
rook-ceph-mon-b ClusterIP 10.97.62.97 <none> 6789/TCP,3300/TCP 40m
rook-ceph-mon-c ClusterIP 10.99.101.215 <none> 6789/TCP,3300/TCP 35m# 访问
https://192.168.120.61:31222/
用户名：admin
密码：2<rI.]qNXqGX#F2c-M@E
相关文章：
K8s环境下rook-v1.13.3部署Ceph-v18.2.1集群
文章目录 1.K8s环境搭建2.Ceph集群部署2.1 部署Rook Operator2.2 镜像准备2.3 配置节点角色2.4 部署operator2.5 部署Ceph集群2.6 强制删除命名空间2.7 验证集群 3.Ceph界面 1.K8s环境搭建 参考：CentOS7搭建k8s-v1.28.6集群详情，把K8s集群完成搭建&…...
【JavaEE】传输层网络协议
传输层网络协议 1. UDP协议 1.1 特点 面向数据报（DatagramSocket）数据报大小限制为64k全双工不可靠传输有接收缓冲区，无发送缓冲区 UDP的特点，我理解起来就是工人组成的**“人工传送带”**： 面向数据报（…...
08-Java过滤器模式 ( Filter Pattern )
Java过滤器模式 实现范例 过滤器模式（Filter Pattern）或允许开发人员使用不同的标准来过滤一组对象，通过逻辑运算以解耦的方式把它们连接起来 过滤器模式（Filter Pattern） 又称 标准模式（Criteria Pattern…...
ChatGPT高效提问—prompt常见用法（续篇八）
ChatGPT高效提问—prompt常见用法（续篇八） 1.1 对抗 对抗是一个重要主题，深入探讨了大型语言模型（LLM）的安全风险。它不仅反映了人们对LLM可能出现的风险和安全问题的理解，而且能够帮助我们识别这些潜在的风险，并通过切实可行的技术手段来规避。 截至目前，网络…...
微软.NET6开发的C#特性——接口和属性
我是荔园微风，作为一名在IT界整整25年的老兵，看到不少初学者在学习编程语言的过程中如此的痛苦，我决定做点什么，下面我就重点讲讲微软.NET6开发人员需要知道的C#特性，然后比较其他各种语言进行认识。 C#经历了多年发展…...
容器基础知识：容器和虚拟化的区别
虚拟化与容器化对比 容器化和虚拟化都是用于优化资源利用率并实现高效应用程序部署的技术。然而，它们在方法和关键特征上存在差异： 虚拟化: 可以理解为创建虚拟机 (VM)。虚拟机模拟一台拥有自己硬件（CPU、内存、存储）和操作系统…...
【Linux】vim的基本操作与配置（下）
Hello everybody!今天我们继续讲解vim的操作与配置，希望大家在看过这篇文章与上篇文章后都能够轻松上手vim! 1.补充 在上一篇文章中我们说过了，在底行模式下set nu可以显示行号。今天补充一条：set nonu可以取消行号。这两条命令大家看看就可…...
[office] 图文演示excel怎样给单元格添加下拉列表 #知识分享#经验分享
图文演示excel怎样给单元格添加下拉列表 在Excel表格中输入数据的时候，为了简便快捷的输入，经常需要给Excel单元格添加一个下拉菜单，这样在输入数据时不必按键盘，只是用鼠标选择选项就可以了。 比的位置。 4、可以看到一个预览的…...
【RT-DETR有效改进】利用SENetV2重构化网络结构 (ILSVRC冠军得主，全网独家首发)
👑欢迎大家订阅本专栏，一起学习RT-DETR👑 一、本文介绍 本文给大家带来的改进机制是SENetV2，其是2023.11月的最新机制(所以大家想要发论文的可以在上面下点功夫)，其是一种通过调整卷积网络中的通道关系来提升性能的网络结构。SENet并不是一个独立的网络模型，而…...
【Linux】学习-进程间通信
进程间通信 介绍 进程间通信的本质 进程间通信的前提，首先需要让不同的进程看到同一块“内存”此“内存”一定不属于任何进程，而应该强调共享二字 进程间通信的目的 数据传输：一个进程需要将它的数据发送给另一个进程 资源共享：…...
45. C++ 字符指针和字符串
一、字符指针可以指向一个字符串。 我们可以用字符串常量对字符指针进行初始化。例如，有说明语句： char *str "This is a string.";是对字符指针进行初始化。此时，字符指针指向的是一个字符串常量的首地址，即指向字符…...
如何制作一款3D FPS游戏
制作一款3D FPS游戏是一个复杂的过程，需要涵盖多个方面，包括游戏设计、游戏引擎选择、模型制作、音效制作、关卡设计等。下面是一个关于如何制作一款3D FPS游戏的超长文章。 游戏设计 首先，你需要确定游戏的整体设计和核心玩法。这包括游戏的…...
人工智能|深度学习——使用多层级注意力机制和keras实现问题分类
代码下载 使用多层级注意力机制和keras实现问题分类资源-CSDN文库 1 准备工作 1.1 什么是词向量? ”词向量”（词嵌入）是将一类将词的语义映射到向量空间中去的自然语言处理技术。即将一个词用特定的向量来表示，向量之间的距离（例…...
C语言常见面试题：C语言中如何进行网页开发编程？
在C语言中进行网页开发通常不是一个直接的过程，因为C语言主要用于系统级编程，而不是Web开发。传统的Web开发主要使用高级语言如JavaScript、Python、Ruby、PHP等，以及与Web相关的技术，如HTML、CSS和数据库。 然而，如果…...
DevOps落地笔记-20|软件质量：决定系统成功的关键
上一课时介绍通过提高工程效率来提高价值交付效率，从而提高企业对市场的响应速度。在提高响应速度的同时，也不能降低软件的质量，这就是所谓的“保质保量”。具备高质量软件，高效率的企业走得更快更远。相反，低劣的软件…...
政安晨：梯度与导数～示例演绎《机器学习·神经网络》的高阶理解
这篇文章确实需要一定的数学基础，第一次接触的小伙伴可以先看一下我示例演绎这个主题的前两篇文章： 示例演绎机器学习中（深度学习）神经网络的数学基础——快速理解核心概念（一）： 政安晨&#…...
CTFSHOW命令执行web入门29-54
description: >- 这里就记录一下ctfshow的刷题记录是web入门的命令执行专题里面的题目，他是有分类，并且覆盖也很广泛，所以就通过刷这个来，不过里面有一些脚本的题目发现我自己根本不会笑死。 如果还不怎么知道写题的话，可以去看我的gitbook，当然csdn我也转载了我自己的…...
探索ChatGPT4：新一代人工智能语言模型的突破
ChatGPT4，作为最新一代的语言处理模型，代表了人工智能在自然语言理解和生成方面的最新突破。本文将深入介绍ChatGPT4的新特性，探讨其在各个领域的潜在应用。 ChatGPT4概述 在继承了前一代模型的强大基础之上，ChatGPT4引入了多项…...
PVST详解
PVST（Per-VLAN Spanning Tree）是Cisco公司的一种扩展的Spanning Tree协议，允许在每个VLAN中独立运行一个Spanning Tree实例，从而提高网络的可靠性和性能。 PVST协议在每个交换机中维护多个Spanning Tree实例，每个实例…...
c++ 子进程交互 逻辑
目录 一、主进程逻辑 1、创建子进程时候，写入自己的HWND 2、响应子进程消息...
云启出海，智联未来｜阿里云网络「企业出海」系列客户沙龙上海站圆满落地
借阿里云中企出海大会的东风，以**「云启出海，智联未来｜打造安全可靠的出海云网络引擎」为主题的阿里云企业出海客户沙龙云网络&安全专场于5.28日下午在上海顺利举办，现场吸引了来自携程、小红书、米哈游、哔哩哔哩、波克城市、…...
23-Oracle 23 ai 区块链表（Blockchain Table）
小伙伴有没有在金融强合规的领域中遇见，必须要保持数据不可变，管理员都无法修改和留痕的要求。比如医疗的电子病历中，影像检查检验结果不可篡改行的，药品追溯过程中数据只可插入无法删除的特性需求；登录日志、修改日志…...
Docker 运行 Kafka 带 SASL 认证教程
Docker 运行 Kafka 带 SASL 认证教程 Docker 运行 Kafka 带 SASL 认证教程一、说明二、环境准备三、编写 Docker Compose 和 jaas文件docker-compose.yml代码说明：server_jaas.conf 四、启动服务五、验证服务六、连接kafka服务七、总结 Docker 运行 Kafka 带 SASL 认…...
【位运算】消失的两个数字（hard）
消失的两个数字（hard） 题⽬描述：解法（位运算）：Java 算法代码：更简便代码 题⽬链接：⾯试题 17.19. 消失的两个数字 题⽬描述： 给定⼀个数组，包含从 1 到 N 所有…...
基于Uniapp开发HarmonyOS 5.0旅游应用技术实践
一、技术选型背景 1.跨平台优势 Uniapp采用Vue.js框架，支持"一次开发，多端部署"，可同步生成HarmonyOS、iOS、Android等多平台应用。 2.鸿蒙特性融合 HarmonyOS 5.0的分布式能力与原子化服务，为旅游应用带来…...
【算法训练营Day07】字符串part1
文章目录 反转字符串反转字符串II替换数字 反转字符串 题目链接：344. 反转字符串 双指针法，两个指针的元素直接调转即可 class Solution {public void reverseString(char[] s) {int head 0;int end s.length - 1;while(head < end) {char temp …...
在鸿蒙HarmonyOS 5中使用DevEco Studio实现录音机应用
1. 项目配置与权限设置 1.1 配置module.json5 {"module": {"requestPermissions": [{"name": "ohos.permission.MICROPHONE","reason": "录音需要麦克风权限"},{"name": "ohos.permission.WRITE…...
均衡后的SNRSINR
本文主要摘自参考文献中的前两篇，相关文献中经常会出现MIMO检测后的SINR不过一直没有找到相关数学推到过程，其中文献[1]中给出了相关原理在此仅做记录。 1. 系统模型 复信道模型 n t n_t nt 根发送天线， n r n_r nr 根接收天线的 MIMO 系…...
软件工程 期末复习
瀑布模型：计划 螺旋模型：风险低 原型模型: 用户反馈 喷泉模型:代码复用 高内聚 低耦合：模块内部功能紧密 模块之间依赖程度小 高内聚：指的是一个模块内部的功能应该紧密相关。换句话说，一个模块应当只实现单一的功能…...
Monorepo架构: Nx Cloud 扩展能力与缓存加速
借助 Nx Cloud 实现项目协同与加速构建 1 ） 缓存工作原理分析 在了解了本地缓存和远程缓存之后，我们来探究缓存是如何工作的。以计算文件的哈希串为例，若后续运行任务时文件哈希串未变，系统会直接使用对应的输出和制品文件。 2 …...