JuiceFS-K8s部署

官网文档地址：https://juicefs.com/docs/zh/csi/introduction/

1、部署JuiceFS-CSI驱动

部署yaml如下，镜像已进行更改为镜像仓库内地址

---

# DO NOT EDIT: generated by 'kustomize build'
apiVersion: v1
kind: ServiceAccount
metadata:labels:app.kubernetes.io/instance: juicefs-csi-driverapp.kubernetes.io/name: juicefs-csi-driverapp.kubernetes.io/version: mastername: juicefs-csi-controller-sanamespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:labels:app.kubernetes.io/instance: juicefs-csi-driverapp.kubernetes.io/name: juicefs-csi-driverapp.kubernetes.io/version: mastername: juicefs-csi-node-sanamespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:labels:app.kubernetes.io/instance: juicefs-csi-driverapp.kubernetes.io/name: juicefs-csi-driverapp.kubernetes.io/version: mastername: juicefs-csi-external-node-service-role
rules:
- apiGroups:- ""resources:- pods- pods/log- secretsverbs:- get- list- create- update- delete- patch- watch
- apiGroups:- batchresources:- jobsverbs:- get- list- create- update- delete- patch- watch
- apiGroups:- ""resources:- nodes/proxyverbs:- '*'
- apiGroups:- ""resources:- persistentvolumesverbs:- get- list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:labels:app.kubernetes.io/instance: juicefs-csi-driverapp.kubernetes.io/name: juicefs-csi-driverapp.kubernetes.io/version: mastername: juicefs-external-provisioner-role
rules:
- apiGroups:- ""resources:- persistentvolumesverbs:- get- list- watch- create- delete
- apiGroups:- ""resources:- persistentvolumeclaimsverbs:- get- list- watch- update
- apiGroups:- storage.k8s.ioresources:- storageclassesverbs:- get- list- watch
- apiGroups:- ""resources:- eventsverbs:- get- list- watch- create- update- patch
- apiGroups:- storage.k8s.ioresources:- csinodesverbs:- get- list- watch
- apiGroups:- ""resources:- nodesverbs:- get- list- watch
- apiGroups:- ""resources:- secretsverbs:- get- list- create- update- patch- delete
- apiGroups:- ""resources:- pods- pods/logverbs:- get- list- watch- create- update- patch- delete
- apiGroups:- batchresources:- jobsverbs:- get- list- watch- create- update- patch- delete
- apiGroups:- ""resources:- endpointsverbs:- get- list- watch- create- update- patch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:labels:app.kubernetes.io/instance: juicefs-csi-driverapp.kubernetes.io/name: juicefs-csi-driverapp.kubernetes.io/version: mastername: juicefs-csi-node-service-binding
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: juicefs-csi-external-node-service-role
subjects:
- kind: ServiceAccountname: juicefs-csi-node-sanamespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:labels:app.kubernetes.io/instance: juicefs-csi-driverapp.kubernetes.io/name: juicefs-csi-driverapp.kubernetes.io/version: mastername: juicefs-csi-provisioner-binding
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: juicefs-external-provisioner-role
subjects:
- kind: ServiceAccountname: juicefs-csi-controller-sanamespace: kube-system
---
apiVersion: apps/v1
kind: StatefulSet
metadata:labels:app.kubernetes.io/component: controllerapp.kubernetes.io/instance: juicefs-csi-driverapp.kubernetes.io/name: juicefs-csi-driverapp.kubernetes.io/version: mastername: juicefs-csi-controllernamespace: kube-system
spec:replicas: 1selector:matchLabels:app: juicefs-csi-controllerapp.kubernetes.io/instance: juicefs-csi-driverapp.kubernetes.io/name: juicefs-csi-driverapp.kubernetes.io/version: masterserviceName: juicefs-csi-controllertemplate:metadata:labels:app: juicefs-csi-controllerapp.kubernetes.io/instance: juicefs-csi-driverapp.kubernetes.io/name: juicefs-csi-driverapp.kubernetes.io/version: masterspec:containers:- args:- --endpoint=$(CSI_ENDPOINT)- --logtostderr- --nodeid=$(NODE_NAME)- --v=5env:- name: CSI_ENDPOINTvalue: unix:///var/lib/csi/sockets/pluginproxy/csi.sock- name: NODE_NAMEvalueFrom:fieldRef:fieldPath: spec.nodeName- name: JUICEFS_MOUNT_NAMESPACEvalueFrom:fieldRef:fieldPath: metadata.namespace- name: POD_NAMEvalueFrom:fieldRef:fieldPath: metadata.name- name: HOST_IPvalueFrom:fieldRef:fieldPath: status.hostIP- name: KUBELET_PORTvalue: "10250"- name: JUICEFS_MOUNT_PATHvalue: /var/lib/juicefs/volume- name: JUICEFS_CONFIG_PATHvalue: /var/lib/juicefs/configimage: harbor.t3caic.com/juicefs/juicefs-csi-driver:v0.17.5livenessProbe:failureThreshold: 5httpGet:path: /healthzport: healthzinitialDelaySeconds: 10periodSeconds: 10timeoutSeconds: 3name: juicefs-pluginports:- containerPort: 9909name: healthzprotocol: TCPresources:limits:cpu: 1000mmemory: 1Girequests:cpu: 100mmemory: 512MisecurityContext:capabilities:add:- SYS_ADMINprivileged: truevolumeMounts:- mountPath: /var/lib/csi/sockets/pluginproxy/name: socket-dir- mountPath: /jfsmountPropagation: Bidirectionalname: jfs-dir- mountPath: /root/.juicefsmountPropagation: Bidirectionalname: jfs-root-dir- args:- --csi-address=$(ADDRESS)- --timeout=60s- --v=5env:- name: ADDRESSvalue: /var/lib/csi/sockets/pluginproxy/csi.sockimage: harbor.t3caic.com/juicefs/csi-provisioner:v1.6.0name: csi-provisionervolumeMounts:- mountPath: /var/lib/csi/sockets/pluginproxy/name: socket-dir- args:- --csi-address=$(ADDRESS)- --health-port=$(HEALTH_PORT)env:- name: ADDRESSvalue: /csi/csi.sock- name: HEALTH_PORTvalue: "9909"image: harbor.t3caic.com/juicefs/livenessprobe:v1.1.0name: liveness-probevolumeMounts:- mountPath: /csiname: socket-dirpriorityClassName: system-cluster-criticalserviceAccount: juicefs-csi-controller-satolerations:- key: CriticalAddonsOnlyoperator: Existsvolumes:- emptyDir: {}name: socket-dir- hostPath:path: /var/lib/juicefs/volumetype: DirectoryOrCreatename: jfs-dir- hostPath:path: /var/lib/juicefs/configtype: DirectoryOrCreatename: jfs-root-dirvolumeClaimTemplates: []
---
apiVersion: apps/v1
kind: DaemonSet
metadata:labels:app.kubernetes.io/component: nodeapp.kubernetes.io/instance: juicefs-csi-driverapp.kubernetes.io/name: juicefs-csi-driverapp.kubernetes.io/version: mastername: juicefs-csi-nodenamespace: kube-system
spec:selector:matchLabels:app: juicefs-csi-nodeapp.kubernetes.io/instance: juicefs-csi-driverapp.kubernetes.io/name: juicefs-csi-driverapp.kubernetes.io/version: mastertemplate:metadata:labels:app: juicefs-csi-nodeapp.kubernetes.io/instance: juicefs-csi-driverapp.kubernetes.io/name: juicefs-csi-driverapp.kubernetes.io/version: masterspec:containers:- args:- --endpoint=$(CSI_ENDPOINT)- --logtostderr- --nodeid=$(NODE_NAME)- --v=5- --enable-manager=trueenv:- name: CSI_ENDPOINTvalue: unix:/csi/csi.sock- name: NODE_NAMEvalueFrom:fieldRef:fieldPath: spec.nodeName- name: JUICEFS_MOUNT_NAMESPACEvalueFrom:fieldRef:fieldPath: metadata.namespace- name: POD_NAMEvalueFrom:fieldRef:fieldPath: metadata.name- name: HOST_IPvalueFrom:fieldRef:fieldPath: status.hostIP- name: KUBELET_PORTvalue: "10250"- name: JUICEFS_MOUNT_PATHvalue: /var/lib/juicefs/volume- name: JUICEFS_CONFIG_PATHvalue: /var/lib/juicefs/configimage: harbor.t3caic.com/juicefs/juicefs-csi-driver:v0.17.5lifecycle:preStop:exec:command:- /bin/sh- -c- rm /csi/csi.socklivenessProbe:failureThreshold: 5httpGet:path: /healthzport: healthzinitialDelaySeconds: 10periodSeconds: 10timeoutSeconds: 3name: juicefs-pluginports:- containerPort: 9909name: healthzprotocol: TCPresources:limits:cpu: 1000mmemory: 1Girequests:cpu: 100mmemory: 512MisecurityContext:privileged: truevolumeMounts:- mountPath: /var/lib/kubeletmountPropagation: Bidirectionalname: kubelet-dir- mountPath: /csiname: plugin-dir- mountPath: /devname: device-dir- mountPath: /jfsmountPropagation: Bidirectionalname: jfs-dir- mountPath: /root/.juicefsmountPropagation: Bidirectionalname: jfs-root-dir- args:- --csi-address=$(ADDRESS)- --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)- --v=5env:- name: ADDRESSvalue: /csi/csi.sock- name: DRIVER_REG_SOCK_PATHvalue: /var/lib/kubelet/csi-plugins/csi.juicefs.com/csi.sockimage: harbor.t3caic.com/juicefs/csi-node-driver-registrar:v2.1.0name: node-driver-registrarvolumeMounts:- mountPath: /csiname: plugin-dir- mountPath: /registrationname: registration-dir- args:- --csi-address=$(ADDRESS)- --health-port=$(HEALTH_PORT)env:- name: ADDRESSvalue: /csi/csi.sock- name: HEALTH_PORTvalue: "9909"image: harbor.t3caic.com/juicefs/livenessprobe:v1.1.0name: liveness-probevolumeMounts:- mountPath: /csiname: plugin-dirdnsPolicy: ClusterFirstWithHostNetpriorityClassName: system-node-criticalserviceAccount: juicefs-csi-node-satolerations:- key: CriticalAddonsOnlyoperator: Existsvolumes:- hostPath:path: /var/lib/kubelettype: Directoryname: kubelet-dir- hostPath:path: /var/lib/kubelet/csi-plugins/csi.juicefs.com/type: DirectoryOrCreatename: plugin-dir- hostPath:path: /var/lib/kubelet/plugins_registry/type: Directoryname: registration-dir- hostPath:path: /devtype: Directoryname: device-dir- hostPath:path: /var/lib/juicefs/volumetype: DirectoryOrCreatename: jfs-dir- hostPath:path: /var/lib/juicefs/configtype: DirectoryOrCreatename: jfs-root-dir
---
apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:labels:app.kubernetes.io/instance: juicefs-csi-driverapp.kubernetes.io/name: juicefs-csi-driverapp.kubernetes.io/version: mastername: csi.juicefs.com
spec:attachRequired: falsepodInfoOnMount: false

---

2、创建OBS认证信息Secret

需要修改的信息
metaurl: “用户名:密码@IP:端口/库名”
postgres://用户名:密码@IP:端口/库名 数据库需要提前进入进行创建库

storage: “obs” 存储类型标识
bucket: “” 桶的域名
access-key: “”
secret-key: “”

---

apiVersion: v1
kind: Secret
metadata:name: juicefs-sc-secretnamespace: kube-system
type: Opaque
stringData:name: "test"metaurl: ""storage: ""bucket: ""access-key: ""secret-key: ""

---

3、创建存储类

---

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:name: juicefs-sc
provisioner: csi.juicefs.com
reclaimPolicy: Retain
volumeBindingMode: Immediate
parameters:csi.storage.k8s.io/node-publish-secret-name: juicefs-sc-secretcsi.storage.k8s.io/node-publish-secret-namespace: kube-systemcsi.storage.k8s.io/provisioner-secret-name: juicefs-sc-secretcsi.storage.k8s.io/provisioner-secret-namespace: kube-system

---

4、创建PVC–PVC创建时会自动创建PV

目前而言，JuiceFS CSI 驱动不支持设置存储容量。在 PersistentVolume 和 PersistentVolumeClaim 中指定的容量会被忽略，填写任意有效值即可，例如 10Gi

---

apiVersion: v1
kind: PersistentVolumeClaim
metadata:name: "juicefs-test-pvc"namespace: test-juicefs
spec:storageClassName: sc-topolvmaccessModes:- ReadWriteOncevolumeMode: Filesystemresources:requests:storage: 10GistorageClassName: juicefs-sc

---

5、创建测试Pod–测试Pod创建容器内是否挂载成功

---

apiVersion: apps/v1
kind: Deployment
metadata:name: nginx-run
spec:selector:matchLabels:app: nginxtemplate:metadata:labels:app: nginxspec:containers:- name: nginximage: harbor.t3caic.com/nginxdemos/hello:plain-textports:- containerPort: 80volumeMounts:- mountPath: /configname: web-datavolumes:- name: web-datapersistentVolumeClaim:claimName: juicefs-test-pvc

---

ERR注意：如果测试Pod创建卡死，查看kube-system命名空间的juicefs-XXX-pvc容器起来了吗
如因为拉取不下镜像，请在此容器所在节点手动解决Pod拉取 docker pull
如果镜像拉取下来Pod一直在重启可以考虑一下是否是OBS连接不上，如果OBS需要做域名解析可以在CoreDNS里配置

kubectl edit configmap coredns -n kube-system
添加hosts {100.125.32.189 juicefs.obs.cn-east-263.t3caic.comfallthrough}

---