NFS 组件容器化部署实战指南

前言

使用nfs-client-provisioner这个应用，利用nfs server给kubernets提供作为持久化后端，并且动态提供pv。所有节点需要安装nfs-utils组件，并且nfs服务器与kubernets worker节点都能网络连通。

部署NFS服务器

注：这里使用master节点作为nfs服务器，根据自己资源情况进行调整部署。

安装nfs组件及创建数据目录
[root@k8s-master ~]# yum install -y nfs-utils rpcbind 
[root@k8s-master ~]# mkdir -p  /data/nfs_data编辑exportfs文件
# cat /etc/exports
/data/nfs_data 10.255.82.0/24(rw,no_root_squash)配置生效
# exportfs -rav启动rpcbind、nfs服务
[root@k8s-master ~]# systemctl restart rpcbind && systemctl enable rpcbind
[root@k8s-master ~]# systemctl restart nfs && systemctl enable nfsshowmount测试
[root@k8s-master ~]# showmount -e 10.255.82.25
Export list for 10.255.82.25
/data/nfs_data 10.255.82.0/24
[root@k8s-master ~]#

K8S部署NFS

nfs部署资源文件如下
[root@k8s-master nfs]# ls -l
总用量 12
-rw-r--r-- 1 root root  181 12月  8 17:47 class.yaml
-rw-r--r-- 1 root root  987 12月  8 17:43 deployment.yaml
-rw-r--r-- 1 root root 1509 12月  8 17:34 rbac.yaml
[root@k8s-master nfs]#创建nfs角色权限
[root@k8s-master nfs]# kubectl apply -f rbac.yaml
clusterrole.rbac.authorization.k8s.io/nfs-client-provisioner-runner created
clusterrolebinding.rbac.authorization.k8s.io/run-nfs-client-provisioner created
role.rbac.authorization.k8s.io/leader-locking-nfs-client-provisioner created
rolebinding.rbac.authorization.k8s.io/leader-locking-nfs-client-provisioner created
[root@k8s-master nfs]# 部署nfs-client-provisioner
[root@k8s-master nfs]# kubectl apply -f deployment.yaml
serviceaccount/nfs-client-provisioner created
deployment.apps/nfs-client-provisioner created
[root@k8s-master nfs]#创建nfs storageclass
[root@k8s-master nfs]# kubectl apply -f class.yaml
storageclass.storage.k8s.io/managed-nfs-storage created
[root@k8s-master nfs]#验证nfs部署成功
[root@k8s-master nfs]# kubectl get pod -l app=nfs-client-provisioner
NAME                                      READY   STATUS    RESTARTS      AGE
nfs-client-provisioner-647d8f5c7b-5lk5b   1/1     Running   3 (13h ago)   15h
[root@k8s-master nfs]#查看nfs storageclass
[root@k8s-master nfs]# kubectl get storageclasses.storage.k8s.io
NAME                  PROVISIONER                     RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
managed-nfs-storage   fuseim.pri/ifs                  Delete          Immediate           true                   15h部署nginx服务使用nfs存储持久化[root@k8s-master nfs]# cat nfs-ngx-sts.yaml
apiVersion: v1
kind: Service
metadata:name: nginx-svclabels:app: nginx
spec:ports:- port: 80name: webclusterIP: Noneselector:app: nginx
---
apiVersion: apps/v1
kind: StatefulSet
metadata:name: web
spec:selector:matchLabels:app: nginx # has to match .spec.template.metadata.labelsserviceName: "nginx"replicas: 1 # by default is 1minReadySeconds: 10 # by default is 0template:metadata:labels:app: nginx # has to match .spec.selector.matchLabelsspec:terminationGracePeriodSeconds: 10containers:- name: nginximage: nginx:latestimagePullPolicy: IfNotPresentports:- containerPort: 80name: webvolumeMounts:- name: wwwmountPath: /usr/share/nginx/htmlvolumeClaimTemplates:- metadata:name: wwwspec:accessModes: [ "ReadWriteOnce" ]storageClassName: "managed-nfs-storage"resources:requests:storage: 1Gi
[root@k8s-master nfs]#
[root@k8s-master nfs]# kubectl apply -f nfs-ngx-sts.yaml查看ngx-pod运行状态
[root@k8s-master nfs]# kubectl get pod | grep web
web-0                                     1/1     Running   0             13h
[root@k8s-master nfs]#查看pvc状态
[root@k8s-master nfs]# kubectl get pvc -l app=nginx
NAME        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS          AGE
www-web-0   Bound    pvc-782925cf-307b-4602-a226-871825fa86ac   1Gi        RWO            managed-nfs-storage   13h
[root@k8s-master nfs]#验证ngx服务存储nfs持久化[root@k8s-master nfs]# kubectl exec -it web-0 -- bash
root@web-0:/# df -h
Filesystem                                                                                Size  Used Avail Use% Mounted on
overlay                                                                                    17G  5.1G   12G  30% /
tmpfs                                                                                      64M     0   64M   0% /dev
tmpfs                                                                                     2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/sda2                                                                                  17G  5.1G   12G  30% /etc/hosts
shm                                                                                        64M     0   64M   0% /dev/shm
192.168.31.102:/data/nfs_data/default-www-web-0-pvc-782925cf-307b-4602-a226-871825fa86ac   17G  7.5G  9.6G  45% /usr/share/nginx/html
tmpfs                                                                                     3.8G   12K  3.8G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs                                                                                     2.0G     0  2.0G   0% /proc/acpi
tmpfs                                                                                     2.0G     0  2.0G   0% /proc/scsi
tmpfs                                                                                     2.0G     0  2.0G   0% /sys/firmware
root@web-0:/# ls -lh /usr/share/nginx/html/
total 0
root@web-0:/# echo "nfs storage" > /usr/share/nginx/html/index.html
root@web-0:/# cat /usr/share/nginx/html/index.html
nfs storage
root@web-0:/# exit
exitnfs服务器端验证数据
[root@k8s-master nfs]# cat /data/nfs_data/default-www-web-0-pvc-782925cf-307b-4602-a226-871825fa86ac/index.html
nfs storage
[root@k8s-master nfs]#ngx资源删除验证数据是否持久化nfs存储
[root@k8s-master nfs]# kubectl delete -f nfs-ngx-sts.yaml
service "nginx-svc" deleted
statefulset.apps "web" deleted
[root@k8s-master nfs]# cat /data/nfs_data/default-www-web-0-pvc-782925cf-307b-4602-a226-871825fa86ac/index.html
nfs storage
[root@k8s-master nfs]#

问题记录

pvc创建处于pending状态事件描述：Normal ExternalProvisioning 2s (x4 over 48s) persistentvolume-controller waiting for a volume to be created, either by external provisioner “fuseim.pri/ifs” or manually created by system administrator

原因：
Kubernetes v1.20 (opens new window)开始，默认删除了 metadata.selfLink 字段，然而，部分应用仍然依赖于这个字段，例如 nfs-client-provisioner。如果仍然要继续使用这些应用，您将需要重新启用该字段。

解决方法：
通过配置 apiserver 启动参数中的 --feature-gates 中的 RemoveSelfLink=false，可以重新启用 metadata.selfLink 字段。

1. 如果使用kubeadm安装Kubernetes，请修改/etc/kubernetes/manifests/kube-apiserver.yaml文件，并在其启动参数中增加一行- --feature-gates=RemoveSelfLink=false，如下所示：
   
   然后更新kube-apiserver.yaml即可
   kubectl apply -f /etc/kubernetes/manifests/kube-apiserver.yaml

2. ②　如果是通过二进制部署kube-apiserver
   通过systemctl status kube-apiserver 这个命令找到 kube-apiserver.service文件，
   最好先备份，然后再修改，即在 kube-apiserver.service中添加 --feature-gates=RemoveSelfLink=false。

# systemctl daemon-reload 
# systemctl restart kube-apiserver