研发工程师玩转Kubernetes——PVC通过storageClassName进行延迟绑定

不同的PV可以使用相同的StorageClass，它们是一对多的关系。

PV可以设置节点亲和性。比如下图，local-storage-class-waitforfirstconsumer-pv-ubuntuc只能在节点ubuntuc上；local-storage-class-waitforfirstconsumer-pv-ubuntud只能在节点ubuntud上。

如果我们使用《研发工程师玩转Kubernetes——PVC使用storageClassName选择PV》一文中的“立即绑定”型的StorageClass。

# local_storage_class_immediate.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:name: local-storage-class-immediate
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: Immediate

则在创建PVC时，立即会选择一个PV。这样这个PVC可能绑定到节点ubuntuc对应的PV上，也可能绑定到节点ubuntud对应的PV上。现在我们假定它绑定到ubuntuc对应的PV——local-storage-class-immediate-pv-ubuntuc上。
假如PVC的使用者——Pod在调度时，清单文件要求它只能在ubuntud上使用。而其PVC却在ubuntuc上，则会调度失败。

立即绑定导致Pod调度失败的案例

StorageClass

注意volumeBindingMode是Immediate，即PVC创建时立即绑定PV。

# local_storage_class_immediate.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:name: local-storage-class-immediate
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: Immediate

PersistentVolume

ubuntuc

# local_storage_class_immediate_pv_ubuntuc.yaml
apiVersion: v1
kind: PersistentVolume
metadata:name: local-storage-class-immediate-pv-ubuntuc
spec:capacity:storage:  1MivolumeMode: FilesystemaccessModes:- ReadWriteOncepersistentVolumeReclaimPolicy: DeletestorageClassName: local-storage-class-immediatelocal:path: /tmpnodeAffinity:required:nodeSelectorTerms:- matchExpressions:- key: kubernetes.io/hostnameoperator: Invalues:- ubuntuc

ubuntud

# local_storage_class_immediate_pv_ubuntud.yaml
apiVersion: v1
kind: PersistentVolume
metadata:name: local-storage-class-immediate-pv-ubuntud
spec:capacity:storage:  1MivolumeMode: FilesystemaccessModes:- ReadWriteOncepersistentVolumeReclaimPolicy: DeletestorageClassName: local-storage-class-immediatelocal:path: /tmpnodeAffinity:required:nodeSelectorTerms:- matchExpressions:- key: kubernetes.io/hostnameoperator: Invalues:- ubuntud

PersistentVolumeClaim

# local_storage_class_immediate_pvc_600k.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:name: local-storage-class-immediate-pvc-600k
spec:resources:requests:storage: 600KiaccessModes:- ReadWriteOncestorageClassName: local-storage-class-immediate

创建完上述组件，我们查看下PVC的状态。

kubectl describe persistentvolumeclaims local-storage-class-immediate-pvc-600k 

Name:          local-storage-class-immediate-pvc-600k
Namespace:     default
StorageClass:  local-storage-class-immediate
Status:        Bound
Volume:        local-storage-class-immediate-pv-ubuntuc
Labels:        <none>
Annotations:   pv.kubernetes.io/bind-completed: yespv.kubernetes.io/bound-by-controller: yes
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      1Mi
Access Modes:  RWO
VolumeMode:    Filesystem
Used By:       <none>
Events:        <none>

可以看到这个PVC处于绑定状态。

Deployment

# local_deployment_immediate.yaml
apiVersion: apps/v1
kind: Deployment
metadata:name: local-pv-app-immediate-deployment
spec:replicas: 1selector:matchLabels:app: local-pv-app-immediatetemplate:metadata:labels:app: local-pv-app-immediatespec:containers:- name: local-pv-app-immediateimage: busyboxcommand: ["/bin/sh", "-c", "if [ -f /tempdir/lockfile ] && ! { set -C; 2>/dev/null >/tempdir/lockfile; }; then tail -f /tempdir/lockfile; else exec 3>/tempdir/lockfile; if [ -n \"$POD_NAME\" ]; then name=$POD_NAME; else name=\"unknown\"; fi; while true; do echo \"this is $name.$name write something to lockfile\"; echo \"$name write something to lockfile\" >&3; sleep 1; done; fi"]  volumeMounts:- name: local-pvc-volumemountPath: /tempdirenv:- name: POD_NAMEvalueFrom:fieldRef:fieldPath: metadata.namevolumes:- name: local-pvc-volumepersistentVolumeClaim:claimName: local-storage-class-immediate-pvc-600kaffinity:nodeAffinity:requiredDuringSchedulingIgnoredDuringExecution:nodeSelectorTerms:- matchExpressions:- key: kubernetes.io/hostnameoperator: Invalues:- ubuntud    

这个清单要求Pod使用local-storage-class-immediate-pvc-600k这个PVC，但是要求自己只能被部署在节点ubuntud上。

错误表现

kubectl describe pod local-pv-app-immediate-deployment-6dd57d98f5-s5vpz 

Name:             local-pv-app-immediate-deployment-6dd57d98f5-s5vpz
Namespace:        default
Priority:         0
Service Account:  default
Node:             <none>
Labels:           app=local-pv-app-immediatepod-template-hash=6dd57d98f5
Annotations:      <none>
Status:           Pending
IP:               
IPs:              <none>
Controlled By:    ReplicaSet/local-pv-app-immediate-deployment-6dd57d98f5
Containers:local-pv-app-immediate:Image:      busyboxPort:       <none>Host Port:  <none>Command:/bin/sh-cif [ -f /tempdir/lockfile ] && ! { set -C; 2>/dev/null >/tempdir/lockfile; }; then tail -f /tempdir/lockfile; else exec 3>/tempdir/lockfile; if [ -n "$POD_NAME" ]; then name=$POD_NAME; else name="unknown"; fi; while true; do echo "this is $name.$name write something to lockfile"; echo "$name write something to lockfile" >&3; sleep 1; done; fiEnvironment:POD_NAME:  local-pv-app-immediate-deployment-6dd57d98f5-s5vpz (v1:metadata.name)Mounts:/tempdir from local-pvc-volume (rw)/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-r48fn (ro)
Conditions:Type           StatusPodScheduled   False 
Volumes:local-pvc-volume:Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)ClaimName:  local-storage-class-immediate-pvc-600kReadOnly:   falsekube-api-access-r48fn:Type:                    Projected (a volume that contains injected data from multiple sources)TokenExpirationSeconds:  3607ConfigMapName:           kube-root-ca.crtConfigMapOptional:       <nil>DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300snode.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:Type     Reason            Age   From               Message----     ------            ----  ----               -------Warning  FailedScheduling  8s    default-scheduler  0/5 nodes are available: 1 node(s) didn't match Pod's node affinity/selector. preemption: 0/5 nodes are available: 1 Preemption is not helpful for scheduling, 4 No preemption victims found for incoming pod..

可以看到Pod调度失败，处于Pending状态。

延迟绑定导致Pod调度成功的案例

StorageClass

注意volumeBindingMode是WaitForFirstConsumer，即PVC创建时不绑定PV。而在PVC被使用（Pod被调度到）时绑定PV。

# local_storage_class_waitforfirstconsumer.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:name: local-storage-class-waitforfirstconsumer
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

PersistentVolume

和之前的PV设置相似，核心就是storageClassName不同，使用了延迟绑定的StorageClass。

ubuntuc

# local_storage_class_waitforfirstconsumer_pv_ubuntuc.yaml
apiVersion: v1
kind: PersistentVolume
metadata:name: local-storage-class-waitforfirstconsumer-pv-ubuntuc
spec:capacity:storage:  1MivolumeMode: FilesystemaccessModes:- ReadWriteOncepersistentVolumeReclaimPolicy: DeletestorageClassName: local-storage-class-waitforfirstconsumerlocal:path: /tmpnodeAffinity:required:nodeSelectorTerms:- matchExpressions:- key: kubernetes.io/hostnameoperator: Invalues:- ubuntuc

ubuntud

# local_storage_class_waitforfirstconsumer_pv_ubuntud.yaml
apiVersion: v1
kind: PersistentVolume
metadata:name: local-storage-class-waitforfirstconsumer-pv-ubuntud
spec:capacity:storage:  1MivolumeMode: FilesystemaccessModes:- ReadWriteOncepersistentVolumeReclaimPolicy: DeletestorageClassName: local-storage-class-waitforfirstconsumerlocal:path: /tmpnodeAffinity:required:nodeSelectorTerms:- matchExpressions:- key: kubernetes.io/hostnameoperator: Invalues:- ubuntud

PersistentVolumeClaim

和之前例子的区别就是storageClassName选择了延迟绑定的local-storage-class-waitforfirstconsumer。

# local_storage_class_waitforfirstconsumer_pvc_600k.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:name: local-storage-class-waitforfirstconsumer-pvc-600k
spec:resources:requests:storage: 600KiaccessModes:- ReadWriteOncestorageClassName: local-storage-class-waitforfirstconsumer

我们先创建上述组件，然后观察PVC的状态。

kubectl describe persistentvolumeclaims local-storage-class-waitforfirstconsumer-pvc-600k

Name:          local-storage-class-waitforfirstconsumer-pvc-600k
Namespace:     default
StorageClass:  local-storage-class-waitforfirstconsumer
Status:        Pending
Volume:        
Labels:        <none>
Annotations:   <none>
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Used By:       <none>
Events:Type    Reason                Age               From                         Message----    ------                ----              ----                         -------Normal  WaitForFirstConsumer  2s (x2 over 11s)  persistentvolume-controller  waiting for first consumer to be created before binding

可以看到这次PVC没有立即绑定，而是处于Pending状态，且原因是等待第一个使用者触发绑定。

Deployment

# local_deployment_waitforfirstconsumer.yaml
apiVersion: apps/v1
kind: Deployment
metadata:name: local-pv-app-waitforfirstconsumer-deployment
spec:replicas: 1selector:matchLabels:app: local-pv-app-waitforfirstconsumertemplate:metadata:labels:app: local-pv-app-waitforfirstconsumerspec:containers:- name: local-pv-app-waitforfirstconsumerimage: busyboxcommand: ["/bin/sh", "-c", "if [ -f /tempdir/lockfile ] && ! { set -C; 2>/dev/null >/tempdir/lockfile; }; then tail -f /tempdir/lockfile; else exec 3>/tempdir/lockfile; if [ -n \"$POD_NAME\" ]; then name=$POD_NAME; else name=\"unknown\"; fi; while true; do echo \"this is $name.$name write something to lockfile\"; echo \"$name write something to lockfile\" >&3; sleep 1; done; fi"]  volumeMounts:- name: local-pvc-volumemountPath: /tempdirenv:- name: POD_NAMEvalueFrom:fieldRef:fieldPath: metadata.namevolumes:- name: local-pvc-volumepersistentVolumeClaim:claimName: local-storage-class-waitforfirstconsumer-pvc-600kaffinity:nodeAffinity:requiredDuringSchedulingIgnoredDuringExecution:nodeSelectorTerms:- matchExpressions:- key: kubernetes.io/hostnameoperator: Invalues:- ubuntud    

创建完我们查看Pod的状态。

kubectl describe pod local-pv-app-waitforfirstconsumer-deployment-84449895c4-x7ddx

Name:             local-pv-app-waitforfirstconsumer-deployment-84449895c4-x7ddx
Namespace:        default
Priority:         0
Service Account:  default
Node:             ubuntud/172.22.244.197
Start Time:       Wed, 09 Aug 2023 17:07:01 +0000
Labels:           app=local-pv-app-waitforfirstconsumerpod-template-hash=84449895c4
Annotations:      cni.projectcalico.org/containerID: cb10dba20771f872b242bc6284eb9d790565b7f2c1a2fbb096ff1581a73d4de5cni.projectcalico.org/podIP: 10.1.202.206/32cni.projectcalico.org/podIPs: 10.1.202.206/32
Status:           Running
IP:               10.1.202.206
IPs:IP:           10.1.202.206
Controlled By:  ReplicaSet/local-pv-app-waitforfirstconsumer-deployment-84449895c4
Containers:local-pv-app-waitforfirstconsumer:Container ID:  containerd://3fda11a2670236dc37409dd1fd6c5efae36d48bbcf1ce71266f72bd7b0b55b98Image:         busyboxImage ID:      docker.io/library/busybox@sha256:3fbc632167424a6d997e74f52b878d7cc478225cffac6bc977eedfe51c7f4e79Port:          <none>Host Port:     <none>Command:/bin/sh-cif [ -f /tempdir/lockfile ] && ! { set -C; 2>/dev/null >/tempdir/lockfile; }; then tail -f /tempdir/lockfile; else exec 3>/tempdir/lockfile; if [ -n "$POD_NAME" ]; then name=$POD_NAME; else name="unknown"; fi; while true; do echo "this is $name.$name write something to lockfile"; echo "$name write something to lockfile" >&3; sleep 1; done; fiState:          RunningStarted:      Wed, 09 Aug 2023 17:07:04 +0000Ready:          TrueRestart Count:  0Environment:POD_NAME:  local-pv-app-waitforfirstconsumer-deployment-84449895c4-x7ddx (v1:metadata.name)Mounts:/tempdir from local-pvc-volume (rw)/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-52426 (ro)
Conditions:Type              StatusInitialized       True Ready             True ContainersReady   True PodScheduled      True 
Volumes:local-pvc-volume:Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)ClaimName:  local-storage-class-waitforfirstconsumer-pvc-600kReadOnly:   falsekube-api-access-52426:Type:                    Projected (a volume that contains injected data from multiple sources)TokenExpirationSeconds:  3607ConfigMapName:           kube-root-ca.crtConfigMapOptional:       <nil>DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300snode.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:Type    Reason     Age   From               Message----    ------     ----  ----               -------Normal  Scheduled  8s    default-scheduler  Successfully assigned default/local-pv-app-waitforfirstconsumer-deployment-84449895c4-x7ddx to ubuntudNormal  Pulling    8s    kubelet            Pulling image "busybox"Normal  Pulled     5s    kubelet            Successfully pulled image "busybox" in 2.266071612s (2.266078813s including waiting)Normal  Created    5s    kubelet            Created container local-pv-app-waitforfirstconsumerNormal  Started    5s    kubelet            Started container local-pv-app-waitforfirstconsumer

可以看到Pod按清单要求被成功调度到ubuntud上。

kubectl describe persistentvolumeclaims local-storage-class-waitforfirstconsumer-pvc-600k 

Name:          local-storage-class-waitforfirstconsumer-pvc-600k
Namespace:     default
StorageClass:  local-storage-class-waitforfirstconsumer
Status:        Bound
Volume:        local-storage-class-waitforfirstconsumer-pv-ubuntud
Labels:        <none>
Annotations:   pv.kubernetes.io/bind-completed: yespv.kubernetes.io/bound-by-controller: yes
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      1Mi
Access Modes:  RWO
VolumeMode:    Filesystem
Used By:       local-pv-app-waitforfirstconsumer-deployment-84449895c4-x7ddx
Events:Type    Reason                Age                    From                         Message----    ------                ----                   ----                         -------Normal  WaitForFirstConsumer  2m23s (x16 over 6m2s)  persistentvolume-controller  waiting for first consumer to be created before binding

这个Pod使用的PVC也被分配到ubuntud上。

参考资料

• https://www.qikqiak.com/k8strain/storage/local/