【Kubernetes】第二十篇 - k8s 污点和容忍度

一，前言

上一篇，介绍了 k8s ConfigMap 管理服务环境变量；

本篇，介绍 k8s 污点和容忍度；

---

二，污点与容忍度介绍

通过污点和容忍度配置可以干预 Pod 部署到特定的节点；

比如：
不想让某些服务、deploy、pod 部署到某台机器上；
专门负责部署 mysql 的机器，可以设置污点默认不能部署其他服务；

污点和容忍度

• 在 Kubernetes 中， Pod 被部署到 Node 上面去的规则和逻辑是由 Kubernetes 的调度组件根据 Node 的剩余资源，地位，以及其他规则自动选择调度的
• 但前端和后端往往服务器资源的分配都是不均衡的，甚至有的服务只能让特定的服务器来跑
• 在这种情况下，我们选择自动调度是不均衡的，就需要人工去干预匹配选择规则了
• 这时候，就需要在给 Node 添加一个叫做污点的东西，以确保 Node 不被 Pod 调度到
• 当你给 Node 设置一个污点后，除非给 Pod 设置一个相对应的容忍度，否则 Pod 才能被调度上去。这也就是污点和容忍的来源
• 污点的格式是 key=value，可以自定义自己的内容，就像是一组 Tag 一样
• Node_Name 为要添加污点的 node 名称
• key 和 value 为一组键值对，代表一组标示标签
• NoSchedule 则为不被调度的意思，和它同级别的还有其他的值：PreferNoSchedule 和 NoExecute

---

三，清理环境

先清理一下现有环境，释放出资源：

[root@k8s-master deployment]# kubectl get pods
NAME                      READY   STATUS             RESTARTS   AGE
pay-v1-655587b6f5-gv8hc   1/1     Running            0          24h
user-v1-9f4d589cc-rdmnz   1/1     Running            0          10m
v4-57b4cf7fd9-zcl45       0/1     ImagePullBackOff   0          5d23h
v4-fb4cd75f5-bf2pf        0/1     ImagePullBackOff   0          40h// 删掉 pay-v1 的部署(pay-v1 的容器会被干掉)
[root@k8s-master deployment]# kubectl delete deploy pay-v1
deployment.apps "pay-v1" deleted[root@k8s-master deployment]# kubectl get pods
NAME                      READY   STATUS             RESTARTS   AGE
user-v1-9f4d589cc-rdmnz   1/1     Running            0          13m
v4-57b4cf7fd9-zcl45       0/1     ImagePullBackOff   0          5d23h
v4-fb4cd75f5-bf2pf        0/1     ImagePullBackOff   0          40h

---

四，设置污点

为 k8s-node 设置污点：

kubectl taint nodes [Node_Name] [key]=[value]:NoSchedule
// 添加污点-k8s-nodes 不部署 pay-v1
[root@k8s-master deployment]# kubectl taint nodes k8s-node pay-v1=true:NoSchedule
node/k8s-node tainted//查看污点
[root@k8s-master deployment]# kubectl describe node k8s-node
Name:               k8s-node
Roles:              <none>
Labels:             beta.kubernetes.io/arch=amd64beta.kubernetes.io/os=linuxkubernetes.io/arch=amd64kubernetes.io/hostname=k8s-nodekubernetes.io/os=linux
Annotations:        flannel.alpha.coreos.com/backend-data: {"VNI":1,"VtepMAC":"96:c0:15:7d:c1:a9"}flannel.alpha.coreos.com/backend-type: vxlanflannel.alpha.coreos.com/kube-subnet-manager: trueflannel.alpha.coreos.com/public-ip: 172.17.178.106kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.socknode.alpha.kubernetes.io/ttl: 0volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Wed, 22 Dec 2021 00:41:20 +0800
// 以下就是污点信息-如果部署的名称是 pay-v1，k8s-node 不参与调度
Taints:             pay-v1=true:NoSchedule
Unschedulable:      false
Lease:HolderIdentity:  k8s-nodeAcquireTime:     <unset>RenewTime:       Wed, 05 Jan 2022 15:35:38 +0800
Conditions:Type                 Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message----                 ------  -----------------                 ------------------                ------                       -------NetworkUnavailable   False   Fri, 24 Dec 2021 17:43:49 +0800   Fri, 24 Dec 2021 17:43:49 +0800   FlannelIsUp                  Flannel is running on this nodeMemoryPressure       False   Wed, 05 Jan 2022 15:32:38 +0800   Fri, 24 Dec 2021 18:22:47 +0800   KubeletHasSufficientMemory   kubelet has sufficient memory availableDiskPressure         False   Wed, 05 Jan 2022 15:32:38 +0800   Fri, 24 Dec 2021 18:22:47 +0800   KubeletHasNoDiskPressure     kubelet has no disk pressurePIDPressure          False   Wed, 05 Jan 2022 15:32:38 +0800   Fri, 24 Dec 2021 18:22:47 +0800   KubeletHasSufficientPID      kubelet has sufficient PID availableReady                True    Wed, 05 Jan 2022 15:32:38 +0800   Fri, 24 Dec 2021 18:22:47 +0800   KubeletReady                 kubelet is posting ready status
Addresses:InternalIP:  172.17.178.106Hostname:    k8s-node
Capacity:cpu:                2ephemeral-storage:  41152812Kihugepages-1Gi:      0hugepages-2Mi:      0memory:             951856Kipods:               110
Allocatable:cpu:                2ephemeral-storage:  37926431477hugepages-1Gi:      0hugepages-2Mi:      0memory:             849456Kipods:               110
System Info:Machine ID:                 20211123171600472607520636465043System UUID:                71F14756-1816-4DFF-86DF-5129F0234463Boot ID:                    336150f9-ea6b-4de6-b4f5-c06967b5b344Kernel Version:             3.10.0-1160.45.1.el7.x86_64OS Image:                   CentOS Linux 7 (Core)Operating System:           linuxArchitecture:               amd64Container Runtime Version:  docker://20.10.12Kubelet Version:            v1.20.4Kube-Proxy Version:         v1.20.4
PodCIDR:                      10.244.1.0/24
PodCIDRs:                     10.244.1.0/24
Non-terminated Pods:          (6 in total)Namespace                   Name                                         CPU Requests  CPU Limits  Memory Requests  Memory Limits  AGE---------                   ----                                         ------------  ----------  ---------------  -------------  ---default                     user-v1-9f4d589cc-rdmnz                      0 (0%)        0 (0%)      0 (0%)           0 (0%)         17mdefault                     v4-57b4cf7fd9-zcl45                          0 (0%)        0 (0%)      0 (0%)           0 (0%)         5d23hdefault                     v4-fb4cd75f5-bf2pf                           0 (0%)        0 (0%)      0 (0%)           0 (0%)         40hingress-nginx               ingress-nginx-controller-6b6497d95d-9j7qn    100m (5%)     0 (0%)      90Mi (10%)       0 (0%)         13dkube-system                 kube-flannel-ds-ckhq8                        100m (5%)     100m (5%)   50Mi (6%)        50Mi (6%)      14dkube-system                 kube-proxy-sp6r2                             0 (0%)        0 (0%)      0 (0%)           0 (0%)         14d
Allocated resources:(Total limits may be over 100 percent, i.e., overcommitted.)Resource           Requests     Limits--------           --------     ------cpu                200m (10%)   100m (5%)memory             140Mi (16%)  50Mi (6%)ephemeral-storage  0 (0%)       0 (0%)hugepages-1Gi      0 (0%)       0 (0%)hugepages-2Mi      0 (0%)       0 (0%)
Events:              <none>

生效 pay-v1 部署配置：deployment-pay-v1.yaml

[root@k8s-master deployment]# kubectl apply -f deployment-pay-v1.yaml
deployment.apps/pay-v1 created[root@k8s-master deployment]# kubectl get pods
NAME                      READY   STATUS             RESTARTS   AGE
pay-v1-655587b6f5-k26lx   0/1     Pending            0          19s
user-v1-9f4d589cc-rdmnz   1/1     Running            0          20m
v4-57b4cf7fd9-zcl45       0/1     ImagePullBackOff   0          5d23h
v4-fb4cd75f5-bf2pf        0/1     ImagePullBackOff   0          40h

新创建的pod：pay-v1-655587b6f5-k26lx 处于 Pending 状态：

[root@k8s-master deployment]# kubectl describe pod pay-v1-655587b6f5-k26lx
Name:           pay-v1-655587b6f5-k26lx
Namespace:      default
Priority:       0
Node:           <none>
Labels:         app=pay-v1pod-template-hash=655587b6f5
Annotations:    <none>
Status:         Pending
IP:             
IPs:            <none>
Controlled By:  ReplicaSet/pay-v1-655587b6f5
Containers:nginx:Image:        nginx:payPort:         80/TCPHost Port:    0/TCPEnvironment:  <none>Mounts:/var/run/secrets/kubernetes.io/serviceaccount from default-token-q4qxd (ro)
Conditions:Type           StatusPodScheduled   False 
Volumes:default-token-q4qxd:Type:        Secret (a volume populated by a Secret)SecretName:  default-token-q4qxdOptional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300snode.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:Type     Reason            Age               From               Message----     ------            ----              ----               -------Warning  FailedScheduling  3s (x3 over 75s)  default-scheduler  0/2 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 1 node(s) had taint {pay-v1: true}, that the pod didn't tolerate.

报错了，调度失败，共 2 个节点，0 个可用， k8s-node 污点效果生效

---

五，设置容忍度

为 Pod 设置容忍度：

• 想让 Pod 被调度过去，需要在 Pod 一侧添加相同的容忍度才能被调度到
• 给 Pod 设置一组容忍度，以匹配对应的 Node 的污点
• key 和 value 是你配置 Node 污点的 key 和 value
• effect 是 Node 污点的调度效果，和 Node 的设置项也是匹配的
• operator 是运算符，equal 代表只有 key 和 value 相等才算数。当然也可以配置 exists ，代表只要 key 存在就匹配，不需要校验 value 的值

修改前： deployment-pay-v1.yaml

apiVersion: apps/v1  #API版本号
kind: Deployment     #资源类型部署
metadata:name: pay-v1      #资源名称
spec:selector:matchLabels:app: pay-v1   #告诉deployment根据规则匹配相应的Pod进行控制和管理，matchLabels字段匹配Pod的label值replicas: 1        #声明Pod副本的数量template:metadata:labels:app: pay-v1 #Pod名称spec:            #描述Pod内的容器信息containers:- name: nginx  #容器的名称image: nginx:pay #镜像ports:- containerPort: 80 #容器内映射的端口

修改后：

apiVersion: apps/v1  #API版本号
kind: Deployment     #资源类型部署
metadata:name: pay-v1      #资源名称
spec:selector:matchLabels:app: pay-v1   #告诉deployment根据规则匹配相应的Pod进行控制和管理，matchLabels字段匹配Pod的label值replicas: 1        #声明Pod副本的数量template:metadata:labels:app: pay-v1 #Pod名称spec:            #描述Pod内的容器信息tolerations:- key: "pay-v1"value: "true"operator: "Equal"effect: "NoSchedule"containers:- name: nginx  #容器的名称image: nginx:pay #镜像ports:- containerPort: 80 #容器内映射的端口

生效配置：

[root@k8s-master deployment]# kubectl apply -f deployment-pay-v1.yaml
deployment.apps/pay-v1 configured// pay-v1的 pod 成功 Running
[root@k8s-master deployment]# kubectl get pods
NAME                      READY   STATUS             RESTARTS   AGE
pay-v1-6cd6d4cc78-sgvnv   1/1     Running            0          45s
user-v1-9f4d589cc-rdmnz   1/1     Running            0          86m
v4-57b4cf7fd9-zcl45       0/1     ImagePullBackOff   0          6d1h
v4-fb4cd75f5-bf2pf        0/1     ImagePullBackOff   0          42h// 容忍污点，部署到了k8s-node上
[root@k8s-master deployment]# kubectl describe pod pay-v1-6cd6d4cc78-sgvnv
Name:         pay-v1-6cd6d4cc78-sgvnv
Namespace:    default
Priority:     0
Node:         k8s-node/172.17.178.106

---

六，修改 node 节点的污点

先删除 pay-v1 的部署

[root@k8s-master deployment]# kubectl delete deploy pay-v1
deployment.apps "pay-v1" deleted

// 之前是 pay-v1=true ，这次是 pay-v1=1
[root@k8s-master deployment]# kubectl taint nodes k8s-node pay-v1=1:NoSchedule --overwrite
node/k8s-node modified[root@k8s-master deployment]# kubectl apply -f deployment-pay-v1.yaml
deployment.apps/pay-v1 created// pending 了
[root@k8s-master deployment]# kubectl get pods
NAME                      READY   STATUS             RESTARTS   AGE
pay-v1-6cd6d4cc78-shfp9   0/1     Pending            0          20s
user-v1-9f4d589cc-rdmnz   1/1     Running            0          91m
v4-57b4cf7fd9-zcl45       0/1     ImagePullBackOff   0          6d1h
v4-fb4cd75f5-bf2pf        0/1     ImagePullBackOff   0          42h

为什么是 pending 呢？因为污点不被允许；

这时因为，在 yaml 中的配置是：

    spec:            #描述Pod内的容器信息tolerations:- key: "pay-v1"value: "true"operator: "Equal"effect: "NoSchedule"

key 和 value 都要相等才可以，但 value 是 1，不相等；

修改：改为存在 key 即可，不关心 value 值

    spec:            #描述Pod内的容器信息tolerations:- key: "pay-v1"operator: "Exists"effect: "NoSchedule"[root@k8s-master deployment]# vi deployment-pay-v1.yamlapiVersion: apps/v1  #API版本号
kind: Deployment     #资源类型部署
metadata:name: pay-v1      #资源名称
spec:selector:matchLabels:app: pay-v1   #告诉deployment根据规则匹配相应的Pod进行控制和管理，matchLabels字段匹配Pod的label值replicas: 1        #声明Pod副本的数量template:metadata:labels:app: pay-v1 #Pod名称spec:            #描述Pod内的容器信息tolerations:- key: "pay-v1"operator: "Exists"effect: "NoSchedule"containers:- name: nginx  #容器的名称image: nginx:pay #镜像ports:- containerPort: 80 #容器内映射的端口[root@k8s-master deployment]# kubectl apply -f deployment-pay-v1.yaml
deployment.apps/pay-v1 configured// 成功 Running
[root@k8s-master deployment]# kubectl get pods
NAME                      READY   STATUS             RESTARTS   AGE
pay-v1-6d6cdc544b-fbhdd   1/1     Running            0          28s
user-v1-9f4d589cc-rdmnz   1/1     Running            0          97m
v4-57b4cf7fd9-zcl45       0/1     ImagePullBackOff   0          6d1h
v4-fb4cd75f5-bf2pf        0/1     ImagePullBackOff   0          42h

即匹配规则可以自由指定，容忍规则也可以自由指定

七，删除 Node 的污点

kubectl taint nodes k8s-node pay-v1-  // 最后的 - 就是删除的意思[root@k8s-master ~]# kubectl taint nodes k8s-node pay-v1-
node/k8s-node untainted  // 在k8s-node节点上取消污点

取消污点后，就可以随便部署了

八，如何在 master 上布署 pod

master 之所以不能部署 pod，是因为有污点

// 添加污点
[root@k8s-master ~]# kubectl taint nodes k8s-node pay-v1=true:NoSchedule --overwrite
node/k8s-node modified// 删掉部署
[root@k8s-master ~]# kubectl delete deploy pay-v1
deployment.apps "pay-v1" deleted// 查看k8s-master污点信息
[root@k8s-master ~]# kubectl describe node k8s-master
Name:               k8s-master
Taints:             node-role.kubernetes.io/master:NoSchedule
Unschedulable:      false

修改 pay-v1 配置，将 tolerations 容忍度，修改为 node-role.kubernetes.io/master，使之匹配

[root@k8s-master deployment]# vi deployment-pay-v1.yaml apiVersion: apps/v1  #API版本号
kind: Deployment     #资源类型部署
metadata:name: pay-v1      #资源名称
spec:selector:matchLabels:app: pay-v1   #告诉deployment根据规则匹配相应的Pod进行控制和管理，matchLabels字段匹配Pod的label值replicas: 1        #声明Pod副本的数量template:metadata:labels:app: pay-v1 #Pod名称spec:            #描述Pod内的容器信息
+     tolerations:
+     - key: "node-role.kubernetes.io/master"
+       operator: "Exists"
+       effect: "NoSchedule"containers:- name: nginx  #容器的名称image: nginx:pay #镜像ports:- containerPort: 80 #容器内映射的端口// 部署
[root@k8s-master deployment]#  kubectl apply -f deployment-pay-v1.yaml 
deployment.apps/pay-v1 created

查看部署 pod:

[root@k8s-master deployment]# kubectl get pods
NAME                      READY   STATUS             RESTARTS   AGE
pay-v1-6db6455b8-np2hw    1/1     Running            0          37s
user-v1-9f4d589cc-rdmnz   1/1     Running            0          18h
v4-57b4cf7fd9-zcl45       0/1     ImagePullBackOff   0          6d17h
v4-fb4cd75f5-bf2pf        0/1     ImagePullBackOff   0          2d10h[root@k8s-master deployment]# kubectl describe pod pay-v1-6db6455b8-np2hw
Name:         pay-v1-6db6455b8-np2hw
Namespace:    default
Priority:     0
Node:         k8s-master/172.17.178.105

可以发现，pod 被部署到了 master 节点上

---

九，结尾

本篇，介绍了 k8s 污点和容忍度；

下一篇，待定；