Kubernetes金丝雀发布

Canary金丝雀发布

什么是金丝雀发布

• 金丝雀发布也称为灰度发布，是一种软件发布策略
• 主要目的是在将新版本的软件全面推广到生产环境之前，先在一小部分用户或服务器上进行测试和验证，以降低因新版本引入重大问题而对整个系统造成的影响
• 是一种Pod的发布方式。金丝雀发布采取先添加、再删除的方式，保证Pod的总量不低于期望值。并且在更新部分Pod后，暂停更新，当确认新Pod版本运行正常后再进行其他版本的Pod的更新

Canary发布方式

三种发布方式：

• 优先级：head大于cookie大于weight
• 其中header和weight用的最多

基于header（http包头）灰度发布

我们可以看到如果包头中有stage=gray的键值对，就访问新版本，包头中没有那个键值对，就访问旧版本

• 通过Annotaion扩展
• 创建灰度ingress，配置灰度头部key以及value
• 灰度流量验证完毕后，切换正式ingress到新版本
• 之前我们在做升级时可以通过控制器做滚动更新，默认25%利用header可以使升级更为平滑，通过key和value 测试新的业务体系是否有问题

创建2个deployment控制器

#发现没有运行的pod
[root@k8s-master service]# kubectl get pods
No resources found in default namespace.#创建一个deployment控制器，控制器中运行一个pod
[root@k8s-master service]# kubectl create deployment deployment --image myapp:v1 --dry-run=client -o yaml > deployment-v1.yml[root@k8s-master service]# vim deployment-v1.yml
[root@k8s-master service]# cat deployment-v1.yml
apiVersion: apps/v1
kind: Deployment   #指明这是一个deployment控制器
metadata:   #控制器的元数据labels:    #控制器的标签app: deployment1   #控制器的标签name: deployment1   #控制器的名字
spec:  控制器的规格replicas: 1  #pod数量selector:  #pod选择器matchLabels:   #声明的要管理的podapp: myappv1  #标签为app=myappv1的pod会被管理template:    #pod的模板metadata:    #pod的元数据labels:     #pod的标签app: myappv1spec:     #pod的规格containers:- image: myapp:v1name: myappv1#在克隆一份
[root@k8s-master service]# cp deployment-v1.yml deployment-v2.yml
[root@k8s-master service]# vim deployment-v2.yml
[root@k8s-master services]# cat deployment-v2.yml
apiVersion: apps/v1
kind: Deployment  #指明这个一个deployment控制器
metadata:     #控制器的元数据labels:     app: deployment2   #控制器的标签name: deployment2   #控制器的名字
spec:    #控制器的规格replicas: 1    #pod数量selector:     #pod选择器matchLabels:    app: myappv2   #声明需要管理的标签template:    #pod的模板metadata:    #pod的元数据labels:   app: myappv2  #pod的标签spec:   #pod的规格containers:- image: myapp:v2name: myappv2 

[root@k8s-master service]# kubectl apply -f deployment-v1.yml
deployment.apps/deployment1 created
[root@k8s-master service]# kubectl apply -f deployment-v2.yml
deployment.apps/deployment2 created[root@k8s-master service]# kubectl get pods -o wide --show-labels
NAME                           READY   STATUS    RESTARTS   AGE     IP            NODE            NOMINATED NODE   READINESS GATES   LABELS
deployment1-5c47495d84-ds4cl   1/1     Running   0          3m20s   10.244.2.15   k8s-node2.org   <none>           <none>            app=myappv1,pod-template-hash=5c47495d84
deployment2-67cc8c4845-bfnvx   1/1     Running   0          6m49s   10.244.2.14   k8s-node2.org   <none>           <none>            app=myappv2,pod-template-hash=67cc8c4845

创建2个service微服务

[root@k8s-master service]# kubectl expose deployment deployment1 --port 8080 --target-port 80 --dry-run=client -o yaml >> deployment-v1.yml[root@k8s-master service]# kubectl expose deployment deployment2 --port 8080 --target-port 80 --dry-run=client -o yaml >> deployment-v2.yml

[root@k8s-master service]# vim deployment-v1.yml
[root@k8s-master service]# cat deployment-v1.yml
apiVersion: apps/v1
kind: Deployment
metadata:labels:app: deployment1name: deployment1
spec:replicas: 1selector:matchLabels:app: myappv1template:metadata:labels:app: myappv1spec:containers:- image: myapp:v1name: myappv1---
apiVersion: v1
kind: Service   #指明这是一个service微服务
metadata:   #微服务元数据labels:  app: deployment1  #微服务标签 name: deployment1   #微服务的名字
spec:    #微服务的规格ports:   #是一个端口列表，用于描述service应该监听的端口以及如何将流量转发给pod- port: 8080   #service微服务监听的端口号protocol: TCP   #使用的协议，这里是TCPtargetPort: 80   #pod上应用程序监听的端口selector:    #标签选择器，用于确定哪些pods应该被这个service管理app: myappv1  #pod标签为app=myappv1的被该service管理

[root@k8s-master service]# vim deployment-v2.yml
[root@k8s-master service]# cat deployment-v2.yml
apiVersion: apps/v1
kind: Deployment
metadata:labels:app: deployment2name: deployment2
spec:replicas: 1selector:matchLabels:app: myappv2template:metadata:labels:app: myappv2spec:containers:- image: myapp:v2name: myappv2---
apiVersion: v1
kind: Service
metadata:labels:app: deployment2name: deployment2
spec:ports:- port: 8080protocol: TCPtargetPort: 80selector:app: myappv2

[root@k8s-master service]# kubectl apply -f deployment-v1.yml
deployment.apps/deployment1 unchanged
service/deployment1 created
[root@k8s-master service]# kubectl apply -f deployment-v2.yml
deployment.apps/deployment2 unchanged
service/deployment2 created

[root@k8s-master service]# kubectl get pods -o wide --show-labels
NAME                           READY   STATUS    RESTARTS   AGE   IP            NODE            NOMINATED NODE   READINESS GATES   LABELS
deployment1-5c47495d84-ds4cl   1/1     Running   0          15m   10.244.2.15   k8s-node2.org   <none>           <none>            app=myappv1,pod-template-hash=5c47495d84
deployment2-67cc8c4845-bfnvx   1/1     Running   0          18m   10.244.2.14   k8s-node2.org   <none>           <none>            app=myappv2,pod-template-hash=67cc8c4845[root@k8s-master service]# kubectl get svc -o wide
NAME          TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE    SELECTOR
deployment1   ClusterIP   10.108.37.167   <none>        8080/TCP   3m8s   app=myappv1
deployment2   ClusterIP   10.96.202.197   <none>        8080/TCP   3m2s   app=myappv2
kubernetes    ClusterIP   10.96.0.1       <none>        443/TCP    27d    <none>[root@k8s-master service]# kubectl describe svc deployment1
Name:              deployment1
Namespace:         default
Labels:            app=deployment1
Annotations:       <none>
Selector:          app=myappv1
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.108.37.167
IPs:               10.108.37.167
Port:              <unset>  8080/TCP
TargetPort:        80/TCP
Endpoints:         10.244.2.15:80
Session Affinity:  None
Events:            <none>[root@k8s-master service]# kubectl describe svc deployment2
Name:              deployment2
Namespace:         default
Labels:            app=deployment2
Annotations:       <none>
Selector:          app=myappv2
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.96.202.197
IPs:               10.96.202.197
Port:              <unset>  8080/TCP
TargetPort:        80/TCP
Endpoints:         10.244.2.14:80
Session Affinity:  None
Events:            <none>

• 创建ingress1.yml

[root@k8s-master service]# kubectl create ingress ingress1 --class nginx --rule='/=deployment1:8080' --dry-run=client -o yaml > ingress1.yml[root@k8s-master service]# vim ingress1.yml
[root@k8s-master service]# cat ingress1.yml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:name: ingress1
spec:ingressClassName: nginxrules:- http:paths:- backend:service:name: deployment1port:number: 8080path: /pathType: Prefix

• 创建ingress2.yml

[root@k8s-master service]# kubectl create ingress ingress2 --class nginx --rule='/=deployment2:8080' --dry-run=client -o yaml > ingress2.yml[root@k8s-master service]# vim ingress2.yml
[root@k8s-master service]# cat ingress2.yml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:annotations:nginx.ingress.kubernetes.io/canary: "true"nginx.ingress.kubernetes.io/canary-by-header: "name"   #键nginx.ingress.kubernetes.io/canary-by-header-value: "huazi"  #值name: ingress2
spec:ingressClassName: nginxrules:- http:paths:- backend:service:name: deployment2port:number: 8080path: /pathType: Prefix

[root@k8s-master service]# kubectl apply -f ingress1.yml
ingress.networking.k8s.io/ingress1 created
[root@k8s-master service]# kubectl apply -f ingress2.yml
ingress.networking.k8s.io/ingress2 created

[root@k8s-master service]# kubectl describe ingress ingress1
Name:             ingress1
Labels:           <none>
Namespace:        default
Address:          172.25.254.10
Ingress Class:    nginx
Default backend:  <default>
Rules:Host        Path  Backends----        ----  --------*/   deployment1:8080 (10.244.2.15:80)
Annotations:  <none>
Events:Type    Reason  Age                From                      Message----    ------  ----               ----                      -------Normal  Sync    12m (x2 over 13m)  nginx-ingress-controller  Scheduled for sync[root@k8s-master service]# kubectl describe ingress ingress2
Name:             ingress2
Labels:           <none>
Namespace:        default
Address:          172.25.254.10
Ingress Class:    nginx
Default backend:  <default>
Rules:Host        Path  Backends----        ----  --------*/   deployment2:8080 (10.244.2.14:80)
Annotations:  nginx.ingress.kubernetes.io/canary: truenginx.ingress.kubernetes.io/canary-by-header: namenginx.ingress.kubernetes.io/canary-by-header-value: huazi
Events:Type    Reason  Age                   From                      Message----    ------  ----                  ----                      -------Normal  Sync    2m21s (x2 over 3m3s)  nginx-ingress-controller  Scheduled for sync

基于权重的金丝雀发布

• 通过Annotaion拓展
• 创建灰度ingress，配置灰度权重以及总权重
• 灰度流量验证完毕后，切换正式ingress到新版本

#删掉基于header的ingress
[root@k8s-master service]# kubectl delete -f ingress2.yml
ingress.networking.k8s.io "ingress2" deleted

[root@k8s-master service]# vim ingress2.yml
[root@k8s-master service]# cat ingress2.yml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:annotations:nginx.ingress.kubernetes.io/canary: "true"nginx.ingress.kubernetes.io/canary-weight: "10"    #10/100=10%，将有10%的流量打到新版本上nginx.ingress.kubernetes.io/canary-weight-total: "100"name: ingress2
spec:ingressClassName: nginxrules:- http:paths:- backend:service:name: deployment2port:number: 8080path: /pathType: Prefix

[root@k8s-master service]# kubectl apply -f ingress2.yml
ingress.networking.k8s.io/ingress2 created

[root@k8s-master service]# kubectl describe ingress ingress1
Name:             ingress1
Labels:           <none>
Namespace:        default
Address:          172.25.254.10
Ingress Class:    nginx
Default backend:  <default>
Rules:Host        Path  Backends----        ----  --------*/   deployment1:8080 (10.244.2.15:80)
Annotations:  <none>
Events:Type    Reason  Age                From                      Message----    ------  ----               ----                      -------Normal  Sync    27m (x2 over 27m)  nginx-ingress-controller  Scheduled for sync[root@k8s-master service]# kubectl describe ingress ingress2
Name:             ingress2
Labels:           <none>
Namespace:        default
Address:          172.25.254.10
Ingress Class:    nginx
Default backend:  <default>
Rules:Host        Path  Backends----        ----  --------*/   deployment2:8080 (10.244.2.14:80)
Annotations:  nginx.ingress.kubernetes.io/canary: truenginx.ingress.kubernetes.io/canary-weight: 10nginx.ingress.kubernetes.io/canary-weight-total: 100
Events:Type    Reason  Age                From                      Message----    ------  ----               ----                      -------Normal  Sync    34s (x2 over 44s)  nginx-ingress-controller  Scheduled for sync

• 写检测脚本

[root@harbor ~]# vim check.sh
#!/bin/bash
v1=0
v2=0
for ((i=0;i<100;i++))
doresponse=`curl -s 172.25.254.50 | grep -c v1`v1=`expr $v1 + $response`v2=`expr $v2 + 1 - $response`
done
echo "v1:$v1  v2:$v2"

[root@harbor ~]# bash check.sh
v1:89  v2:11
[root@harbor ~]# bash check.sh
v1:87  v2:13
#我们发现比例接近于9:1

当我们增加权重后

[root@k8s-master service]# vim ingress2.yml
[root@k8s-master service]# cat ingress2.yml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:annotations:nginx.ingress.kubernetes.io/canary: "true"nginx.ingress.kubernetes.io/canary-weight: "20"nginx.ingress.kubernetes.io/canary-weight-total: "100"name: ingress2
spec:ingressClassName: nginxrules:- http:paths:- backend:service:name: deployment2port:number: 8080path: /pathType: Prefix

[root@k8s-master service]# kubectl apply -f ingress2.yml
ingress.networking.k8s.io/ingress2 configured

[root@harbor ~]# bash check.sh
v1:76  v2:24
[root@harbor ~]# bash check.sh
v1:79  v2:21
#我们发现比例进阶于8:2