k8s笔记29--使用kyverno提高运维效率

介绍

Kyverno是一个云原生的策略引擎，它最初是为k8s构建的，现在也可以在k8s集群之外用作统一的策略语言。其允许平台工程师自动化安全、合规和关于验证的最佳实践，并为相关团队提供安全的自助服务。
在 K8S的复杂生态系统中，确保集群资源的合规性和安全性是一项极具挑战的任务。Kyverno 作为一个强大的k8s策略引擎，为我们提供了一种有效的解决方案。本文将深入探讨 Kyverno 的核心原理、经典案例以及使用过程中的注意事项。

原理

Kyverno 通过 Kubernetes 的准入控制器（Admission Controller）机制来实现策略的执行。当用户向 Kubernetes API Server 提交资源创建、更新或删除请求时，准入控制器会拦截这些请求，并将其发送给 Kyverno 进行策略检查。
Kyverno 的策略以自定义资源定义（CRD）的形式存在，用户可以通过编写 YAML 文件来定义各种策略规则。这些规则可以基于资源的类型、名称、标签选择器等属性进行匹配，并执行相应的操作，如拒绝请求、修改资源或添加注释等。
例如，我们可以定义一个策略，要求所有新建的 Pod 必须包含特定的标签。当用户尝试创建一个不包含该标签的 Pod 时，Kyverno 会根据策略规则拒绝这个请求，从而确保集群中所有 Pod 都符合我们的标签规范。

kyverno逻辑架构图：
https://kyverno.io/docs/introduction/how-kyverno-works/

策略和规则的运行方式：
https://kyverno.io/docs/kyverno-policies/
如下图所示，一个Policy可以包含多个Rule，每个规则必须包含一个Match(Exclude可选)和validate,mutate,generate,verifyImages等选项之一。
策略可以定义为集群层面的ClusterPolicy，也可以定义为命名空间层面的Policy。

安装

安装文档：https://kyverno.io/docs/installation/
可以直接通过yaml的方式安装，也可以通过helm的方式安装
yaml安装方法:

kubectl create -f https://github.com/kyverno/kyverno/releases/download/v1.13.0/install.yaml

helm安装方法:

单机版本：
helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update
helm install kyverno kyverno/kyverno -n kyverno --create-namespace高可用版本：
helm install kyverno kyverno/kyverno -n kyverno --create-namespace \
--set admissionController.replicas=3 \
--set backgroundController.replicas=3 \
--set cleanupController.replicas=3 \
--set reportsController.replicas=3

安装成功会有如下4个running的deploy

$ kubectl -n kyverno get deploy
NAME                            READY   UP-TO-DATE   AVAILABLE   AGE
kyverno-admission-controller    1/1     1            1           24d
kyverno-background-controller   1/1     1            1           24d
kyverno-cleanup-controller      1/1     1            1           24d
kyverno-reports-controller      1/1     1            1           24d

注意：不同版本的kyverno支持的k8s版本有限，安装前可以在官方文档查看版本支持信息

应用场景

自动添加DR的outlierDetection
火山迁移期间，新建DR的时候会默认添加outlierDetection，确保优先网格内服务的稳定性
使用一下策略后，新建DR，如果没有trafficPolicy属性则会自动补充策略中的内容。
策略内容

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:name: general-namespace-add-workload-dr-outlier-detection
spec:admission: truebackground: truerules:- exclude:all:- resources:namespaces:- quicksilver- appsvrmatch:all:- resources:kinds:- networking.istio.io/v1beta1/DestinationRulemutate:patchStrategicMerge:spec:+(trafficPolicy):outlierDetection:baseEjectionTime: 60sconsecutive5xxErrors: 0consecutiveLocalOriginFailures: 10interval: 10smaxEjectionPercent: 80splitExternalLocalOriginErrors: truename: general-namespace-add-workload-dr-outlier-detection

自动修正测试环境pod资源

测试环境主动限制指定命名空间的服务资源request值，可以显著降低业务request过多空资源

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:name: ebc-default-cpu-resources
spec:background: truerules:- name: default-cpu-resourcesmatch:any:- resources:kinds:- Podnamespaces:- ebc-aladdinsite- ebc-bpmexclude:any:- resources:kinds:- Podselector:matchLabels:app: nacos- resources:kinds:- Podselector:matchLabels:app.kubernetes.io/name: ebc-ldapmutate:patchStrategicMerge:spec:containers:- (name): "*"resources:requests:cpu: "100m"memory: "500Mi"limits:cpu: "2000m"memory: "4000Mi"- name: set-resources-init-containersmatch:any:- resources:kinds:- Podnamespaces:- ebc-aladdinsite- ebc-bpmpreconditions:all:- key: "{{ request.object.spec.initContainers || '' }}"operator: NotEqualsvalue: ""mutate:patchStrategicMerge:spec:initContainers:- (name): "*"resources:requests:cpu: "100m"memory: "200Mi"limits:cpu: "1000m"memory: "2000Mi"

也可以通过如下方式设置资源请求和限制

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:name: test-kyverno-set-resource-limits
spec:validationFailureAction: enforcerules:- name: set-cpu-memory-limitsmatch:resources:kinds:- Deploymentnamespaces:- test-kyverno mutate:patchesJson6902:- op: addpath: /spec/template/spec/containers/0/resourcesvalue:requests:cpu: "100m"memory: "128Mi"limits:cpu: "500m"memory: "512Mi"

强制 Pod 标签

确保所有部署到集群中的 Pod 都带有特定的标签，用于资源管理和监控。

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:name: test-kyverno-enforce-pod-labels
spec:validationFailureAction: enforcerules:- name: check-pod-labelsmatch:resources:kinds:- Podnamespaces:- test-kyvernovalidate:message: "Pod must have 'environment' and 'app' labels."pattern:metadata:labels:environment: "?*"app: "?*"

此时创建一个缺少label的deploy就会报错, 加上 environment 标签后就可以正常访创建了

$ vim test-nginx-with-label.yaml
apiVersion: apps/v1
kind: Deployment
metadata:name: test2-deploymentnamespace: test-kyvernolabels:app: test2# environment: prod
spec:replicas: 1selector:matchLabels:app: test2# environment: prodtemplate:metadata:creationTimestamp: nulllabels:app: test2# environment: prodspec:containers:- name: test2image: nginx:1.23ports:- containerPort: 80protocol: TCPresources: {}$ kubectl apply -f test-nginx-with-label.yaml 
The Deployment "test2-deployment" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app":"test2"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable

限制容器镜像来源

为了保障安全，只允许从特定的镜像仓库拉取容器镜像

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:name: test-kyverno-restrict-image-registry
spec:validationFailureAction: enforcerules:- name: check-image-registrymatch:resources:kinds:- Podnamespaces:- test-kyverno validate:message: "Only images from 'my-allowed-registry.com' are allowed."pattern:spec:containers:- image: my-allowed-registry.com/*

此时创建非my-allowed-registry.com 镜像的pod会报错

$ vim test-nginx-with-image-registry.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:name: test2-deploymentnamespace: test-kyvernolabels:app: test2
spec:replicas: 1selector:matchLabels:app: test2template:metadata:labels:app: test2spec:containers:- name: test2image: nginx:1.23# image: my-allowed-registry.com/nginx:1.23ports:- containerPort: 80protocol: TCPresources: {}$ kubectl apply -f test-nginx-with-image-registry.yaml 
Error from server: error when creating "test-nginx-with-image-registry.yaml": admission webhook "validate.kyverno.svc-fail" denied the request: resource Deployment/test-kyverno/test2-deployment was blocked due to the following policies test-kyverno-restrict-image-registry:autogen-check-image-registry: 'validation error: Only images from ''my-allowed-registry.com''are allowed. rule autogen-check-image-registry failed at path /spec/template/spec/containers/0/image/'

禁止特权容器

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:name: test-kyverno-prohibit-privileged-containers
spec:validationFailureAction: enforcerules:- name: check-privilegedmatch:resources:kinds:- Podnamespaces:- test-kyverno validate:message: "Privileged containers are not allowed."pattern:spec:containers:- privileged: false

开启特权就会报错

$ vim test-nginx-with-prohibit-privileged-containers.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:name: test2-deploymentnamespace: test-kyvernolabels:app: test2
spec:replicas: 1selector:matchLabels:app: test2template:metadata:labels:app: test2spec:containers:- name: test2image: nginx:1.23ports:- containerPort: 80protocol: TCPresources: {}securityContext:privileged: true$ kubectl apply -f test-nginx-with-prohibit-privileged-containers.yaml 
Error from server: error when creating "test-nginx-with-prohibit-privileged-containers.yaml": admission webhook "validate.kyverno.svc-fail" denied the request: resource Deployment/test-kyverno/test2-deployment was blocked due to the following policies test-kyverno-prohibit-privileged-containers:autogen-check-privileged: 'validation error: Privileged containers are not allowed.rule autogen-check-privileged failed at path /spec/template/spec/containers/0/privileged/'

其它潜在场景

1. 新建命名空间的时候创建对应的configmap
2. 新建命名空间的时候创建对应的resource quote
3. 给deploy添加默认的环境变量
4. 限制pod默认安全策略

注意事项

1. 策略优先级
   当存在多个策略时，需要注意策略的优先级设置。Kyverno 按照策略的名称顺序进行匹配和执行，因此需要合理命名策略以确保正确的执行顺序。
2. 资源匹配规则
   在编写策略时，要仔细定义资源的匹配规则，避免误判或漏判。例如，使用通配符时要确保其范围符合预期。
3. 测试环境
   在将策略应用到生产环境之前，务必在测试环境中进行全面且充分的测试，确保策略的正确性与有效性，防止因策略错误导致业务中断。
4. 性能影响
   尽管Kyverno的设计旨在实现高效运行，但大量复杂的策略仍可能对Kubernetes API Server的性能产生一定影响。因此，需要定期评估和优化策略，以保障系统的稳定运行。

说明

环境：
k8s: v1.30.0
kyverno: v1.11.1
参考文档：
https://kyverno.io/docs/introduction/#quick-start-guides
https://github.com/kyverno/kyverno/