k8s 1.28.2 集群部署 ingress 1.11.1 包含 admission-webhook

• ingress 官方 yaml 文件：deploy.yaml
• 基于官方 yaml 文件做了一些修改
  • 官方的 svc 是 ClusterIP 和 LoadBalancer，我这边把 LoadBalancer 改成 NodePort
  • 官方的 yaml 没有给 deployment 配置副本数，我这边改成双副本了
    • 增加了亲和性和反亲和，把 controller 绑定到固定的两个节点，方便前面加反代
  • 官方的镜像是 registry.k8s.io 仓库，国内咱们也不好拉，这边改用 k8s.m.daocloud.io 国内的 daocloud 的仓库
  • 开启 admission-webhook 需要自己生成 tls 证书，官方的模板有名字定义了，sercret 的名字是 ingress-nginx-admission
• 开启了 admission-webhook 功能，这个功能类似于 nginx -s reload 之前去执行 nginx -t 检查配置文件是否正确，如果 admission-webhook 发现配置不正确就不会更新 ingress 的配置，就不会因为错误的配置导致 ingress reload 失败，全部崩了，影响到业务的访问

证书创建

证书这块，我用的是 cert-manager 创建的，可以看我之前的博客：cert-manager - kubernetes 集群中 TLS 证书管理工具

部署 ingress-controller

节点打标签

k label node 192.168.22.112 ingress=true
k label node 192.168.22.113 ingress=true

通过官方 helm 做了一些修改

---
apiVersion: v1
kind: ServiceAccount
metadata:labels:helm.sh/chart: ingress-nginx-4.11.1app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/version: "1.11.1"app.kubernetes.io/part-of: ingress-nginxapp.kubernetes.io/managed-by: Helmapp.kubernetes.io/component: controllername: ingress-nginxnamespace: ingress-nginx
automountServiceAccountToken: true
---
apiVersion: v1
kind: ConfigMap
metadata:labels:helm.sh/chart: ingress-nginx-4.11.1app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/version: "1.11.1"app.kubernetes.io/part-of: ingress-nginxapp.kubernetes.io/managed-by: Helmapp.kubernetes.io/component: controllername: ingress-nginx-controllernamespace: ingress-nginx
data:allow-snippet-annotations: "true"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:labels:helm.sh/chart: ingress-nginx-4.11.1app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/version: "1.11.1"app.kubernetes.io/part-of: ingress-nginxapp.kubernetes.io/managed-by: Helmname: ingress-nginx
rules:- apiGroups:- ""resources:- configmaps- endpoints- nodes- pods- secrets- namespacesverbs:- list- watch- apiGroups:- coordination.k8s.ioresources:- leasesverbs:- list- watch- apiGroups:- ""resources:- nodesverbs:- get- apiGroups:- ""resources:- servicesverbs:- get- list- watch- apiGroups:- networking.k8s.ioresources:- ingressesverbs:- get- list- watch- apiGroups:- ""resources:- eventsverbs:- create- patch- apiGroups:- networking.k8s.ioresources:- ingresses/statusverbs:- update- apiGroups:- networking.k8s.ioresources:- ingressclassesverbs:- get- list- watch- apiGroups:- discovery.k8s.ioresources:- endpointslicesverbs:- list- watch- get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:labels:helm.sh/chart: ingress-nginx-4.11.1app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/version: "1.11.1"app.kubernetes.io/part-of: ingress-nginxapp.kubernetes.io/managed-by: Helmname: ingress-nginx
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: ingress-nginx
subjects:- kind: ServiceAccountname: ingress-nginxnamespace: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:labels:helm.sh/chart: ingress-nginx-4.11.1app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/version: "1.11.1"app.kubernetes.io/part-of: ingress-nginxapp.kubernetes.io/managed-by: Helmapp.kubernetes.io/component: controllername: ingress-nginxnamespace: ingress-nginx
rules:- apiGroups:- ""resources:- namespacesverbs:- get- apiGroups:- ""resources:- configmaps- pods- secrets- endpointsverbs:- get- list- watch- apiGroups:- ""resources:- servicesverbs:- get- list- watch- apiGroups:- networking.k8s.ioresources:- ingressesverbs:- get- list- watch# Omit Ingress status permissions if `--update-status` is disabled.- apiGroups:- networking.k8s.ioresources:- ingresses/statusverbs:- update- apiGroups:- networking.k8s.ioresources:- ingressclassesverbs:- get- list- watch- apiGroups:- coordination.k8s.ioresources:- leasesresourceNames:- ingress-nginx-leaderverbs:- get- update- apiGroups:- coordination.k8s.ioresources:- leasesverbs:- create- apiGroups:- ""resources:- eventsverbs:- create- patch- apiGroups:- discovery.k8s.ioresources:- endpointslicesverbs:- list- watch- get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:labels:helm.sh/chart: ingress-nginx-4.11.1app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/version: "1.11.1"app.kubernetes.io/part-of: ingress-nginxapp.kubernetes.io/managed-by: Helmapp.kubernetes.io/component: controllername: ingress-nginxnamespace: ingress-nginx
roleRef:apiGroup: rbac.authorization.k8s.iokind: Rolename: ingress-nginx
subjects:- kind: ServiceAccountname: ingress-nginxnamespace: ingress-nginx
---
apiVersion: v1
kind: Service
metadata:labels:helm.sh/chart: ingress-nginx-4.11.1app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/version: "1.11.1"app.kubernetes.io/part-of: ingress-nginxapp.kubernetes.io/managed-by: Helmapp.kubernetes.io/component: controllername: ingress-nginx-controller-metricsnamespace: ingress-nginx
spec:type: ClusterIPports:- name: metricsport: 10254protocol: TCPtargetPort: metricsselector:app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/component: controller
---
apiVersion: v1
kind: Service
metadata:labels:helm.sh/chart: ingress-nginx-4.11.1app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/version: "1.11.1"app.kubernetes.io/part-of: ingress-nginxapp.kubernetes.io/managed-by: Helmapp.kubernetes.io/component: controllername: ingress-nginx-controller-admissionnamespace: ingress-nginx
spec:type: ClusterIPports:- name: https-webhookport: 443targetPort: webhookappProtocol: httpsselector:app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/component: controller
---
apiVersion: v1
kind: Service
metadata:annotations:labels:helm.sh/chart: ingress-nginx-4.11.1app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/version: "1.11.1"app.kubernetes.io/part-of: ingress-nginxapp.kubernetes.io/managed-by: Helmapp.kubernetes.io/component: controllername: ingress-nginx-controllernamespace: ingress-nginx
spec:type: NodePortports:- name: httpport: 80protocol: TCPtargetPort: httpnodePort: 31080- name: httpsport: 443protocol: TCPtargetPort: httpsnodePort: 31443selector:app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/component: controller
---
apiVersion: apps/v1
kind: Deployment
metadata:labels:helm.sh/chart: ingress-nginx-4.11.1app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/version: "1.11.1"app.kubernetes.io/part-of: ingress-nginxapp.kubernetes.io/managed-by: Helmapp.kubernetes.io/component: controllername: ingress-nginx-controllernamespace: ingress-nginx
spec:selector:matchLabels:app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/component: controllerreplicas: 2revisionHistoryLimit: 10minReadySeconds: 0template:metadata:labels:helm.sh/chart: ingress-nginx-4.11.1app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/version: "1.11.1"app.kubernetes.io/part-of: ingress-nginxapp.kubernetes.io/managed-by: Helmapp.kubernetes.io/component: controllerspec:affinity:nodeAffinity:requiredDuringSchedulingIgnoredDuringExecution:nodeSelectorTerms:- matchExpressions:- key: ingressoperator: Invalues:- "true"podAntiAffinity:requiredDuringSchedulingIgnoredDuringExecution:- labelSelector:matchExpressions:- key: app.kubernetes.io/nameoperator: Invalues:- ingress-nginxtopologyKey: kubernetes.io/hostnamednsPolicy: ClusterFirstcontainers:- name: controllerimage: k8s.m.daocloud.io/ingress-nginx/controller:v1.11.1imagePullPolicy: IfNotPresentlifecycle:preStop:exec:command:- /wait-shutdownargs:- /nginx-ingress-controller- --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller- --election-id=ingress-nginx-leader- --controller-class=k8s.io/ingress-nginx- --ingress-class=nginx- --configmap=$(POD_NAMESPACE)/ingress-nginx-controller- --validating-webhook=:8443- --validating-webhook-certificate=/usr/local/certificates/cert- --validating-webhook-key=/usr/local/certificates/keysecurityContext:runAsNonRoot: truerunAsUser: 101allowPrivilegeEscalation: falseseccompProfile:type: RuntimeDefaultcapabilities:drop:- ALLadd:- NET_BIND_SERVICEreadOnlyRootFilesystem: falseenv:- name: POD_NAMEvalueFrom:fieldRef:fieldPath: metadata.name- name: POD_NAMESPACEvalueFrom:fieldRef:fieldPath: metadata.namespace- name: LD_PRELOADvalue: /usr/local/lib/libmimalloc.solivenessProbe:failureThreshold: 5httpGet:path: /healthzport: 10254scheme: HTTPinitialDelaySeconds: 10periodSeconds: 10successThreshold: 1timeoutSeconds: 1readinessProbe:failureThreshold: 3httpGet:path: /healthzport: 10254scheme: HTTPinitialDelaySeconds: 10periodSeconds: 10successThreshold: 1timeoutSeconds: 1ports:- name: httpcontainerPort: 80protocol: TCP- name: httpscontainerPort: 443protocol: TCP- name: metricscontainerPort: 10254protocol: TCP- name: webhookcontainerPort: 8443protocol: TCPvolumeMounts:- name: webhook-certmountPath: /usr/local/certificates/readOnly: trueresources:requests:cpu: 100mmemory: 90Miaffinity:nodeAffinity:requiredDuringSchedulingIgnoredDuringExecution:nodeSelectorTerms:- matchExpressions:- key: ingressoperator: Invalues:- "true"podAntiAffinity:requiredDuringSchedulingIgnoredDuringExecution:- labelSelector:matchExpressions:- key: app.kubernetes.io/nameoperator: Invalues:- ingress-nginxtopologyKey: kubernetes.io/hostnameserviceAccountName: ingress-nginxterminationGracePeriodSeconds: 300volumes:- name: webhook-certsecret:secretName: ingress-nginx-admissionitems:- key: tls.crtpath: cert- key: tls.keypath: key
---
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:labels:helm.sh/chart: ingress-nginx-4.11.1app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/version: "1.11.1"app.kubernetes.io/part-of: ingress-nginxapp.kubernetes.io/managed-by: Helmapp.kubernetes.io/component: controllername: nginx
spec:controller: k8s.io/ingress-nginx
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:name: ingress-nginx-root-certnamespace: ingress-nginx
spec:secretName: ingress-nginx-root-certduration: "43800h0m0s"issuerRef:name: ingress-nginx-self-signed-issuercommonName: "ca.webhook.ingress-nginx"isCA: truesubject:organizations:- ingress-nginx
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:name: ingress-nginx-admissionnamespace: ingress-nginx
spec:secretName: ingress-nginx-admissionduration: "8760h0m0s"issuerRef:name: ingress-nginx-root-issuerdnsNames:- ingress-nginx-controller-admission- ingress-nginx-controller-admission.ingress-nginx- ingress-nginx-controller-admission.ingress-nginx.svcsubject:organizations:- ingress-nginx-admission
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:name: ingress-nginx-self-signed-issuernamespace: ingress-nginx
spec:selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:name: ingress-nginx-root-issuernamespace: ingress-nginx
spec:ca:secretName: ingress-nginx-root-cert
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:annotations:certmanager.k8s.io/inject-ca-from: "ingress-nginx/ingress-nginx-admission"cert-manager.io/inject-ca-from: "ingress-nginx/ingress-nginx-admission"labels:helm.sh/chart: ingress-nginx-4.11.1app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/version: "1.11.1"app.kubernetes.io/part-of: ingress-nginxapp.kubernetes.io/managed-by: Helmapp.kubernetes.io/component: admission-webhookname: ingress-nginx-admission
webhooks:- name: validate.nginx.ingress.kubernetes.iomatchPolicy: Equivalentrules:- apiGroups:- networking.k8s.ioapiVersions:- v1operations:- CREATE- UPDATEresources:- ingressesfailurePolicy: FailsideEffects: NoneadmissionReviewVersions:- v1clientConfig:service:name: ingress-nginx-controller-admissionnamespace: ingress-nginxpath: /networking/v1/ingresses

ingress 验证

创建测试 nginx pod

cat << EOF | kubectl apply -f -
---
apiVersion: v1
kind: Service
metadata:annotations:labels:app: nginxname: nginx-testnamespace: default
spec:internalTrafficPolicy: Clusterports:- port: 80protocol: TCPtargetPort: 80selector:app: nginxsessionAffinity: Nonetype: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:annotations:labels:app: nginxname: nginx-testnamespace: default
spec:progressDeadlineSeconds: 600replicas: 1revisionHistoryLimit: 10selector:matchLabels:app: nginxstrategy:rollingUpdate:maxSurge: 25%maxUnavailable: 25%type: RollingUpdatetemplate:metadata:labels:app: nginxspec:containers:- image: docker.m.daocloud.io/nginx:1.26.0imagePullPolicy: IfNotPresentname: nginxports:- containerPort: 80protocol: TCPresources: {}terminationMessagePath: /dev/termination-logterminationMessagePolicy: FilednsPolicy: ClusterFirstrestartPolicy: AlwaysschedulerName: default-schedulersecurityContext: {}terminationGracePeriodSeconds: 30
EOF

创建错误的 ingress 配置

cat << EOF | kubectl apply -f -
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:annotations:nginx.ingress.kubernetes.io/configuration-snippet: |proxy_pass          http://$targetbackend;proxy_set_header    X-Real-IP       $remote_addr;proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;name: ingress-testnamespace: default
spec:ingressClassName: nginxrules:- http:paths:- backend:service:name: nginx-testport:number: 80path: /pathType: Prefix
EOF

• 此时会出现报错，这个报错是 admission webhook 触发的，并且 ingress-controller 也不会触发重启，因为不会更新和创建 ingress 规则，所以不会影响到业务

Error from server (BadRequest): error when creating "STDIN": admission webhook "validate.nginx.ingress.kubernetes.io" denied the request:
-------------------------------------------------------------------------------
Error: exit status 1
2024/09/26 07:30:35 [emerg] 166#166: no host in upstream "" in /tmp/nginx/nginx-cfg2607502217:404
nginx: [emerg] no host in upstream "" in /tmp/nginx/nginx-cfg2607502217:404
nginx: configuration file /tmp/nginx/nginx-cfg2607502217 test failed

创建正确的 ingress 配置

cat << EOF | kubectl apply -f -
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:annotations:name: ingress-testnamespace: default
spec:ingressClassName: nginxrules:- http:paths:- backend:service:name: nginx-testport:number: 80path: /pathType: Prefix
EOF

此时会返回 created，说明 ingress 的规则配置成功了

ingress.networking.k8s.io/ingress-test created

验证 ingress

curl 192.168.22.112:31080

正常会返回下面的内容，是 nginx 的默认页面

<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p><p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p><p><em>Thank you for using nginx.</em></p>
</body>
</html>