k8s中部署prometheus并监控k8s集群以及nginx案例

4台主机node1主机k8s集群中的masternode2主机搭建了harbor仓库存储所需的docker镜像test3、4主机k8s集群中的woker搭建prometheushttps://github.com/prometheus-operator/kube-prometheus获取prometheus压缩包的途径不同k8s的版本对应不同的prometheus版本我的k8s环境版本是1.30对应prometheus0.14的版本[rootnode1 ~]# tar zxf kube-prometheus-0.14.0.tar.gz [rootnode1 ~]# mv kube-prometheus-0.14.0 kube-prometheus [rootnode1 ~]# cd kube-prometheus/manifests/ [rootnode1 manifests]# kubectl create -f setup/ #创建prometheus所需要的命名空间等因为更接近生产环境我把k8s集群的docker镜像源写的是我harbor仓库所以这些镜像需要到node2主机上使用dockerpull下来再tag、push到harbor仓库让node1主机拉取。node1主机中这些文件的镜像地址都修改成harbor仓库的地址。然后到prometheus-service.yaml、grafana-service.yaml、alertmanager-service.yaml进入其中将网络类型改成nodeport或则LoadBalancer方便外网访问。我这使用的是LoadBalancer。LoadBalancer网络类型需要依赖第三方软件进行IP分布我环境中使用的是MetlLB。示例prometheus-service.yamlapiVersion: v1 kind: Service metadata: labels: app.kubernetes.io/component: prometheus app.kubernetes.io/instance: k8s app.kubernetes.io/name: prometheus app.kubernetes.io/part-of: kube-prometheus app.kubernetes.io/version: 2.54.1 name: prometheus-k8s namespace: monitoring spec: type: LoadBalancer ports: - name: web port: 9090 targetPort: web - name: reloader-web port: 8080 targetPort: reloader-web selector: app.kubernetes.io/component: prometheus app.kubernetes.io/instance: k8s app.kubernetes.io/name: prometheus app.kubernetes.io/part-of: kube-prometheus sessionAffinity: ClientIP执行apply命令让其生效[rootnode1 manifests]# kubectl apply -f .svc情况[rootnode1 manifests]# kubectl -n monitoring get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE alertmanager-main LoadBalancer 10.100.86.83 192.168.247.100 9093:31226/TCP,8080:30967/TCP 22h alertmanager-operated ClusterIP None none 9093/TCP,9094/TCP,9094/UDP 22h blackbox-exporter ClusterIP 10.104.69.13 none 9115/TCP,19115/TCP 22h grafana LoadBalancer 10.96.27.188 192.168.247.101 3000:31805/TCP 22h kube-state-metrics ClusterIP None none 8443/TCP,9443/TCP 22h node-exporter ClusterIP None none 9100/TCP 22h prometheus-adapter ClusterIP 10.99.151.111 none 443/TCP 22h prometheus-k8s LoadBalancer 10.98.57.219 192.168.247.102 9090:32758/TCP,8080:32292/TCP 22h prometheus-operated ClusterIP None none 9090/TCP 22h prometheus-operator ClusterIP None none 8443/TCP 22h现在就能正常访问grafana界面现在prometheus和grafana就搭建完成了prometheus监控k8s中的nginx pod资源每个服务都有其对应的exporterexporter主要是将服务资源转换成metrics使prometheus能够读懂并显现出来。创建nginx的pod yaml文件使用deployment控制器创建nginx podapiVersion: apps/v1 kind: Deployment metadata: labels: app: test name: test spec: replicas: 1 selector: matchLabels: app: test template: metadata: labels: app: test spec: containers: - image: test/nginx:v1 name: nginx volumeMounts: - name: nginx-config mountPath: /etc/nginx/conf.d/default.conf subPath: default.conf - image: nginx/nginx-prometheus-exporter:1.4.2 name: nginx-exporter args: - -nginx.scrape-urihttp://127.0.0.1:80/nginx_status ports: - containerPort: 9113 name: metrics volumes: - name: nginx-config configMap: name: nginx-config --- apiVersion: v1 kind: ConfigMap metadata: name: nginx-config data: default.conf: | server { listen 80; server_name localhost; location / { root /usr/share/nginx/html; index index.html; } # Nginx 监控状态页 location /nginx_status { stub_status; allow all; # 生产可限制IP测试先放开 } } --- apiVersion: v1 kind: Service metadata: labels: app: test name: test spec: ports: - port: 80 targetPort: 80 name: http - port: 9113 targetPort: 9113 name: metrics selector: app: test type: LoadBalancer然后再创建nginx-serviceMonitor.yaml配置让prometheus自动发现pod并进行监控apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: nginx namespace: monitoring labels: release: prometheus # 必须和你 Prometheus 的 release 标签一致 spec: selector: matchLabels: app: test namespaceSelector: matchNames: - default endpoints: - port: metrics path: /metrics interval: 15s scheme: httpapply一下就能正常监测