prometheus通过Endpoints自定义grafana的dashboard模块

1、prometheus自定义的dashboard模块
文件路径/etc/prometheus/config_out/prometheus-env.yaml

- job_name: serviceMonitor/monitoring/pfil/0honor_labels: falsekubernetes_sd_configs:- role: endpointsnamespaces:names:- monitoringrelabel_configs:- source_labels:- jobtarget_label: __tmp_prometheus_job_name- action: keepsource_labels:- __meta_kubernetes_service_label_app- __meta_kubernetes_service_labelpresent_appregex: (pfil);true- action: keepsource_labels:- __meta_kubernetes_service_label_source- __meta_kubernetes_service_labelpresent_sourceregex: (external);true- action: keepsource_labels:- __meta_kubernetes_endpoint_port_nameregex: metrics- source_labels:- __meta_kubernetes_endpoint_address_target_kind- __meta_kubernetes_endpoint_address_target_nameseparator: ;regex: Node;(.*)replacement: ${1}target_label: node- source_labels:- __meta_kubernetes_endpoint_address_target_kind- __meta_kubernetes_endpoint_address_target_nameseparator: ;regex: Pod;(.*)replacement: ${1}target_label: pod- source_labels:- __meta_kubernetes_namespacetarget_label: namespace- source_labels:- __meta_kubernetes_service_nametarget_label: service- source_labels:- __meta_kubernetes_pod_nametarget_label: pod- source_labels:- __meta_kubernetes_pod_container_nametarget_label: container- action: dropsource_labels:- __meta_kubernetes_pod_phaseregex: (Failed|Succeeded)- source_labels:- __meta_kubernetes_service_nametarget_label: jobreplacement: ${1}- source_labels:- __meta_kubernetes_service_label_jobnametarget_label: jobregex: (.+)replacement: ${1}- target_label: endpointreplacement: metrics- source_labels:- __address__target_label: __tmp_hashmodulus: 1action: hashmod- source_labels:- __tmp_hashregex: 0action: keep

注释

Prometheus 配置文件 (/etc/prometheus/config_out/prometheus-env.yaml) 使用的是 Kubernetes 服务发现（kubernetes_sd_configs），并且指定了 role: endpoints。这意味着 Prometheus 会自动去查找 Kubernetes 集群里的 Endpoints（端点） 对象来发现监控目标。

根据配置，Prometheus 只会抓取满足以下所有条件的目标：

命名空间（Namespace）: 目标必须位于 monitoring 这个命名空间内。
服务标签（Service Labels）: 与目标关联的 Kubernetes Service（服务） 必须 同时 具有 app=pfil 和 source=external 这两个标签。
端口名称（Port Name）: 在 Service 定义中，目标所使用的那个端口，其 名称（name） 必须是 metrics。
Pod 状态（Pod Phase）: 不会抓取状态是 Failed (失败) 或 Succeeded (成功) 的 Pod 对应的目标

2、创建ServiceMonitor
注意标签selector

  selector:matchLabels:app: pfilsource: external

---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:annotations: {}labels:app: pfilrelease: kube-prometheussource: externalname: pfilnamespace: monitoringresourceVersion: '308908134'
spec:endpoints:- port: metricsjobLabel: jobnamenamespaceSelector:matchNames:- monitoringselector:matchLabels:app: pfilsource: external

3、创建Endpoints
Endpoints1

apiVersion: v1
kind: Endpoints
metadata:labels:app: pfiljobname: pfilsource: externalname: pfilnamespace: monitoringresourceVersion: "552235238"uid: dbd41ece-caf3-4b7b-8510-3c6eda714f7a
subsets:
- addresses:- hostname: test-mainip: 1.1.1.1- hostname: test-east-1ip: 2.2.2.2- hostname: test-west-1ip: 3.3.3.3ports:- name: metricsport: 9002protocol: TCP

Endpoints2

apiVersion: v1
kind: Endpoints
metadata:name: parasail-minernamespace: monitoringlabels:app: pfilsource: externalcomponent: parasail-miner
subsets:
- addresses:- ip: 1.1.1.1hostname: parasail-minerports:- name: metrics port: 8002protocol: TCP

4、创建Service
service1

---
apiVersion: v1
kind: Service
metadata:annotations: {}labels:app: pfilcomponent: parasail-minersource: externalname: parasail-minernamespace: monitoringresourceVersion: '552542525'
spec:clusterIP: 10.100.103.0clusterIPs:- 10.100.103.0internalTrafficPolicy: ClusteripFamilies:- IPv4ipFamilyPolicy: SingleStackports:- name: metricsport: 8002protocol: TCPtargetPort: 8002sessionAffinity: Nonetype: ClusterIP
status:loadBalancer: {}

service2

---
apiVersion: v1
kind: Service
metadata:annotations: {}labels:app: pfiljobname: pfilsource: externalname: pfilnamespace: monitoringresourceVersion: '552219945'
spec:clusterIP: 10.100.78.9clusterIPs:- 10.100.78.9internalTrafficPolicy: ClusteripFamilies:- IPv4ipFamilyPolicy: SingleStackports:- name: metricsport: 9002protocol: TCPtargetPort: 9002- name: metrics-8002port: 8002protocol: TCPtargetPort: 8002sessionAffinity: Nonetype: ClusterIP