prometheus + alertmanager 搭建告警通知

prometheus

下载prometheus-2.53.2

prometheus.yml文件修改

global:scrape_interval: 15sevaluation_interval: 15salerting:alertmanagers:- static_configs:- targets:- 127.0.0.1:9093rule_files:- "rules/rule-*.yml"scrape_configs:- job_name: "prometheus"static_configs:- targets: ["localhost:9090", "127.0.0.1:9104"]

其中127.0.0.1:9104是mysqld_exporter的metric地址

新建rules目录，并创建规则 rule-first.yml

groups:- name: InstanceDown_Rulerules:- alert: InstanceDown  # 告警名称expr: up == 0        # 告警条件for: 30s              # 告警触发前需要持续满足条件的时间labels:severity: critical # 告警的严重程度annotations:summary: "Instance {{ $labels.instance }} down"description: "Instance {{ $labels.instance }} has been down for more than 5 minutes."

启动Prometheus

prometheus --config.file=prometheus.yml --storage.tsdb.path=./data --web.enable-lifecycle

访问Prometheus

http://127.0.0.1:9090/

alertmanager

下载alertmanager-0.27.0

修改配置文件alertmanager.yml

route:group_by: ['instance']group_wait: 10sgroup_interval: 20s#repeat_interval: 1hreceiver: 'web.hook'
receivers:- name: 'web.hook'webhook_configs:- url: 'http://127.0.0.1:5001/alert/hook'send_resolved: true
#inhibit_rules:
#  - source_match:
#      severity: 'critical'
#    target_match:
#      severity: 'warning'
#    equal: ['instance']

其中 http://127.0.0.1:5001/alert/hook 是接收告警的钩子接口

启动alertmanager

alertmanager --config.file=alertmanager.yml

访问alertmanager

http://127.0.0.1:9093/

grafana

下载grafana-11.2.1

启动grafana

grafana-server

访问grafana

http://127.0.0.1:3000

mysqld_exporter

下载 mysqld_exporter-0.15.1

在mysqld_exporter根目录创建.my.cnf文件

[client]
user=root
password=root

user和password分别是MySQL数据库的用户和密码；mysqld_exporter需要安装与mysql_server同一个服务器上。

启动mysqld_exporter

mysqld_exporter

访问mysqld_exporter

http://127.0.0.1:9104/

编写hook

@PostMapping("/alert/hook")
public Response<String> alertHook(@RequestBody Map<String, Object> alertDataMap) {//TODO 在这里实现告警处理，发微信、邮件、钉钉都可以System.out.println(JSON.toJSONString(alertDataMap));return Response.ok("success");
}

以上使用Java编写接口，根据实际可以使用任何一种语言编写。

可以关闭mysqld_exporter，模拟服务宕机。/alert/hook接口收到的数据如下：

{"receiver": "web\\.hook","status": "firing","alerts": [{"status": "firing","labels": {"alertname": "InstanceDown","instance": "127.0.0.1:9104","job": "prometheus","severity": "critical"},"annotations": {"description": "Instance 127.0.0.1:9104 has been down for more than 5 minutes.","summary": "Instance 127.0.0.1:9104 down"},"startsAt": "2024-10-10T11:27:58.11Z","endsAt": "0001-01-01T00:00:00Z","generatorURL": "http://olive-my:9090/graph?g0.expr=up+%3D%3D+0&g0.tab=1","fingerprint": "106b3a6075af7628"}],"groupLabels": {"instance": "127.0.0.1:9104"},"commonLabels": {"alertname": "InstanceDown","instance": "127.0.0.1:9104","job": "prometheus","severity": "critical"},"commonAnnotations": {"description": "Instance 127.0.0.1:9104 has been down for more than 5 minutes.","summary": "Instance 127.0.0.1:9104 down"},"externalURL": "http://olive-my:9093","version": "4","groupKey": "{}:{instance=\"127.0.0.1:9104\"}","truncatedAlerts": 0
}