prometheus监控之黑盒(blackbox)监控

1.简单介绍

blackbox-exporter项目地址：https://github.com/prometheus/blackbox_exporter

blackbox-exporter是Prometheus官方提供的一个黑盒监控解决方案，blackbox-exporter无须安装在被监控的目标环境中，用户只需要将其安装在与Prometheus和被监控目标互通的环境中，通过HTTP、HTTPS(URL/API可用性检测)、DNS(域名解析)、TCP(端口存活检测)、ICMP(主机存活检测)等方式对网络进行探测监控，还可以探测SSL证书过期时间。

2.二进制安装

2.1 下载并解压

mkdir -p /root/exporter/blackbox_exporter && cd /root/exporter/blackbox_exporter 
wget https://github.com/prometheus/blackbox_exporter/releases/download/v0.23.0/blackbox_exporter-0.23.0.linux-amd64.tar.gz
tar zxvf blackbox_exporter-0.23.0.linux-amd64.tar.gz

2.2 创建systemd服务

执行vi /etc/systemd/system/blackbox_exporter.service并把下面内容复制到文件中

[Service]
ExecStart=/root/exporter/blackbox_exporter/blackbox_exporter --config.file=/root/exporter/blackbox_exporter/blackbox.yml --web.listen-address=:9115
Restart=on-failure[Install]
WantedBy=multi-user.target

2.3 配置

# /root/exporter/blackbox_exporter/blackbox.yml
modules:http_2xx: # http检测模块,blockbox-exporter中所有的探针均是以module的信息进行配置prober: httphttp:preferred_ip_protocol: "ip4"tls_config:insecure_skip_verify: truehttp_post_2xx: # http post监测模块prober: httphttp:method: POSTtcp_connect: # tcp检测模块prober: tcppop3s_banner:prober: tcptcp:query_response:- expect: "^+OK"tls: truetls_config:insecure_skip_verify: falsessh_banner:prober: tcptcp:query_response:- expect: "^SSH-2.0-"irc_banner:prober: tcptcp:query_response:- send: "NICK prober"- send: "USER prober prober prober :prober"- expect: "PING :([^ ]+)"send: "PONG ${1}"- expect: "^:[^ ]+ 001"icmp:prober: icmp

2.4 启动

systemctl daemon-reload 
systemctl start blackbox_exporter # 启动
systemctl status blackbox_exporter # 状态
systemctl enable blackbox_exporter # 开机自启动

2.5手动获取指标

执行curl [http://xx.xx.xx.xx:9115/probe?target=[target目标]&module=[模块名]&debug=true](http://10.17.12.10:9115/probe?target=https://monitor.asiainfo.com&module=http_2xx&debug=true)后效果如下：

3.监控

3.1HTTP监控

- job_name: "blackbox_http"metrics_path: /probe # 指定指标接口params: # 指定查询参数，在prometheus向target发送get请求获取指标数据时，会传递到url上module: [http_2xx]honor_labels: trueconsul_sd_configs:- server: 'xx.xx.xx.xx:8500' # 服务发现consul地址services: []relabel_configs:- source_labels: [__meta_consul_tags]regex: .*blackbox-http.*action: keep- regex: __meta_consul_service_metadata_(.+)action: labelmap# 将标签__meta_consul_service_metadata_instance的值赋值给__param_target标签# 以__param开头的标签也会作为查询参数传递prometheus的get请求，作用和上面的params配置类似- source_labels: [__meta_consul_service_metadata_instance]target_label: __param_target# 将标签__param_target的值赋值给instance标签- source_labels: [__param_target]target_label: instance# 将标签__address__的值修改给balckbox-expoter的地址- target_label: __address__replacement: xx.xx.xx.xx:9115 # blackbox-exporter地址

3.2 TCP监控

- job_name: 'blackbox-tcp'metrics_path: /probeparams:module: [tcp_connect]honor_labels: trueconsul_sd_configs:- server: 'xx.xx.xx.xx:8500'services: []relabel_configs:- source_labels: [__meta_consul_tags]regex: .*blackbox-tcp.*action: keep- regex: __meta_consul_service_metadata_(.+)action: labelmap- source_labels: [__meta_consul_service_metadata_instance]target_label: __param_target- source_labels: [__param_target]target_label: instance- target_label: __address__replacement: xx.xx.xx.xx:9115

3.3 ICMP监控

- job_name: "blackbox_icmp"metrics_path: /probeparams:module: [icmp]consul_sd_configs:- server: 'xx.xx.xx.xx:8500'services: []relabel_configs:- source_labels: [__meta_consul_tags]regex: .*blackbox-icmp.*action: keep- regex: __meta_consul_service_metadata_(.+)action: labelmap- source_labels: [__meta_consul_service_metadata_instance]target_label: __param_target- source_labels: [__param_target]target_label: instance- target_label: __address__replacement: xx.xx.xx.xx:9115

3.4 python注册consul服务

注册服务模板请参考: https://blog.csdn.net/liulunan_lln/article/details/140875069?spm=1001.2014.3001.5502

# 模版请参考
def register_service_web_exporter_to_consul() -> None:print("register service exporter to consul.")host = ""exporter_id = f"web-exporter-{host}"params = consul_register_template.render(exporter_id=exporter_id,tags=['service', "web", 'exporter', 'blackbox-http'],exporter_address=host,exporter_port=80,labels={"host": "xxx","port": 80,"instance": "http|https://xx.xx.com",})print(f"Start register svc: {exporter_id}")resp = requests.put(f"https://xx.xx.com/v1/agent/service/register", # consul服务地址json=json.loads(params),verify=False)if not resp.ok:raise Exception(f"register svc {exporter_id} failed.")print(f"register svc {exporter_id} Success.")

4.指标说明

# DNS解析时间,单位 s
probe_dns_lookup_time_seconds 0.000199105
# 探测从开始到结束的时间,单位 s,请求这个页面响应时间
probe_duration_seconds 0.010889113
# HELP probe_failed_due_to_regex Indicates if probe failed due to regex
# TYPE probe_failed_due_to_regex gauge
probe_failed_due_to_regex 0
# HTTP 内容响应的长度
probe_http_content_length -1
# 按照阶段统计每阶段的时间
probe_http_duration_seconds{phase="connect"} 0.001083728    #连接时间
probe_http_duration_seconds{phase="processing"} 0.008365885 #处理请求的时间
probe_http_duration_seconds{phase="resolve"} 0.000199105    #响应时间
probe_http_duration_seconds{phase="tls"} 0                  #校验证书的时间
probe_http_duration_seconds{phase="transfer"} 0.000446424   #传输时间
# 重定向的次数
probe_http_redirects 0
# ssl 指示是否将 SSL 用于最终重定向
probe_http_ssl 0
# 返回的状态码
probe_http_status_code 200
# 未压缩的响应主体长度
probe_http_uncompressed_body_length 1766
# http协议的版本
probe_http_version 1.1
# HELP probe_ip_addr_hash Specifies the hash of IP address. It's useful to detect if the IP address changes.
probe_ip_addr_hash 3.24030434e+09
# 使用的ip协议的版本号
probe_ip_protocol 4
probe_ssl_earliest_cert_expiry 1.749882884e+09
robe_ssl_last_chain_expiry_timestamp_seconds -6.21355968e+10
probe_ssl_last_chain_info{fingerprint_sha256="5ce3bbf06bd1608e04a64b1cd91e3fa69ed86cd9c55a1da52a8187140e0ece5b",issuer="CN=GlobalSign GCC R3 DV TLS CA 2020,O=GlobalSign nv-sa,C=BE",subject="CN=*.asiainfo.com",subjectalternative="*.asiainfo.com,asiainfo.com"} 1
# 是否探测成功
probe_success 1
# tls版本
probe_tls_version_info{version="TLS 1.2"} 1

5.Grafana模板

导入dashboard 
13659 HTTP状态监控
9965 SSL TCP HTTP综合监控图标
13230 SSL证书监控

6.prometheus告警规则

- alert: blackbox-defaultannotations:description: 域名证书7天后过期summary: 域名证书即将过期,VALUE = {{ $value }}expr: probe_ssl_earliest_cert_expiry - time() < 86400 * 7for: 30mlabels:rule_type: blackboxseverity: emergency