Prometheus监控mysql nginx tomcat 黑盒监控

部署consul_exporter，用与服务发现
https://github.com/prometheus/consul_exporter/releases/download/v0.9.0/consul_exporter-0.9.0.linux-amd64.tar.gz

注册

oot@ubuntu20:~# cat consul_export.json 
root@ubuntu20:~# cat consul_export.json 
{"services": [{"id": "consul_exporter","name": "192.168.1.50_cousul_expo","address": "192.168.1.50","port": 9107,"tags": ["consul_exporter"],"checks": [{"http": "http://192.168.1.50:9107/metrics","interval": "5s"}]}]

consul services register consul_export.json
可以访问consul查看，

Prometheus添加配置

 - job_name: "consul_exporter"metrics_path: '/metrics'scheme: "http"consul_sd_configs:- server: "192.168.1.50:8500"tags:- "consul_exporter"refresh_interval: 1m

重新加载Prometheus
root@consumer:/apps/prometheus# curl -XPOST localhost:9090/-/reload

export 获取数据转换为Prometheus数据

pushgetway

查询持久化

grafanf绘图

altermanager 发送告警系统
选择

应用程序不兼容Prometheus

mysqld export

nginx-export

https://github.com/nginxinc/nginx-prometheus-exporter

添加nginx配置，添加这个查看指标

       location /stub_status {# Turn on nginx statsstub_status on;access_log   off;# Only allow access from certain IP addressesallow all;#deny all;}

运行采集指标export
root@ubuntu20:/apps# ./nginx-prometheus-exporter -nginx.scrape-uri=http://192.168.1.50/stub_status

root@ubuntu20:/etc/nginx# netstat -antp |grep 9113
tcp6 0 0 :::9113 ::😗 LISTEN 168719/./nginx-prom

root@ubuntu20:~# cat nginx_export.json 
{"services": [{"id": "nginx_exporter","name": "192.168.1.50_nginx","address": "192.168.1.50","port": 9113,"tags": ["nginx_exporter"],"checks": [{"http": "http://192.168.1.50:9113/metrics","interval": "5s"}]}]
}consul services register  nginx_export.json 

配置Prometheus添加

  - job_name: "nginx_exporter"metrics_path: '/metrics'scheme: "http"consul_sd_configs:- server: "192.168.1.50:8500"tags:- "nginx_exporter"refresh_interval: 1m

curl -XPOST localhost:9090/-/reload

监控tomcat

tomcat
mcat自身并不能提供监控指标数据，需要借助第三方exporter实现：https://github.com/nlighten/tomcat_exporter

JVM的export或者tomcat export

构建tomcat镜像，并暴露8082端口

cat docker-compose.yml 
version: '3.6'volumes:tomcat_webapps: {}networks:monitoring:driver: bridgeipam:config:- subnet: 172.31.130.0/24services:tomcat:#image: tomcat:jdk11build:context: tomcatdockerfile: Dockerfile hostname: tomcat.magedu.comexpose:- 8080ports:- 8082:8080volumes:- tomcat_webapps:/usr/local/tomcat/webapps- ./tomcat/tomcat-users.xml:/usr/local/tomcat/conf/tomcat-users.xmlnetworks:- monitoringenvironment:TZ: Asia/Shanghai

root@jenkins:~/learning-prometheus/08-prometheus-components-compose/tomcat-and-metrics# docker-compose up

注册到consul

root@ubuntu20:~# cat tomcat2.json 
{"services": [{"id": "tomcat","name": "192.168.1.51_tomcat","address": "192.168.1.51","port": 8082,"tags": ["tomcat_exporter"],"checks": [{"http": "http://192.168.1.51:8082/metrics","interval": "5s"}]}]
}consul services register   tomcat2.json 

添加Prometheus配置并重新加载

  - job_name: "tomcat_exporter"metrics_path: '/metrics'scheme: "http"consul_sd_configs:- server: "192.168.1.50:8500"tags:- "tomcat_exporter"refresh_interval: 1m

curl -XPOST localhost:9090/-/reload

黑盒监控

https://github.com/prometheus/blackbox_exporter/releases/download/v0.24.0/blackbox_exporter-0.24.0.linux-amd64.tar.gz

通过容器运行
root@jenkins:~/learning-prometheus/08-prometheus-components-compose/blackbox-exporter# docker-compose up -d

Prometheus添加

  - job_name: "tomcat_exporter"metrics_path: '/metrics'scheme: "http"consul_sd_configs:- server: "192.168.1.50:8500"tags:- "tomcat_exporter"refresh_interval: 1m- job_name: 'blackbox'metrics_path: /probeparams:module: [http_2xx]static_configs:- targets:- www.magedu.com- www.google.comrelabel_configs:- source_labels: [__address__]target_label: __param_target- source_labels: [__param_target]target_label: instance- target_label: __address__replacement: "192.168.1.51:9115"  # Blackbox exporter.- target_label: regionreplacement: "remote"

容器部署在9115端口页面显示失败

使用二进制部署参考https://blog.csdn.net/u011417723/article/details/120783567

正常