Prometheus + Grafana 监控常用服务

一、引言

Prometheus监控常见服务的原理主要包括服务暴露指标和Prometheus抓取指标。一方面，被监控服务通过自身提供的监控接口或借助Exporter将服务的性能指标等数据以HTTP协议的方式暴露出来；另一方面，Prometheus根据配置好的采集任务，定期去拉取这些服务暴露的指标数据，将其存储在本地的时间序列数据库中，之后用户就可以通过PromQL查询来分析这些数据，实现对服务的监控。

 二、Prometheus监控MySQL服务

1. 安装数据库

这里就不展示怎么安装啦~

2. 启动数据库

[root@elk93 ~]# /etc/init.d/mysql.server start
Starting mysql.server (via systemctl): mysql.server.service.
[root@elk93 ~]# 
[root@elk93 ~]# ss -ntl | grep 3306
LISTEN 0      151                *:3306             *:*          
LISTEN 0      70                 *:33060            *:*          
[root@elk93 ~]# # 创建用户并且授权，这里我就用已经存在的用户。
mysql> SELECT user,host,plugin FROM mysql.user;
+------------------+-----------+-----------------------+
| user             | host      | plugin                |
+------------------+-----------+-----------------------+
| linux96          | %         | mysql_native_password |
| nacos            | %         | mysql_native_password |
| mysql.infoschema | localhost | caching_sha2_password |
| mysql.session    | localhost | caching_sha2_password |
| mysql.sys        | localhost | caching_sha2_password |
| root             | localhost | caching_sha2_password |
+------------------+-----------+-----------------------+
6 rows in set (0.00 sec)mysql> 
mysql> GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO linux96;
Query OK, 0 rows affected (0.00 sec)# 查看权限
mysql> SHOW GRANTS FOR linux96;
+-------------------------------------------------------------------+
| Grants for linux96@%                                              |
+-------------------------------------------------------------------+
| GRANT SELECT, PROCESS, REPLICATION CLIENT ON *.* TO `linux96`@`%` |
| GRANT ALL PRIVILEGES ON `nova_kafka`.* TO `linux96`@`%`      |
+-------------------------------------------------------------------+

3. 下载mysql_exporter

[root@elk93 ~]# wget https://github.com/prometheus/mysqld_exporter/releases/download/v0.17.2/mysqld_exporter-0.17.2.linux-amd64.tar.gz3.解压程序 
[root@elk91 ~]# tar xf mysqld_exporter-0.17.2.linux-amd64.tar.gz  -C /usr/local/bin/ mysqld_exporter-0.17.2.linux-amd64/mysqld_exporter --strip-components=1mysqld_exporter-0.17.2.linux-amd64/mysqld_exporter显式指定要解压的归档文件内的具体路径。即从压缩包中仅提取 mysqld_exporter-0.17.2.linux-amd64/ 目录下的 mysqld_exporter 文件。
--strip-components=1移除解压路径中的前 1层目录。[root@elk91 ~]# 
[root@elk91 ~]# ll /usr/local/bin/
total 17936
drwxr-xr-x  2 root root     4096 Mar 27 10:56 ./
drwxr-xr-x 18 root root     4096 Mar 24 18:03 ../
-rwxr-xr-x  1 1001 1002 18356306 Feb 26 15:16 mysqld_exporter*
[root@elk91 ~]# 

4. 启动MySQL exporter

# MySQL-exporter可以和要暴露http(s)接口服务在同一个节点，这里分开是为了证明，不在同一个节点也可以。
[root@elk91 ~]# cat .my.cnf 
[client]
user=linux96
password=novacao666
host=10.0.0.93
[root@elk91 ~]# [root@elk91 ~]# mysqld_exporter --config.my-cnf="/root/.my.cnf" 
time=2025-03-27T03:09:09.709Z level=INFO source=mysqld_exporter.go:239 msg="Starting mysqld_exporter" version="(version=0.17.2, branch=HEAD, revision=e84f4f22f8a11089d5f04ff9bfdc5fc042605773)"
time=2025-03-27T03:09:09.709Z level=INFO source=mysqld_exporter.go:240 msg="Build context" build_context="(go=go1.23.6, platform=linux/amd64, user=root@18b69b4b0fea, date=20250226-07:16:19, tags=unknown)"
time=2025-03-27T03:09:09.709Z level=INFO source=mysqld_exporter.go:252 msg="Scraper enabled" scraper=global_status
......
time=2025-03-27T03:09:09.710Z level=INFO source=tls_config.go:350 msg="TLS is disabled." http2=false address=[::]:9104

5. 测试访问

# 一般数据量大的都是正确的，如果是几十几百就说明我们暴露的数据不太对。
[root@elk93 ~]# curl -s http://10.0.0.91:9104/metrics| wc -l
2573
[root@elk93 ~]# 

6. 配置Prometheus监控MySQL项

[root@prometheus-server31 ~]# tail -5 /softwares/prometheus-2.53.4.linux-amd64/prometheus.yml- job_name: linux96-mysql-exporterstatic_configs:- targets: - 10.0.0.91:9104# 热加载
[root@prometheus-server31 ~]# 
[root@prometheus-server31 ~]# curl -X POST http://10.0.0.31:9090/-/reload

7.  验证监控结果并导入grafana模板

# 访问http://10.0.0.31:9090/targets?search=grafana导入模板ID 17320
18949
14057
7362

 三、Prometheus监控zookeeper集群

zookeeper没有Prometheus提供的exporter组件，但是他自带metrics暴露方式；打开以下注释即可。

1. 修改zookeeper集群的配置信息

[root@elk91 ~]# vim /usr/local/zookeeper/conf/zoo.cfg 
[root@elk91 ~]# 
[root@elk91 ~]# tail -5 /usr/local/zookeeper/conf/zoo.cfg 
# zookeeper没有Prometheus提供的exporter组件，但是他自带metrics暴露方式；打开以下注释即可。
# https://prometheus.io Metrics Exporter
metricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider
metricsProvider.httpHost=0.0.0.0
metricsProvider.httpPort=7000
metricsProvider.exportJvmInfo=true
[root@elk91 ~]# # 拷贝配置文件到其他集群的节点
[root@elk91 ~]# scp /usr/local/zookeeper/conf/zoo.cfg 10.0.0.92:/usr/local/zookeeper/conf/
[root@elk91 ~]# scp /usr/local/zookeeper/conf/zoo.cfg 10.0.0.93:/usr/local/zookeeper/conf/

 2. 重启zookeeper服务

[root@elk91 ~]# zkServer.sh restart 
[root@elk92 ~]# zkServer.sh restart 
[root@elk93 ~]# zkServer.sh restart 

3. 访问测试 

http://10.0.0.91:7000/metrics
http://10.0.0.92:7000/metrics
http://10.0.0.93:7000/metrics

4. 配置prometheus监控zookeeper

[root@prometheus-server31 ~]# tail -7 /softwares/prometheus-2.53.4.linux-amd64/prometheus.yml- job_name: linux96-zk-exporterstatic_configs:- targets: - 10.0.0.91:7000- 10.0.0.92:7000- 10.0.0.93:7000
.....
[root@prometheus-server31 ~]# # 热加载Prometheus
[root@prometheus-server31 ~]# curl -X POST http://10.0.0.31:9090/-/reload
[root@prometheus-server31 ~]# 

5. 验证监控结果并导入grafana模板

5.验证配置是否生效
http://10.0.0.31:9090/targets?search=
# 看到Prometheus的targets出现zookeeper集群就行6.grafana导入模板ID 
10465

四、Prometheus监控kafka集群

1. 启动kafka集群

[root@elk91 ~]# kafka-server-start.sh -daemon /usr/local/kafka_2.13-3.9.0/config/server.properties 
[root@elk91 ~]# ss -ntl | grep 9092
LISTEN 0      50     [::ffff:10.0.0.91]:9092             *:*          
[root@elk91 ~]# [root@elk92 ~]# kafka-server-start.sh -daemon /usr/local/kafka_2.13-3.9.0/config/server.properties 
[root@elk92 ~]# ss -ntl | grep 9092
LISTEN 0      50     [::ffff:10.0.0.92]:9092             *:*          
[root@elk92 ~]# [root@elk93 ~]# kafka-server-start.sh -daemon /usr/local/kafka_2.13-3.9.0/config/server.properties 
[root@elk93 ~]# ss -ntl | grep 9092
LISTEN 0      50     [::ffff:10.0.0.93]:9092             *:*          
[root@elk93 ~]# 

 2. 下载kafka exporter 

wget https://github.com/danielqsj/kafka_exporter/releases/download/v1.9.0/kafka_exporter-1.9.0.linux-amd64.tar.gz3.解压软件包 
[root@elk92 ~]# tar xf kafka_exporter-1.9.0.linux-amd64.tar.gz -C /usr/local/bin/ kafka_exporter-1.9.0.linux-amd64/kafka_exporter --strip-components=1
[root@elk92 ~]# 
[root@elk92 ~]# ll /usr/local/bin/
total 24520
drwxr-xr-x  2 root root              4096 Mar 27 11:44 ./
drwxr-xr-x 14 root root              4096 Mar 18 14:38 ../
-rwxr-xr-x  1 1001 fwupd-refresh 25099148 Feb 17 11:04 kafka_exporter*
[root@elk92 ~]# 

3. 启动kafka exporter

# 这里我们指定了集群信息，和kafka的版本信息
[root@elk92 ~]# kafka_exporter  --kafka.server=10.0.0.91:9092  --kafka.server=10.0.0.92:9092 --kafka.server=10.0.0.93:9092 --kafka.version="3.9.0"

4. 访问kafka exporter的WebUI 

# 出现大量信息即可
http://10.0.0.92:9308/metrics

5. 配置Prometheus监控kafka集群

[root@prometheus-server31 ~]# tail -5 /softwares/prometheus-2.53.4.linux-amd64/prometheus.yml- job_name: linux96-kafka-exporterstatic_configs:- targets: - 10.0.0.92:9308
...
# 热加载
[root@prometheus-server31 ~]# curl -X POST http://10.0.0.31:9090/-/reload
[root@prometheus-server31 ~]# 

6. 验证监控结果并导入grafana模板

# 出现kafka集群的targets即可
http://10.0.0.31:9090/targets?search=8.grafana导入模板ID
12460
10122

五、prometheus监控nacos服务

nacos服务是自己有提供metrics接口，在配置文件中打开注释即可

参考链接: 监控手册 | Nacos 官网

1. 启动nacos集群

[root@elk91 ~]# /usr/local/nacos/bin/startup.sh 
[root@elk92 ~]# /usr/local/nacos/bin/startup.sh 
[root@elk93 ~]# /usr/local/nacos/bin/startup.sh 
访问测试 ：http://10.0.0.93:8848/

 2. 修改nacos服务启用Metrics接口

# nacos和zk服务都是自己有提供metrics接口，在配置文件中打开注释即可[root@elk91 ~]# vim  /usr/local/nacos/conf/application.properties
...
management.endpoints.web.exposure.include=prometheus,health
...
# 将配置文件拷贝给集群的其他节点
[root@elk91 ~]# scp /usr/local/nacos/conf/application.properties 10.0.0.92:/usr/local/nacos/conf/
[root@elk91 ~]# scp /usr/local/nacos/conf/application.properties 10.0.0.92:/usr/local/nacos/conf/

3. 重启服务 

[root@elk91 ~]# /usr/local/nacos/bin/shutdown.sh 
[root@elk92 ~]# /usr/local/nacos/bin/shutdown.sh 
[root@elk93 ~]# /usr/local/nacos/bin/shutdown.sh [root@elk91 ~]# /usr/local/nacos/bin/startup.sh 
[root@elk92 ~]# /usr/local/nacos/bin/startup.sh 
[root@elk93 ~]# /usr/local/nacos/bin/startup.sh 

4. 访问nacos的WebUI

http://10.0.0.91:8848/actuator/prometheus
http://10.0.0.92:8848/actuator/prometheus
http://10.0.0.93:8848/actuator/prometheus

5. 配置Prometheus监控nacos集群 

# 在Prometheus配置文件中，他默认只会发现到是以/metrics结尾的URI，所以这里我们要指引一下Prometheus。加一行配置文件指定。
[root@prometheus-server31 ~]# tail  -8 /softwares/prometheus-2.53.4.linux-amd64/prometheus.yml - job_name: linux96-nacos-exportermetrics_path: "/actuator/prometheus"static_configs:- targets: - 10.0.0.91:8848- 10.0.0.92:8848- 10.0.0.93:8848......# 热加载Prometheus
[root@prometheus-server31 ~]# curl -X POST http://10.0.0.31:9090/-/reload
[root@prometheus-server31 ~]# 

6. 验证监控结果并导入grafana模板

http://10.0.0.31:9090/targets?search=7.grafana导入模板ID
13221

六、 Prometheus监控docker原生Metrics

参考链接:
    https://docs.docker.com/reference/cli/dockerd/#daemon-metrics

 1. 停止docker

[root@elk93 ~]# systemctl stop docker

2. 修改docker启动脚本启用Metrics监控指标

[root@elk93 ~]# cat /lib/systemd/system/docker.service
[Unit]
Description=novacao linux Docke Engine
Documentation=https://docs.docker.com
Wants=network-online.target[Service]
Type=notify
ExecStart=/usr/bin/dockerd --metrics-addr 0.0.0.0:5200
[Install]
WantedBy=multi-user.target
[root@elk93 ~]# 
[root@elk93 ~]# systemctl daemon-reload
[root@elk93 ~]# 
[root@elk93 ~]# systemctl restart docker.service 
[root@elk93 ~]# 
[root@elk93 ~]# ss -ntl | grep 5200
LISTEN 0      4096                    *:5200             *:*          
[root@elk93 ~]# 

3. 访问docker的Metrics接口 

http://10.0.0.93:5200/metrics[root@node-exporter41 ~]# curl -s http://10.0.0.93:5200/metrics| wc -l
589
[root@node-exporter41 ~]# 

4. prometheus监控docker服务

[root@prometheus-server31 ~]# tail -5 /softwares/prometheus-2.53.4.linux-amd64/prometheus.yml - job_name: linux96-docker-exporterstatic_configs:- targets: - 10.0.0.93:5200[root@prometheus-server31 ~]# 
[root@prometheus-server31 ~]# curl -X POST http://10.0.0.31:9090/-/reload
[root@prometheus-server31 ~]# 

5. 验证监控结果并导入grafana模板

http://10.0.0.31:9090/targets?search=8.grafana导入模板ID
9621# 这里Prometheus提供的原生Metrics接口数据显示不全，所以我们有第二种解决方案。

Prometheus基于cAdvisor监控Docker容器

 1. cAdvisor是什么？

cAdvisor（Container Advisor）是由Google开源的一款轻量级容器监控工具，主要用于实时收集、分析和展示容器运行时资源使用情况与性能数据，帮助用户了解容器化应用的行为和资源消耗。

2. 在docker节点拉取镜像 

[root@elk92 ~]# docker pull gcr.io/cadvisor/cadvisor-amd64:v0.52.1
v0.52.1: Pulling from cadvisor/cadvisor-amd64
44cf07d57ee4: Already exists 
.....
Status: Downloaded newer image for gcr.io/cadvisor/cadvisor-amd64:v0.52.1
gcr.io/cadvisor/cadvisor-amd64:v0.52.1
[root@elk92 ~]# 

3. 运行cAdvisor 

[root@elk92 ~]# docker run \--volume=/:/rootfs:ro \--volume=/var/run:/var/run:ro \--volume=/sys:/sys:ro \--volume=/var/lib/docker/:/var/lib/docker:ro \--volume=/dev/disk/:/dev/disk:ro \--publish=18080:8080 \--detach=true \--name=cadvisor \--privileged \--device=/dev/kmsg \gcr.io/cadvisor/cadvisor-amd64:v0.52.1[root@elk92 ~]# docker ps -l
CONTAINER ID   IMAGE                                    COMMAND                  CREATED         STATUS                        PORTS                                         NAMES
3857cc9ff34e   gcr.io/cadvisor/cadvisor-amd64:v0.52.1   "/usr/bin/cadvisor -…"   2 minutes ago   Up About a minute (healthy)   0.0.0.0:18080->8080/tcp, :::18080->8080/tcp   cadvisor
[root@elk92 ~]# 

4. 访问测试

[root@elk92 ~]# curl -s http://10.0.0.92:18080/metrics | wc -l
4885
[root@elk92 ~]# 

5. 访问cAdVisor的webUI测试

http://10.0.0.92:18080/containers/

6. 配置Prometheus监控docker容器

[root@prometheus-server31 ~]# tail -6 /softwares/prometheus-2.53.4.linux-amd64/prometheus.yml - job_name: linux96-cadVisor-exporterstatic_configs:- targets: - 10.0.0.92:18080- 10.0.0.93:18080# 热加载Prometheus
[root@prometheus-server31 ~]# curl -X POST http://10.0.0.31:9090/-/reload
[root@prometheus-server31 ~]# 

7. 验证监控结果并导入grafana模板

http://10.0.0.31:9090/targets?search=6.grafana导入模板ID
11600 

七、prometheus监控nginx

方案1>nginx-module-vts

1. 编译安装NGINX

1.1 安装编译工具
[root@elk91 ~]# apt -y install git wget gcc make zlib1g-dev build-essential libtool openssl libssl-dev1.2 克隆nginx-module-vts模块# 配置虚拟机VPN
export https_proxy=http://10.0.0.1:7890
export http_proxy=http://10.0.0.1:7890[root@elk91 ~]# env| grep http
https_proxy=http://10.0.0.1:7890
http_proxy=http://10.0.0.1:7890
[root@elk91 ~]# [root@elk91 ~]# git clone https://github.com/vozlt/nginx-module-vts.git
Cloning into 'nginx-module-vts'...
remote: Enumerating objects: 1134, done.
remote: Counting objects: 100% (262/262), done.
remote: Compressing objects: 100% (87/87), done.
remote: Total 1134 (delta 206), reused 175 (delta 175), pack-reused 872 (from 2)
Receiving objects: 100% (1134/1134), 1.22 MiB | 935.00 KiB/s, done.
Resolving deltas: 100% (746/746), done.
[root@elk91 ~]# 1.3 下载nginx软件包
[root@elk91 ~]# wget http://nginx.org/download/nginx-1.27.3.tar.gz1.4 解压nginx
[root@elk91 ~]# tar xf nginx-1.27.3.tar.gz 1.5 配置nginx
[root@elk91 ~]# cd nginx-1.27.3/
[root@elk91 nginx-1.27.3]# 
[root@elk91 nginx-1.27.3]# ./configure --prefix=/softwares/nginx --with-http_ssl_module --with-http_v2_module --with-http_realip_module --without-http_rewrite_module --with-http_stub_status_module --without-http_gzip_module  --with-file-aio --with-stream --with-stream_ssl_module --with-stream_realip_module --add-module=/root/nginx-module-vts1.6 编译并安装nginx
make -j 2 && make install

2. 修改nginx的配置文件

[root@elk91 nginx-1.27.3]# vim /softwares/nginx/conf/nginx.conf
...
http {vhost_traffic_status_zone;upstream novacao-promethues {server 10.0.0.31:9090;}...server {...location / {root   html;# index  index.html index.htm;proxy_pass http://novacao-promethues;}location /nginx_status {vhost_traffic_status_display;vhost_traffic_status_display_format html;}}
}

 3. 启动NGINX

[root@elk91 nginx-1.27.3]# /softwares/nginx/sbin/nginx -t
nginx: the configuration file /softwares/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /softwares/nginx/conf/nginx.conf test is successful
[root@elk91 nginx-1.27.3]# 1.9 启动nginx
[root@elk91 nginx-1.27.3]# systemctl disable --now nginx
Synchronizing state of nginx.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable nginx
Removed /etc/systemd/system/multi-user.target.wants/nginx.service.
[root@elk91 nginx-1.27.3]# 
[root@elk91 nginx-1.27.3]# /softwares/nginx/sbin/nginx 
[root@elk91 nginx-1.27.3]# 

4. 访问nginx的状态页面

http://10.0.0.91/nginx_status/	
http://10.0.0.91/nginx_status/format/prometheus

5. 修改Prometheus的配置文件

[root@prometheus-server31 ~]# tail -6 /softwares/prometheus-2.53.4.linux-amd64/prometheus.yml - job_name: linux96-nginx-vts-modulesmetrics_path: "/nginx_status/format/prometheus"static_configs:- targets: - 10.0.0.91:80# 热加载
[root@prometheus-server31 ~]# curl -X POST http://10.0.0.31:9090/-/reload
[root@prometheus-server31 ~]# 

6. 验证监控结果并导入grafana模板

# 查看Prometheus
http://10.0.0.31:9090/targets?search=1.12 导入模板ID 
9785

方案2>安装nginx-vtx-exporter

# 前提也是需要编译好NGINX

1. 下载nginx-vtx-exporter 

[root@elk92 ~]# wget https://github.com/sysulq/nginx-vts-exporter/releases/download/v0.10.8/nginx-vtx-exporter_0.10.8_linux_amd64.tar.gz[root@elk92 ~]# tar xf nginx-vtx-exporter_0.10.8_linux_amd64.tar.gz  -C /usr/local/bin/ nginx-vtx-exporter
[root@elk92 ~]# 
[root@elk92 ~]# ll /usr/local/bin/nginx-vtx-exporter 
-rwxr-xr-x 1 1001 avahi 7950336 Jul 11  2023 /usr/local/bin/nginx-vtx-exporter*
[root@elk92 ~]# 

2. 运行nginx-vtx-exporter

[root@elk92 ~]# nginx-vtx-exporter -nginx.scrape_uri=http://10.0.0.91/nginx_status/format/json
2025/03/27 16:25:19 Starting nginx_vts_exporter (version=, branch=, revision=)
2025/03/27 16:25:19 Build context (go=go1.20.5, user=, date=)
2025/03/27 16:25:19 Starting Server at : :9913
2025/03/27 16:25:19 Metrics endpoint: /metrics
2025/03/27 16:25:19 Metrics namespace: nginx
2025/03/27 16:25:19 Scraping information from : http://10.0.0.91/nginx_status/format/json

3. 访问nginx exporter的WebUI

http://10.0.0.93:9913/metrics

4. 配置prometheus采集nginx数据

[root@prometheus-server31 ~]# tail -12 /softwares/prometheus-2.53.4.linux-amd64/prometheus.yml - job_name: linux96-nginx-vts-modulesmetrics_path: "/nginx_status/format/prometheus"static_configs:- targets: - 10.0.0.91:80- job_name: "linux96-nginx-vtx-exporter"static_configs:- targets: - 10.0.0.92:9913[root@prometheus-server31 ~]# 
[root@prometheus-server31 ~]# curl -X POST http://10.0.0.31:9090/-/reload

5. 验证监控结果并导入grafana模板

导入grafana模板  
2949

八、Prometheus监控redis

1. 部署Redis服务

1.拉取redis镜像 
[root@elk92 ~]# docker pull redis:7.4.2-alpine
7.4.2-alpine: Pulling from library/redis
...........
Digest: sha256:02419de7eddf55aa5bcf49efb74e88fa8d931b4d77c07eff8a6b2144472b6952
Status: Downloaded newer image for redis:7.4.2-alpine
docker.io/library/redis:7.4.2-alpine2.部署redis服务 
[root@elk92 ~]# docker run -d --network host --name redis-server redis:7.4.2-alpine 
14f6cd2c24f0735bb5bfd189730898b15380101dd11516ad2af3b7776a33a75a
[root@elk92 ~]# 
[root@elk92 ~]# ss -ntl | grep 6379
LISTEN 0      511               0.0.0.0:6379       0.0.0.0:*          
LISTEN 0      511                  [::]:6379          [::]:*          
[root@elk92 ~]# 

2. 部署redis-exporter

wget https://github.com/oliver006/redis_exporter/releases/download/v1.69.0/redis_exporter-v1.69.0.linux-amd64.tar.gz4.解压软件包 
[root@elk93 ~]# tar xf redis_exporter-v1.69.0.linux-amd64.tar.gz -C /usr/local/bin/ redis_exporter-v1.69.0.linux-amd64/redis_exporter --strip-components=1
[root@elk93 ~]# 
[root@elk93 ~]# ll /usr/local/bin/redis_exporter 
-rwxr-xr-x 1 mysql fwupd-refresh 9638072 Mar 15 12:41 /usr/local/bin/redis_exporter*
[root@elk93 ~]# 

3. 运行redis exporter 

[root@elk93 ~]# redis_exporter -redis.addr  redis://10.0.0.92:6379
INFO[0000] Redis Metrics Exporter v1.69.0    build date: 2025-03-15-04:40:26    sha1: 71dbe37fb14a4ae2537c1790a239dc1e568ffba5    Go: go1.24.1    GOOS: linux    GOARCH: amd64 
INFO[0000] Providing metrics at :9121/metrics  

4. 访问redis exporter的WebUI

http://10.0.0.93:9121/metrics

5. 配置Prometheus监控redis容器指标 

[root@prometheus-server31 ~]# tail -7 /softwares/prometheus-2.53.4.linux-amd64/prometheus.yml - job_name: linux96-redis-exporterstatic_configs:- targets: - 10.0.0.93:9121 [root@prometheus-server31 ~]# 
[root@prometheus-server31 ~]# curl -X POST http://10.0.0.31:9090/-/reload
[root@prometheus-server31 ~]# 

6. 验证监控结果并导入grafana模板

验证Prometheus配置是否生效
http://10.0.0.31:9090/targets?search=9.grafana导入模板ID
763
14091 
14615

彩蛋、 grafana安装插件

1. 图表报错提示

Panel plugin not found: natel-discrete-panel说明需要安装对应的插件，使用grafana客户端工具安装即可。

 2. 安装指定插件

[root@prometheus-server31 ~]# ll /var/lib/grafana/
total 5236
drwxr-xr-x  5 grafana grafana    4096 Mar 27 16:53 ./
drwxr-xr-x 61 root    root       4096 Mar 26 11:59 ../
drwxr-x---  3 grafana grafana    4096 Mar 26 12:00 alerting/
drwx------  2 grafana grafana    4096 Mar 26 12:00 csv/
-rw-r-----  1 grafana grafana 5337088 Mar 27 16:53 grafana.db
drwx------  2 grafana grafana    4096 Mar 26 12:00 png/
[root@prometheus-server31 ~]# 
[root@prometheus-server31 ~]# grafana-cli plugins install natel-discrete-panel
✔ Downloaded and extracted natel-discrete-panel v0.1.1 zip successfully to /var/lib/grafana/plugins/natel-discrete-panelPlease restart Grafana after installing or removing plugins. Refer to Grafana documentation for instructions if necessary.[root@prometheus-server31 ~]# 
[root@prometheus-server31 ~]# ll /var/lib/grafana/
total 5240
drwxr-xr-x  6 grafana grafana    4096 Mar 27 16:56 ./
drwxr-xr-x 61 root    root       4096 Mar 26 11:59 ../
drwxr-x---  3 grafana grafana    4096 Mar 26 12:00 alerting/
drwx------  2 grafana grafana    4096 Mar 26 12:00 csv/
-rw-r-----  1 grafana grafana 5337088 Mar 27 16:53 grafana.db
drwxr-xr-x  3 root    root       4096 Mar 27 16:56 plugins/  # 安装后会将插件存储在本地。
drwx------  2 grafana grafana    4096 Mar 26 12:00 png/
[root@prometheus-server31 ~]# 

3. 重启grafana使得插件生效 

[root@prometheus-server31 ~]# systemctl restart grafana-server.service 
[root@prometheus-server31 ~]# 

九、Prometheus监控MongoDB

1. 部署mongoDB服务

1.下载mongoDB镜像 
[root@elk92 ~]# docker pull mongo:8.0.6-noble
8.0.6-noble: Pulling from library/mongo
.................
Status: Downloaded newer image for mongo:8.0.6-noble
docker.io/library/mongo:8.0.6-noble
[root@elk92 ~]# 2.部署mongoDB服务
[root@elk92 ~]# docker run -d --name mongodb-server --network host mongo:8.0.6-noble
4b0f00dea78bb571c216c344984ced026c1210c94db147fdc9e32f549e3135de
[root@elk92 ~]# 
[root@elk92 ~]# docker ps -l
CONTAINER ID   IMAGE               COMMAND                  CREATED         STATUS        PORTS     NAMES
4b0f00dea78b   mongo:8.0.6-noble   "docker-entrypoint.s…"   2 seconds ago   Up 1 second             mongodb-server
[root@elk92 ~]# 
[root@elk92 ~]# ss -ntl | grep 27017
LISTEN 0      4096              0.0.0.0:27017      0.0.0.0:*          
[root@elk92 ~]# 

2. 部署MongoDB的exporter

https://github.com/percona/mongodb_exporter/releases/download/v0.43.1/mongodb_exporter-0.43.1.linux-amd64.tar.gz解压软件包 
[root@elk91 ~]# tar xf mongodb_exporter-0.43.1.linux-amd64.tar.gz -C /usr/local/bin/ mongodb_exporter-0.43.1.linux-amd64/mongodb_exporter --strip-components=1
[root@elk91 ~]# 
[root@elk91 ~]# ll /usr/local/bin/mongodb_exporter 
-rwxr-xr-x 1 1001 geoclue 20467864 Dec 13 20:10 /usr/local/bin/mongodb_exporter*

3. 准备MongoDB测试数据

[root@elk91 ~]# vim insert_data.js
// 创建数据库和集合
db = db.getSiblingDB('testdb');// 插入用户数据（10条）
for (let i = 1; i <= 10; i++) {db.users.insertOne({name: `user${i}`,email: `user${i}@example.com`,age: Math.floor(Math.random() * 50 + 18),created_at: new Date()});
}// 插入订单数据（10条）
for (let i = 1; i <= 10; i++) {db.orders.insertOne({user_id: i,product: `product${Math.floor(Math.random() * 5)}`,amount: Math.random() * 100,status: ["pending", "completed", "cancelled"][Math.floor(Math.random() * 3)],created_at: new Date()});
}// 插入产品数据（5条）
const products = ["product0", "product1", "product2", "product3", "product4"];
products.forEach((product, index) => {db.products.insertOne({name: product,price: Math.random() * 50 + 20,stock: Math.floor(Math.random() * 100)});
});// 创建索引（可选）
db.users.createIndex({ email: 1 });
db.orders.createIndex({ user_id: 1 });################################################################## 将脚本复制到容器内
docker cp insert_data.js mongodb-server:/tmp/insert_data.js# 执行脚本（使用 mongosh）
docker exec -it mongodb-server mongosh --quiet /tmp/insert_data.js

4. 运行mongodb-exporter 

mongodb_exporter --mongodb.uri=mongodb://10.0.0.92:27017 --log.level=info --collector.dbstats --collector.diagnosticdata --collector.replicasetstatus --collector.topmetrics --collector.currentopmetrics --collector.exporter-metrics --collector.collstats --collector.indexstats --mongodb.collstats-colls=testdb.users,testdb.orders,testdb.products --mongodb.indexstats-colls=testdb.users,testdb.orders#####################################3验证mongoDB-exporter的WebUI
http://10.0.0.91:9216/metrics

5. 配置Prometheus监控mongoDB容器

[root@prometheus-server31 ~]# tail -7 /softwares/prometheus-2.53.4.linux-amd64/prometheus.yml - job_name: linux96-mongodb-exporterstatic_configs:- targets: - 10.0.0.91:9216[root@prometheus-server31 ~]#
[root@prometheus-server31 ~]# curl -X POST http://10.0.0.31:9090/-/reload

6. 验证监控结果并导入grafana模板

grafana导入模板ID
16504当然如果实在找不到，就只能自己定义Dashboard。比如: "mongodb_up"

十、Prometheus监控ElasticSearch集群

保障ES集群状态正常
[root@elk91 ~]# curl https://10.0.0.91:9200/_cat/nodes  -u elastic:123456 -k
10.0.0.93 59 33 45 7.11 2.18 0.75 cdfhilmrstw - elk93
10.0.0.91 78 44  4 0.73 0.51 0.29 cdfhilmrstw * elk91
10.0.0.92 54 31 34 1.49 0.51 0.18 cdfhilmrstw - elk92

1. 下载elasticsearch_exporter服务插件

[root@node-exporter41 ~]# wget https://github.com/prometheus-community/elasticsearch_exporter/releases/download/v1.9.0/elasticsearch_exporter-1.9.0.linux-amd64.tar.gz
2.解压软件包
[root@node-exporter41 ~]# tar xf elasticsearch_exporter-1.9.0.linux-amd64.tar.gz -C /usr/local/bin/ elasticsearch_exporter-1.9.0.linux-amd64/elasticsearch_exporter   --strip-components=1
[root@node-exporter41 ~]# ll /usr/local/bin/
total 14728
drwxr-xr-x  2 root root              4096 Mar 27 02:42 ./
drwxr-xr-x 10 root root              4096 Feb 16  2024 ../
-rwxr-xr-x  1 1001 fwupd-refresh 15069336 Mar  3 10:01 elasticsearch_exporter*

2. 启动elasticsearch-exporter

[root@node-exporter41 ~]# elasticsearch_exporter --es.uri="https://elastic:123456@10.0.0.91:9200"  --es.ssl-skip-verify
time=2025-03-27T02:42:46.739Z level=INFO source=clusterinfo.go:211 msg="triggering initial cluster info call"
time=2025-03-27T02:42:46.739Z level=INFO source=clusterinfo.go:182 msg="providing consumers with updated cluster info label"
time=2025-03-27T02:42:46.750Z level=INFO source=main.go:244 msg="started cluster info retriever" interval=5m0s
.....
time=2025-03-27T02:14:19.093Z level=INFO source=tls_config.go:350 msg="TLS is disabled." http2=false address=[::]:9114

3. 配置Prometheus的job服务发现

[root@prometheus-server31 ~]# tail -5  /softwares/prometheus-2.53.4.linux-amd64/prometheus.yml- job_name: linux96-es-exporterstatic_configs:- targets: - 10.0.0.41:91145.热加载Prometheus
[root@prometheus-server31 ~]# curl -X POST http://10.0.0.31:9090/-/reload
[root@prometheus-server31 ~]# 

 4. 验证监控结果并导入grafana模板

验证Prometheus配置是否生效 
http://10.0.0.31:9090/targets?search=7.grafana导入模板ID
14191
9746

十一、小结

Prometheus监控服务分为三种情况。
1. 第一种情况
在Prometheus官网里面提供了exporter的组件，我们只需要部署好组件然后把要监控的服务使用http(s)的方式暴露出去即可，然后Prometheus修改配置文件，配置好服务发现，最后Grafana套用官方的模版ID即可。（当然也可以自己去制作dashboard展示）。
# Prometheus的exporter官网
https://prometheus.io/download/
https://prometheus.io/docs/instrumenting/exporters/
# Grafana的模版官网
https://grafana.com/dashboards

2. 第二种情况
有些服务自己本身就支持云原生的Prometheus监控，在自己的配置文件中就支持metrics的接口暴露。只需要在配置文件中修改或者取消注释好配置即可，然后Prometheus修改配置文件，配置好服务发现，最后Grafana套用官方的模版ID即可。（当然也可以自己去制作dashboard展示）。
例如：nacos，zookeeper

3. 第三种情况
第一二种情况都不能达到预期的标准，那么他就会有第三种服务为其metrics提供接口。
例如：docker的cAdvisor

希望这篇文章可以帮助您在生产环境中高效率的解决问题。

最后有关K8S的如何监控可以看我的基于 Operator 部署 Prometheus 实现 K8S 监控-CSDN博客

这篇文章！