Prometheus监控主机，Grafana成图

全部使用官方 GitHub 源的部署方案下载链接来自官方无需镜像。官方下载地址汇总组件 官方下载地址Node Exporterhttps://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gzPrometheushttps://github.com/prometheus/prometheus/releases/download/v3.5.0/prometheus-3.5.0.linux-amd64.tar.gzAlertManagerhttps://github.com/prometheus/alertmanager/releases/download/v0.28.0/alertmanager-0.28.0.linux-amd64.tar.gzGrafanahttps://dl.grafana.com/oss/release/grafana-11.6.0.linux-amd64.tar.gz一、部署 Node Exporter官方源bash创建工作目录sudo mkdir -p /data/node_exporter/logs下载官方版本GitHub Releasecd /tmpwget https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gztar xvf node_exporter-1.8.2.linux-amd64.tar.gzsudo cp node_exporter-1.8.2.linux-amd64/node_exporter /data/node_exporter/验证/data/node_exporter/node_exporter --version启动脚本bashsudo tee /data/node_exporter/start.sh ‘EOF’#!/bin/bashcd /data/node_exporternohup ./node_exporter–web.listen-address“0.0.0.0:9100”–collector.disable-defaults–collector.cpu–collector.meminfo–collector.filesystem–collector.diskstats–collector.netdev–collector.loadavg–collector.time–collector.uname \logs/node_exporter.log 21 echo $! node_exporter.pidecho “Node Exporter started with PID: $(cat node_exporter.pid)”EOFsudo tee /data/node_exporter/stop.sh ‘EOF’#!/bin/bashif [ -f /data/node_exporter/node_exporter.pid ]; thenkill $(cat /data/node_exporter/node_exporter.pid)rm -f /data/node_exporter/node_exporter.pidecho “Node Exporter stopped”elsepkill -f “node_exporter.*9100”fiEOFsudo chmod x /data/node_exporter/*.shsudo /data/node_exporter/start.sh验证sleep 2curl http://localhost:9100/metrics | head -5二、部署 Prometheus官方源bash创建工作目录sudo mkdir -p /data/prometheus/{data,rules,logs}下载官方版本cd /tmpwget https://github.com/prometheus/prometheus/releases/download/v3.5.0/prometheus-3.5.0.linux-amd64.tar.gztar xvf prometheus-3.5.0.linux-amd64.tar.gzsudo cp prometheus-3.5.0.linux-amd64/prometheus /data/prometheus/sudo cp prometheus-3.5.0.linux-amd64/promtool /data/prometheus/sudo cp -r prometheus-3.5.0.linux-amd64/consoles /data/prometheus/sudo cp -r prometheus-3.5.0.linux-amd64/console_libraries /data/prometheus/设置权限sudo chmod x /data/prometheus/prometheus配置文件bashsudo tee /data/prometheus/prometheus.yml ‘EOF’global:scrape_interval: 15sevaluation_interval: 15sexternal_labels:monitor: ‘vm-monitor’alerting:alertmanagers:- static_configs:- targets: []rule_files: []scrape_configs:job_name: ‘prometheus’static_configs:targets: [‘localhost:9090’]labels:service: ‘prometheus’job_name: ‘node_exporter’scrape_interval: 30sstatic_configs:targets:‘localhost:9100’labels:environment: ‘production’EOF启动脚本bashsudo tee /data/prometheus/start.sh ‘EOF’#!/bin/bashcd /data/prometheusnohup ./prometheus–config.file/data/prometheus/prometheus.yml–storage.tsdb.path/data/prometheus/data–storage.tsdb.retention.time30d–web.enable-lifecycle \logs/prometheus.log 21 echo $! prometheus.pidecho “Prometheus started with PID:(catprometheus.pid)echoWebUI:http://(cat prometheus.pid) echo Web UI: http://(catprometheus.pid)echoWebUI:http://(hostname -I | awk ‘{print $1}’):9090”EOFsudo tee /data/prometheus/stop.sh ‘EOF’#!/bin/bashif [ -f /data/prometheus/prometheus.pid ]; thenkill $(cat /data/prometheus/prometheus.pid)rm -f /data/prometheus/prometheus.pidecho “Prometheus stopped”elsepkill -f “prometheus.*config.file”fiEOFsudo chmod x /data/prometheus/*.shsudo /data/prometheus/start.sh验证sleep 3curl http://localhost:9090/-/healthy三、部署 Grafana官方源bash创建工作目录sudo mkdir -p /data/grafana/{data,logs,plugins}下载官方版本cd /tmpwget https://dl.grafana.com/oss/release/grafana-11.6.0.linux-amd64.tar.gztar xvf grafana-11.6.0.linux-amd64.tar.gzsudo cp -r grafana-11.6.0/* /data/grafana/配置文件sudo tee /data/grafana/conf/custom.ini ‘EOF’[paths]data /data/grafana/datalogs /data/grafana/logsplugins /data/grafana/plugins[server]http_port 3000[auth]disable_login_form false[users]allow_sign_up falseEOF启动脚本bashsudo tee /data/grafana/start.sh ‘EOF’#!/bin/bashcd /data/grafananohup ./bin/grafana-server–config/data/grafana/conf/custom.ini–homepath/data/grafana \logs/grafana.log 21 echo $! grafana.pidecho “Grafana started with PID:(catgrafana.pid)echoGrafanaUI:http://(cat grafana.pid) echo Grafana UI: http://(catgrafana.pid)echoGrafanaUI:http://(hostname -I | awk ‘{print $1}’):3000 (admin/admin)”EOFsudo tee /data/grafana/stop.sh ‘EOF’#!/bin/bashif [ -f /data/grafana/grafana.pid ]; thenkill $(cat /data/grafana/grafana.pid)rm -f /data/grafana/grafana.pidecho “Grafana stopped”elsepkill -f “grafana-server”fiEOFsudo chmod x /data/grafana/*.shsudo /data/grafana/start.sh四、统一管理脚本bashsudo tee /usr/local/bin/monitor-control ‘EOF’#!/bin/bashSERVICES“node_exporter prometheus grafana”case “$1” instart)echo “Starting all monitoring services…”/data/node_exporter/start.sh 2/dev/null || echo “Node Exporter already running”sleep 2/data/prometheus/start.sh 2/dev/null || echo “Prometheus already running”sleep 2/data/grafana/start.sh 2/dev/null || echo “Grafana already running”echo “All services started”;;stop)echo “Stopping all monitoring services…”/data/node_exporter/stop.sh 2/dev/null/data/prometheus/stop.sh 2/dev/null/data/grafana/stop.sh 2/dev/nullecho “All services stopped”;;status)echo “ Node Exporter ps aux | grep -v grep | grep “node_exporter.*9100” echo “Running” || echo “Stopped”echo Prometheus ps aux | grep -v grep | grep “prometheus.*config” echo “Running” || echo “Stopped”echo Grafana ”ps aux | grep -v grep | grep “grafana-server” echo “Running” || echo “Stopped”;;restart)$0 stopsleep 3$0 start;;*)echo “Usage: $0 {start|stop|restart|status}”exit 1;;esacEOFsudo chmod x /usr/local/bin/monitor-control五、测试验证bash启动所有服务sudo monitor-control start查看状态sudo monitor-control status测试各服务curl http://localhost:9090/-/healthy # Prometheuscurl http://localhost:9100/metrics | head # Node Exportercurl http://localhost:3000/api/health # Grafana六、添加其他虚拟机在其他虚拟机上部署 Node Exporterbash在每一台被监控虚拟机上执行mkdir -p /data/node_exporter/logscd /tmpwget https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gztar xvf node_exporter-1.8.2.linux-amd64.tar.gzcp node_exporter-1.8.2.linux-amd64/node_exporter /data/node_exporter/cd /data/node_exporternohup ./node_exporter --web.listen-address“0.0.0.0:9100” logs/node_exporter.log 21 然后在 Prometheus 配置中添加新目标并热加载bash编辑 /data/prometheus/prometheus.yml在 targets 中添加新 IPcurl -X POST http://localhost:9090/-/reload六、Grafana添加Prometheus数据源这里最好是填写http://127.0.0.1:9090这样不会识别到ipv6.七、给数据源添加面板八、配置告警AlertManager3.1 下载并安装 AlertManagerbash创建工作目录mkdir -p /data/alertmanager/{data,logs}下载官方版本cd /tmpwget https://github.com/prometheus/alertmanager/releases/download/v0.28.0/alertmanager-0.28.0.linux-amd64.tar.gztar xvf alertmanager-0.28.0.linux-amd64.tar.gzcp alertmanager-0.28.0.linux-amd64/alertmanager /data/alertmanager/cp alertmanager-0.28.0.linux-amd64/amtool /data/alertmanager/chmod x /data/alertmanager/alertmanager3.2 配置 AlertManagerbashsudo tee /data/alertmanager/alertmanager.yml ‘EOF’global:resolve_timeout: 5msmtp_smarthost: ‘smtp.example.com:587’ # 配置邮件服务器smtp_from: ‘alertmanagerexample.com’smtp_auth_username: ‘your-username’smtp_auth_password: ‘your-password’route:group_by: [‘alertname’, ‘severity’]group_wait: 30sgroup_interval: 5mrepeat_interval: 4hreceiver: ‘default-receiver’routes:- match:severity: criticalreceiver: critical-receiverreceivers:name: ‘default-receiver’email_configs:to: ‘ops-teamexample.com’name: ‘critical-receiver’email_configs:to: ‘oncallexample.com’webhook_configs:url: ‘https://your-webhook-url’ # 可对接钉钉/企微/飞书inhibit_rules:source_match:severity: ‘critical’target_match:severity: ‘warning’equal: [‘instance’, ‘alertname’]EOF3.3 启动 AlertManagerbashsudo tee /data/alertmanager/start.sh ‘EOF’#!/bin/bashcd /data/alertmanagernohup ./alertmanager–config.file/data/alertmanager/alertmanager.yml–web.listen-address“0.0.0.0:9093”–storage.path/data/alertmanager/data \logs/alertmanager.log 21 echo $! alertmanager.pidecho “AlertManager started with PID:(catalertmanager.pid)echoAlertManagerUI:http://(cat alertmanager.pid) echo AlertManager UI: http://(catalertmanager.pid)echoAlertManagerUI:http://(hostname -I | awk ‘{print $1}’):9093”EOFsudo tee /data/alertmanager/stop.sh ‘EOF’#!/bin/bashif [ -f /data/alertmanager/alertmanager.pid ]; thenkill $(cat /data/alertmanager/alertmanager.pid)rm -f /data/alertmanager/alertmanager.pidelsepkill -f “alertmanager.*config.file”fiEOFsudo chmod x /data/alertmanager/*.shsudo /data/alertmanager/start.sh3.4 配置 Prometheus 对接 AlertManager编辑 /data/prometheus/prometheus.yml添加以下配置yaml告警配置alerting:alertmanagers:- static_configs:- targets: [‘localhost:9093’] # AlertManager 地址告警规则文件rule_files:“rules/*.yml”3.5 创建告警规则bashmkdir -p /data/prometheus/rulessudo tee /data/prometheus/rules/node_alerts.yml ‘EOF’groups:name: node_alertsinterval: 30srules:1. 节点宕机告警alert: InstanceDownexpr: up{job“node_exporter”} 0for: 1mlabels:severity: criticalannotations:summary: “实例 {{ $labels.instance }} 已宕机”description: “{{ $labels.instance }} 已宕机超过 1 分钟”2. CPU 使用率过高告警alert: HighCPUUsageexpr: 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode“idle”}[5m])) * 100) 80for: 5mlabels:severity: warningannotations:summary: “{{ $labels.instance }} CPU 使用率过高”description: “CPU 使用率已超过 80%当前值: {{ $value }}%”3. 内存使用率过高告警alert: HighMemoryUsageexpr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 90for: 5mlabels:severity: warningannotations:summary: “{{ $labels.instance }} 内存使用率过高”description: “内存使用率已超过 90%当前值: {{ $value }}%”4. 磁盘空间不足告警alert: DiskSpaceLowexpr: (1 - node_filesystem_avail_bytes{mountpoint“/”} / node_filesystem_size_bytes{mountpoint“/”}) * 100 85for: 5mlabels:severity: warningannotations:summary: “{{ $labels.instance }} 磁盘空间不足”description: “磁盘使用率已超过 85%当前值: {{ $value }}%”5. 系统负载过高告警alert: HighSystemLoadexpr: node_load5 / count by (instance) (node_cpu_seconds_total{mode“idle”}) 2for: 5mlabels:severity: warningannotations:summary: “{{ $labels.instance }} 系统负载过高”description: “5分钟平均负载超过 CPU 核心数的 2 倍”EOF3.6 加载配置并验证bash热加载 Prometheus 配置curl -X POST http://localhost:9090/-/reload验证告警规则是否加载成功curl http://localhost:9090/api/v1/rules | jq ‘.data.groups[].name’访问告警页面http://服务器IP:9090/alerts