Logo

菜单

ithuang
ithuang
发布于 2025-09-26 / 4 阅读
0

Prometheus 告警配置与数据库监控

#Prometheus#监控#Linux

Prometheus 告警配置与数据库监控

一、告警配置

1. 告警规则

一般步骤：

  1. 在 Prometheus Web 编辑好 PromQL 查询
  2. 配置到 Prometheus，并重载生效
  3. 测试告警

http://192.168.88.106:9090/

注意：确保被监控的机器 node_exporter 是否正常运行

nohup node_exporter &

编辑查询

Prometheus 告警配置与数据库监控1.png

配置 Prometheus 告警

# 为告警规则单独创建配置文件
cat > /usr/local/prometheus/node.alerts.yml <<EOF
groups:
- name: test
  rules:

  # 实例 1 分钟无响应
  - alert: InstanceDown
    expr: up == 0
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: "实例 {{ $labels.instance }} 宕机"
      description: "{{ $labels.job }} {{ $labels.instance }} 已经 1 分钟没响应了！"
EOF

# 编辑 prometheus 主配置文件
vim /usr/local/prometheus/prometheus.yml
...
# 告诉 prometheus 加载 node.alerts.yml 这个 rules 文件
rule_files:
  - "/usr/local/prometheus/node.alerts.yml"
...

# 重载生效
ps aux|grep prometheus
kill -1 pid 

或者

# 重启 prometheus
systemctl restart prometheus
[root@server1 ~]# cat > /usr/local/prometheus/node.alerts.yml <<EOF
groups:
- name: test
  rules:

  # 实例 1 分钟无响应
  - alert: InstanceDown
    expr: up == 0
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: "实例 {{ $labels.instance }} 宕机"
      description: "{{ $labels.job }} {{ $labels.instance }} 已经 1 分钟没响应了！"
EOF
[root@server1 ~]# vim /usr/local/prometheus/prometheus.yml
[root@server1 ~]# cat /usr/local/prometheus/prometheus.yml | grep *.yml
[root@server1 ~]# cat /usr/local/prometheus/prometheus.yml | grep .yml
  # - "first_rules.yml"
  # - "second_rules.yml"
  - "/usr/local/prometheus/node.alerts.yml"
[root@server1 ~]# systemctl restart prometheus
[root@server1 ~]#

Prometheus 告警配置与数据库监控2.png
Prometheus 告警配置与数据库监控3.png
Prometheus 告警配置与数据库监控4.png
目前 Prometheus 监控系统处于完全健康的状态：

  • 采集目标：所有被监控的节点（192.168.88.101:9100、192.168.88.102:9100）以及 Prometheus 自身，都处于 UP 状态，指标采集正常。
  • 告警规则：InstanceDown 告警规则运行正常，当前状态为 INACTIVE，没有任何实例触发宕机告警。
  • 服务状态：Prometheus 服务运行稳定，配置和规则都已正确加载并生效。

一句话总结：现在一切正常，没有任何需要处理的告警或异常。

测试节点

# 测试 12 节点
curl -v http://192.168.80.12:9100/metrics
# 测试 13 节点
curl -v http://192.168.80.13:9100/metrics

制造故障

[root@server2 ~]# ps aux | grep node_exporter
root        1508  0.1  0.6 1276180 25896 pts/0   Sl   09:17   0:00 node_exporter
root        1515  0.0  0.0   6640  2176 pts/0    S+   09:25   0:00 grep --color=auto node_exporter
[root@server2 ~]# kill -9 1508
[root@server2 ~]# ps aux | grep node_exporter
root        1517  0.0  0.0   6640  2176 pts/0    S+   09:25   0:00 grep --color=auto node_exporter
[1]+  已杀死               nohup node_exporter
[root@server2 ~]#

Prometheus 告警配置与数据库监控5.png

修复故障

[root@server2 ~]# nohup node_exporter &
[1] 1518
[root@server2 ~]# nohup: 忽略输入并把输出追加到'nohup.out'

[root@server2 ~]# ps aux | grep node_exporter
root        1518  0.0  0.5 1275020 19452 pts/0   Sl   09:26   0:00 node_exporter
root        1524  0.0  0.0   6640  2176 pts/0    S+   09:27   0:00 grep --color=auto node_exporter
[root@server2 ~]#

Prometheus 告警配置与数据库监控6.png

2. Prometheus 指向 Alertmanager

修改 prometheus 配置指向 alertmanager，通过 alertmanager 发送告警

vim /usr/local/prometheus/prometheus.yml

...
# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          - 192.168.80.11:9093
...

systemctl restart prometheus
[root@server1 ~]# vim /usr/local/prometheus/prometheus.yml
[root@server1 ~]# systemctl restart prometheus
[root@server1 ~]# cat /usr/local/prometheus/prometheus.yml | grep 9093
          # - alertmanager:9093
          - 192.168.80.11:9093
[root@server1 ~]#

查看生效

Prometheus 告警配置与数据库监控7.png

制造故障

[root@server2 ~]# ps aux | grep node_exporter
root        1518  0.1  0.6 1276436 25512 pts/0   Sl   09:26   0:00 node_exporter
root        1529  0.0  0.0   6640  2176 pts/0    S+   09:31   0:00 grep --color=auto node_exporter
[root@server2 ~]# kill -9 1518
[root@server2 ~]# ps aux | grep node_exporter
root        1531  0.0  0.0   6640  2176 pts/0    S+   09:32   0:00 grep --color=auto node_exporter
[1]+  已杀死               nohup node_exporter
[root@server2 ~]#

Prometheus 告警配置与数据库监控8.png

查看 alertmanager web，告警已经发送到 alertmanager

Prometheus 告警配置与数据库监控9.png

修复故障

[root@server2 ~]# ps aux | grep node_exporter
root        1533  0.0  0.0   6640  2176 pts/0    S+   09:33   0:00 grep --color=auto node_exporter
[root@server2 ~]# nohup node_exporter &
[1] 1534
[root@server2 ~]# nohup: 忽略输入并把输出追加到'nohup.out'

[root@server2 ~]# ps aux | grep node_exporter
root        1534  0.2  0.4 1274764 17632 pts/0   Sl   09:34   0:00 node_exporter
root        1540  0.0  0.0   6640  2176 pts/0    S+   09:34   0:00 grep --color=auto node_exporter
[root@server2 ~]#

Prometheus 告警配置与数据库监控10.png

Prometheus 告警配置与数据库监控11.png

3. 静默告警

告警属于预期之内，比如例行维护，业务量波动；此时需要暂时让告警不发出来，所以需要静默告警。

创建静默

点击右上角 New Silence

筛选器的 label 可以在 Prometheus Alerts 页面查看

Prometheus 告警配置与数据库监控12.png

alertname="InstanceDown"
env="test"
instance="192.168.88.101:9100"
job="node"
severity="critical"

填写过滤器，填一条点一下加号，才能生效；同时，还要注意填生效时间范围。

Prometheus 告警配置与数据库监控13.png

填写 Creator 和 Comment，点击 Preview Alerts

注意：可以制造故障，以便观察配置是否生效。

Prometheus 告警配置与数据库监控14.png
Prometheus 告警配置与数据库监控15.png

编辑静默

Prometheus 告警配置与数据库监控16.png

点击 Edit 为编辑，进入后和 创建编辑方法一致

点击 Expire 是让静默过期，然后点击 Confirm
Prometheus 告警配置与数据库监控17.png
过期后可以在 Expired 标签查看，自动过期的也会在这里暂时。

静默时间区间可以设置将来的时间，将来的静默会展示在 Pending 标签。

4. 告警恢复

# 在 12 上启动 node_exporter
nohup node_exporter &

Prometheus 告警配置与数据库监控18.png

5. 邮件告警

现在告警只到 Alertmanager，得人类主动查看才知道，怎么通知到人类？

获取登陆凭证，以 qq邮箱为例

Prometheus 告警配置与数据库监控19.png

配置 Alertmanager

vim /usr/local/alertmanager/alertmanager.yml
[root@server1 ~]# vim /usr/local/alertmanager/alertmanager.yml
[root@server1 ~]# cat /usr/local/alertmanager/alertmanager.yml
global:
  # 邮件配置
  smtp_smarthost: 'smtp.qq.com:587'
  smtp_from: '2585785635@qq.com'
  smtp_auth_username: '2585785635@qq.com'
  smtp_auth_password: 'tpzkercrzxhtebfb'   # 必须替换为重新生成的授权码
  smtp_require_tls: true            # 使用587端口的TLS

#新增邮件接收者和子路由
route:
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 1h
  receiver: 'default.all'          # 默认接收者，邮件+webhook
  
  # 子路由：可以根据需要保留/调整
  routes:
    - matchers:
        - job=~"node"
      receiver: 'system.all'      # node告警特殊处理，同样邮件+webhook

receivers:
  # 默认接收者：同时发送邮件和webhook
  - name: 'default.all'
    email_configs:
      - to: '2585785635@qq.com'
        send_resolved: true    # 发送告警恢复通知
    webhook_configs:
      - url: 'http://127.0.0.1:5001/'

  # 系统告警接收者（node类），同样邮件+webhook
  - name: 'system.all'
    email_configs:
      - to: '2585785635@qq.com'
        send_resolved: true
    webhook_configs:
      - url: 'http://127.0.0.1:5001/'

# 保留：原来的抑制规则（critical告警抑制warning告警，很实用）
inhibit_rules:
  - source_matchers: [severity="critical"]
    target_matchers: [severity="warning"]
    equal: [alertname, dev, instance]
[root@server1 ~]# systemctl restart alertmanager
[root@server1 ~]# systemctl status alertmanager

制造故障

[root@server2 ~]# ps aux | grep node_exporter
root        1633  0.1  0.6 1276180 23588 pts/0   Sl   11:46   0:00 node_exporter
root        1638  0.0  0.0   6640  2176 pts/0    S+   11:47   0:00 grep --color=auto node_exporter
[root@server2 ~]# kill -9 1633
[root@server2 ~]# ps aux | grep node_exporter
root        1640  0.0  0.0   6640  2176 pts/0    S+   11:47   0:00 grep --color=auto node_exporter
[1]+  已杀死               nohup node_exporter
[root@server2 ~]#
[root@server3 ~]# ps aux | grep node_exporter
root        1418  0.1  1.4 1276436 26592 pts/0   Sl   09:17   0:11 node_exporter
root        1498  0.0  0.1   6640  2176 pts/0    S+   11:52   0:00 grep --color=auto node_exporter
[root@server3 ~]# kill -9 1418
[root@server3 ~]# ps aux | grep node_exporter
root        1500  0.0  0.1   6640  2176 pts/0    S+   11:52   0:00 grep --color=auto node_exporter
[1]+  已杀死               nohup node_exporter
[root@server3 ~]#

Prometheus 告警配置与数据库监控20.png
Prometheus 告警配置与数据库监控21.png

查看邮箱，收到告警
注意：如果长时间没收到邮件告警，可以考虑换一个邮箱！如果一直收不到邮件，可能是被邮箱平台方禁用了！

Prometheus 告警配置与数据库监控22.png
Prometheus 告警配置与数据库监控23.png

告警恢复

Prometheus 告警配置与数据库监控24.png

二、监控 MySQL

说明：选择其中一台被监控的机器来安装MySQL 8.0

1. mysqld_exporter 安装

下载安装包

https://prometheus.io/download/

Prometheus 告警配置与数据库监控26.png

1.1. 安装 mysqld_exporter

注意：在 MySQL 所在机器上操作！

[root@server2 ~]# wget https://github.com/prometheus/mysqld_exporter/releases/download/v0.19.0/mysqld_exporter-0.19.0.linux-amd64.tar.gz
[root@server2 ~]# ls
anaconda-ks.cfg  mysqld_exporter-0.19.0.linux-amd64.tar.gz
init_centos9.sh  nohup.out
[root@server2 ~]# tar -xvf mysqld_exporter-0.19.0.linux-amd64.tar.gz
mysqld_exporter-0.19.0.linux-amd64/
mysqld_exporter-0.19.0.linux-amd64/LICENSE
mysqld_exporter-0.19.0.linux-amd64/mysqld_exporter
mysqld_exporter-0.19.0.linux-amd64/NOTICE
[root@server2 ~]# ls mysqld_exporter-0.19.0.linux-amd64
LICENSE  mysqld_exporter  NOTICE
[root@server2 ~]# mv mysqld_exporter-0.19.0.linux-amd64/mysqld_exporter /usr/local/bin/
[root@server2 ~]# ls /usr/local/bin/
mysqld_exporter  node_exporter
[root@server2 ~]#

1.2. 配置

注意：在 MySQL 所在机器上操作！

# 登录 MySQL
mysql -uroot -pMySQL@666

# 创建数据库用户并授权
CREATE USER 'mysqld_exporter'@'localhost' IDENTIFIED BY 'MySQL@666';
GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'mysqld_exporter'@'localhost';
FLUSH PRIVILEGES;
EXIT;

# 特意创建隐藏配置文件
cat > /etc/.my.cnf <<EOF
[client]
user=mysqld_exporter
password=MySQL@666
EOF

可以直接执行

nohup mysqld_exporter --config.my-cnf /etc/.my.cnf &

这里设置 Systemd 服务：
创建服务文件 /etc/systemd/system/mysqld_exporter.service

[root@server2 ~]# vim /etc/systemd/system/mysqld_exporter.service
[root@server2 ~]# cat /etc/systemd/system/mysqld_exporter.service
[Unit]
Description=Prometheus MySQL Exporter
After=network.target

[Service]
User=root
Type=simple
ExecStart=/usr/local/bin/mysqld_exporter --config.my-cnf=/etc/.my.cnf

[Install]
WantedBy=multi-user.target

随后启动服务并查看状态

[root@server2 ~]# systemctl daemon-reload
[root@server2 ~]# systemctl restart mysqld_exporter.service
[root@server2 ~]# systemctl status mysqld_exporter.service
● mysqld_exporter.service - Prometheus MySQL Exporter
     Loaded: loaded (/etc/systemd/system/mysqld_exporter.service; enabled; pres>
     Active: active (running) since Wed 2026-05-06 16:05:54 CST; 2s ago
   Main PID: 1888 (mysqld_exporter)
      Tasks: 4 (limit: 22926)
     Memory: 4.7M
        CPU: 5ms
     CGroup: /system.slice/mysqld_exporter.service
             └─1888 /usr/local/bin/mysqld_exporter --config.my-cnf=/etc/.my.cnf

5月 06 16:05:54 server2 mysqld_exporter[1888]: time=2026-05-06T16:05:54.872+08:>
5月 06 16:05:54 server2 mysqld_exporter[1888]: time=2026-05-06T16:05:54.872+08:>
[root@server2 ~]# netstat -pantul|grep mysqld_export
tcp6       0      0 :::9104                 :::*                    LISTEN      1888/mysqld_exporte

1.3. 验证

http://192.168.80.12:9104/

Prometheus 告警配置与数据库监控27.png

2. 配置 Prometheus 采集 mysqld_exporter 的数据

修改配置文件 /usr/local/prometheus/prometheus.yml

vim /usr/local/prometheus/prometheus.yml
[root@server1 ~]# vim /usr/local/prometheus/prometheus.yml
[root@server1 ~]# cat /usr/local/prometheus/prometheus.yml
# my global config
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          # - alertmanager:9093
          - 192.168.80.11:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"
  - "/usr/local/prometheus/node.alerts.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: "prometheus"

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ["localhost:9090"]
       # The label name is added as a label `label_name=<label_value>` to any timeseries scraped from this config.
        labels:
          app: "prometheus"
    # node_exporter
  - job_name: "node"
    static_configs:
    - targets: ['192.168.80.12:9100','192.168.80.13:9100']
      labels:
        env: 'test'
    # ========== 新增：MySQL监控配置 ==========
  - job_name: "mysql"
    # 核心：指定mysqld_exporter的地址（192.168.80.12:9104）
    # 注意：这里是exporter的地址，不是MySQL的3306端口
    static_configs:
    - targets: ['192.168.80.12:9104']
      labels:
        env: 'test'
        mysql_server: '192.168.80.12'  # 标记被监控的MySQL地址，方便识别
[root@server1 ~]# systemctl restart prometheus
[root@server1 ~]# systemctl status prometheus
● prometheus.service - Prometheus Server
     Loaded: loaded (/etc/systemd/system/prometheus.service; enabled; preset: d>
     Active: active (running) since Wed 2026-05-06 16:15:43 CST; 3s ago
       Docs: https://prometheus.io/docs/introduction/overview/
   Main PID: 1481 (prometheus)
      Tasks: 7 (limit: 10867)
     Memory: 33.2M
        CPU: 373ms
     CGroup: /system.slice/prometheus.service
             └─1481 /usr/local/prometheus/prometheus --config.file=/usr/local/p>

5月 06 16:15:44 server1 prometheus[1481]: time=2026-05-06T16:15:44.230+08:00 le>
5月 06 16:15:44 server1 prometheus[1481]: time=2026-05-06T16:15:44.231+08:00 le>
5月 06 16:15:44 server1 prometheus[1481]: time=2026-05-06T16:15:44.231+08:00 le>
5月 06 16:15:44 server1 prometheus[1481]: time=2026-05-06T16:15:44.236+08:00 le>
5月 06 16:15:44 server1 prometheus[1481]: time=2026-05-06T16:15:44.236+08:00 le>
5月 06 16:15:44 server1 prometheus[1481]: time=2026-05-06T16:15:44.236+08:00 le>
5月 06 16:15:44 server1 prometheus[1481]: time=2026-05-06T16:15:44.236+08:00 le>
5月 06 16:15:44 server1 prometheus[1481]: time=2026-05-06T16:15:44.262+08:00 le>
5月 06 16:15:44 server1 prometheus[1481]: time=2026-05-06T16:15:44.262+08:00 le>
5月 06 16:15:44 server1 prometheus[1481]: time=2026-05-06T16:15:44.262+08:00 le>
[root@server1 ~]#

验证

http://192.168.80.11:9090/targets

Prometheus 告警配置与数据库监控28.png

2.1. 配置关键告警规则

注意：在 Prometheus 所在的机器上操作！

在 prometheus.yml 中配置 rule_files 指向告警规则文件，例如 mysql_alerts.yml

告警规则配置示例：

[root@server1 ~]# vim /usr/local/prometheus/mysql_alerts.yml
[root@server1 ~]# cat /usr/local/prometheus/mysql_alerts.yml
groups:
  - name: mysql_alerts
    rules:

      # ---------- 实例存活 ----------
      - alert: MySQLDown
        expr: mysql_up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "MySQL 实例 {{ $labels.instance }} 已宕机"
          description: "MySQL exporter 无法连接到数据库，持续 1 分钟。请立即检查数据库服务状态。"

      # ---------- 连接数 ----------
      - alert: MySQLTooManyConnections
        expr: (mysql_global_status_threads_connected / mysql_global_variables_max_connections) * 100 > 80
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "MySQL 连接数使用率超过 80%（实例 {{ $labels.instance }}）"
          description: "当前使用率 {{ $value | humanizePercentage }}。活跃连接数可能即将耗尽，请检查连接来源或考虑扩大 max_connections。"

      - alert: MySQLMaxConnectionsExhausted
        expr: (mysql_global_status_threads_connected / mysql_global_variables_max_connections) * 100 >= 95
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "MySQL 连接数使用率超过 95%（实例 {{ $labels.instance }}）"
          description: "当前使用率 {{ $value | humanizePercentage }}，数据库即将拒绝新连接！立即释放连接或紧急调整上限。"

      # ---------- 慢查询 ----------
      - alert: MySQLSlowQueriesSpike
        expr: rate(mysql_global_status_slow_queries[5m]) > 5
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "MySQL 慢查询速率飙高（实例 {{ $labels.instance }}）"
          description: "过去 5 分钟平均每秒慢查询 {{ $value }} 条，持续 10 分钟。请检查慢查询日志，定位性能瓶颈。"

      # ---------- 主从复制延迟 ----------
      - alert: MySQLReplicationLag
        expr: mysql_slave_status_seconds_behind_master > 30
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "MySQL 主从复制延迟 > 30s（实例 {{ $labels.instance }}）"
          description: "从库落后主库 {{ $value }} 秒，持续 5 分钟。请检查网络、从库 IO/SQL 线程状态。"

      - alert: MySQLReplicationNotRunning
        expr: mysql_slave_status_slave_io_running == 0 or mysql_slave_status_slave_sql_running == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "MySQL 复制线程停止（实例 {{ $labels.instance }}）"
          description: "IO 或 SQL 线程意外停止，复制中断。请检查从库状态及错误日志。"

      # ---------- InnoDB 缓冲池命中率 ----------
      - alert: MySQLInnoDBBufferPoolHitRateLow
        expr: (rate(mysql_global_status_innodb_buffer_pool_read_requests[5m]) - rate(mysql_global_status_innodb_buffer_pool_reads[5m])) / rate(mysql_global_status_innodb_buffer_pool_read_requests[5m]) < 0.95
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "InnoDB 缓冲池命中率低于 95%（实例 {{ $labels.instance }}）"
          description: "当前命中率 {{ $value | humanizePercentage }}，大量磁盘物理读。考虑增加 innodb_buffer_pool_size 或优化 SQL。"

      # ---------- 线程运行比例 ----------
      - alert: MySQLThreadsRunningHigh
        expr: mysql_global_status_threads_running / mysql_global_variables_max_connections > 0.6
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "MySQL 活跃线程占比过高（实例 {{ $labels.instance }}）"
          description: "当前 threads_running/max_connections 比例为 {{ $value | humanizePercentage }}，可能遇到查询堆积或锁争用。"

      # ---------- 磁盘空间（如果已启用 node_exporter 的磁盘监控）----------
      # 注意：这个指标来自 node_exporter，需要同类标签才能关联。此处仅做示例。
      # 更推荐通过 Prometheus 联合查询，或在 Grafana 中创建多数据源视图。

2.2. 在 Prometheus 中加载规则

编辑 prometheus.yml，增加或修改 rule_files 部分：

rule_files:
  - "/usr/local/prometheus/mysql.alerts.yml"
[root@server1 prometheus]# cat /usr/local/prometheus/prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - 192.168.80.11:9093

rule_files:
  - "/usr/local/prometheus/node.alerts.yml"
  - "/usr/local/prometheus/mysql_alerts.yml"

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]
        labels:
          app: "prometheus"

  - job_name: "node"
    static_configs:
      - targets: ['192.168.80.12:9100','192.168.80.13:9100']
        labels:
          env: 'test'

  - job_name: "mysql"
    static_configs:
      - targets: ['192.168.80.12:9104']
        labels:
          env: 'test'
          mysql_server: '192.168.80.12'

重载配置并重启Prometheus

[root@server1 ~]# systemctl daemon-reload
[root@server1 ~]# systemctl restart prometheus.service
[root@server1 ~]# systemctl status prometheus.service
● prometheus.service - Prometheus Server
     Loaded: loaded (/etc/systemd/system/prometheus.service; enabled; preset: d>
     Active: active (running) since Wed 2026-05-06 16:27:26 CST; 6s ago
       Docs: https://prometheus.io/docs/introduction/overview/
   Main PID: 1671 (prometheus)
      Tasks: 8 (limit: 10867)
     Memory: 43.1M
        CPU: 394ms
     CGroup: /system.slice/prometheus.service
             └─1671 /usr/local/prometheus/prometheus --config.file=/usr/local/p>

查看验证

Prometheus 告警配置与数据库监控29.png

Prometheus 告警配置与数据库监控30.png

2.3. Alertmanager 配置

只需在 route 下新增一条匹配 MySQL 告警的路由规则，并在 receivers 中定义对应的接收器即可。修改后的完整配置如下

[root@server1 ~]# vim /usr/local/alertmanager/alertmanager.yml
[root@server1 ~]# cat /usr/local/alertmanager/alertmanager.yml
global:
  # 邮件配置
  smtp_smarthost: 'smtp.qq.com:587'
  smtp_from: '2585785635@qq.com'
  smtp_auth_username: '2585785635@qq.com'
  smtp_auth_password: 'tpzkercrzxhtebfb'   # 必须替换为重新生成的授权码
  smtp_require_tls: true            # 使用587端口的TLS

route:
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 1h
  receiver: 'default.all'          # 默认接收者，邮件+webhook
  routes:
    - matchers:
        - job=~"node"
      receiver: 'system.all'      # node告警特殊处理，同样邮件+webhook

    # ---------- 新增 MySQL 告警路由 ----------
    - matchers:
        - job=~"mysql"
      receiver: 'mysql.all'       # mysql告警专用接收器
      group_wait: 10s             # 可选：缩短mysql告警的第一次等待时间，更快通知
      repeat_interval: 30m        # 可选：mysql告警的重复发送间隔更短（可根据需要调整）

receivers:
  # 默认接收者：同时发送邮件和webhook
  - name: 'default.all'
    email_configs:
      - to: '2585785635@qq.com'
        send_resolved: true
    webhook_configs:
      - url: 'http://127.0.0.1:5001/'

  # 系统告警接收者（node类），同样邮件+webhook
  - name: 'system.all'
    email_configs:
      - to: '2585785635@qq.com'
        send_resolved: true
    webhook_configs:
      - url: 'http://127.0.0.1:5001/'

  # ---------- 新增 MySQL 告警接收器 ----------
  - name: 'mysql.all'
    email_configs:
      - to: '2585785635@qq.com'
        send_resolved: true
    webhook_configs:
      - url: 'http://127.0.0.1:5001/'

inhibit_rules:
  - source_matchers: [severity="critical"]
    target_matchers: [severity="warning"]
    equal: [alertname, dev, instance]
[root@server1 ~]#

生效操作

systemctl daemon-reload
systemctl restart alertmanager
systemctl status alertmanager

# 或使用 API 热加载（确保 alertmanager 启用 --web.enable-lifecycle）
curl -X POST http://localhost:9093/-/reload

2.4. 制造故障

[root@server2 ~]# systemctl stop mysqld
[root@server2 ~]# systemctl status mysqld
○ mysqld.service - MySQL Server
     Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; preset: d>
     Active: inactive (dead) since Wed 2026-05-06 17:55:20 CST; 6s ago
   Duration: 22min 3.176s
       Docs: man:mysqld(8)
             http://dev.mysql.com/doc/refman/en/using-systemd.html
    Process: 2075 ExecStartPre=/usr/bin/mysqld_pre_systemd (code=exited, status>
    Process: 2102 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS (code=exited, status=>
   Main PID: 2102 (code=exited, status=0/SUCCESS)
     Status: "Server shutdown complete"
        CPU: 8.296s

5月 06 17:33:14 server2 systemd[1]: Starting MySQL Server...
5月 06 17:33:16 server2 systemd[1]: Started MySQL Server.
5月 06 17:55:19 server2 systemd[1]: Stopping MySQL Server...
5月 06 17:55:20 server2 systemd[1]: mysqld.service: Deactivated successfully.
5月 06 17:55:20 server2 systemd[1]: Stopped MySQL Server.
5月 06 17:55:20 server2 systemd[1]: mysqld.service: Consumed 8.296s CPU time.
[root@server2 ~]#

Prometheus 告警配置与数据库监控31.png

Prometheus 告警配置与数据库监控32.png

Prometheus 告警配置与数据库监控33.png

Prometheus 告警配置与数据库监控34.png

2.5. 修复故障

[root@server2 ~]# systemctl start mysqld
[root@server2 ~]# systemctl status mysqld
● mysqld.service - MySQL Server
     Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; preset: d>
     Active: active (running) since Wed 2026-05-06 17:58:06 CST; 6s ago
       Docs: man:mysqld(8)
             http://dev.mysql.com/doc/refman/en/using-systemd.html
    Process: 2158 ExecStartPre=/usr/bin/mysqld_pre_systemd (code=exited, status>
   Main PID: 2185 (mysqld)
     Status: "Server is operational"
      Tasks: 39 (limit: 22926)
     Memory: 373.8M
        CPU: 2.174s
     CGroup: /system.slice/mysqld.service
             └─2185 /usr/sbin/mysqld

5月 06 17:58:04 server2 systemd[1]: Starting MySQL Server...
5月 06 17:58:06 server2 systemd[1]: Started MySQL Server.
[root@server2 ~]#

Prometheus 告警配置与数据库监控35.png

Prometheus 告警配置与数据库监控36.png

3. 导入 MySQL dashboard

打开https://grafana.com/grafana/dashboards/搜索 MySQL

https://grafana.com/grafana/dashboards/?search=MySQL

Prometheus 告警配置与数据库监控37.png

复制 dashboard ID 或下载 json

Prometheus 告警配置与数据库监控38.png

转到 grafana，点击 Import

Prometheus 告警配置与数据库监控39.png

Prometheus 告警配置与数据库监控40.png

Prometheus 告警配置与数据库监控41.png

Prometheus 告警配置与数据库监控42.png

选择数据源，点击 Import

Prometheus 告警配置与数据库监控43.png
查看 MySQL 监控

Prometheus 告警配置与数据库监控44.png

补充：

http://192.168.80.11:9090/

mysql_global_status_uptime

Prometheus 告警配置与数据库监控45.png

总结

  1. 成功构建了完整的告警响应流程，从告警规则触发、Alertmanager 接收到多渠道通知（邮件、企业微信、webhook），实现了告警的实时感知与处理。
  2. 掌握了 Alertmanager 的静默与路由高级功能，能够灵活应对日常运维中的告警洪流与分级通知需求。
  3. 将监控范围从服务器扩展至数据库中间件，成功部署 MySQL 监控，实现了关键指标的采集、可视化和告警，为保障核心业务数据稳定运行提供了有力支撑。
  4. 本次实践进一步深化了对 Prometheus 监控体系的理解，为后续监控更多业务组件（如 Redis、Nginx、容器等）奠定了坚实基础。
DM 达梦数据库
Prometheus 监控采集与可视化