目录
linux docker监控
linux 系统进程监控
linux 系统os监控
windows 系统os监控
配置文件&告警规则
Prometheus配置文件
node_alert.rules
docker_container.rules
mysql_alert.rules
vmware.rules
Alertmanager告警规则
consoul注册服务
Dashboard JSON文件
linux docker监控
获取的是docker stats命令的统计结果，可以页面方式展示出来。
cadvisor.tar
上传cadvisor.tar包，导入后修改tag，运行容器
docker load -i cadvisor.tardocker tag gcr.io/cadvisor/cadvisor:latest google/cadvisor:latestdocker run -d --volume=/:/rootfs:ro --volume=/var/run:/var/run:rw --volume=/sys:/sys:ro --volume=/var/lib/docker/:/var/lib/docker:ro --publish=8080:8080 --name=cadvisor google/cadvisor:latest容器运行后如下：
访问cadvisor http://ip:8080
linux 系统进程监控
通过正则、绝对路径、名字等获取指定进程的运行状况
process-exporter-0.7.5.linux-amd64.tar.gz
参考我的另一篇文章
Prometheus监控主机进程-CSDN博客
默认端口 9256
linux 系统os监控
通过exporter获取当前系统的Cpu、内存、硬盘等OS资源
node_exporter放到指定路径后
cat /etc/systemd/system/node-exporter.service
[Unit]
Description=Prometheus Node exporter
After=network.target[Service]
ExecStart=/opt/monitoring/node_exporter[Install]
WantedBy=multi-user.target
默认端口：9100
windows 系统os监控
通过exporter获取当前系统的Cpu、内存、硬盘等OS资源
windows_exporter-0.26.0-amd64.msi
1.关闭防火墙
2.管理员模式双击执行
3.services.msc服务管理检查windows-exporter服务自动启动即可
默认端口：9182
配置文件&告警规则
/opt/monitor/prometheus目录下
Prometheus配置文件
cat /opt/monitor/prometheus/prometheus.yml
# my global config
global:scrape_interval: 10s # By default, scrape targets every 15 seconds.scrape_timeout: 5sevaluation_interval: 10s # By default, scrape targets every 15 seconds.# scrape_timeout is set to the global default (10s).# Attach these labels to any time series or alerts when communicating with# external systems (federation, remote storage, Alertmanager).external_labels:monitor: 'zqa_monitor'# Load and evaluate rules in this file every 'evaluation_interval' seconds.
rule_files:- 'node_alert.rules'- 'mysql_alert.rules'- 'docker_container.rules'# - "first.rules"# - "second.rules"# alert
alerting:alertmanagers:- scheme: httpstatic_configs:- targets:- "alertmanager:9093"# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.- job_name: 'prometheus'# Override the global default and scrape targets from this job every 5 seconds.scrape_interval: 5sstatic_configs:- targets: ['localhost:9090']#- job_name: 'cadvisor'# Override the global default and scrape targets from this job every 5 seconds.# scrape_interval: 5s#dns_sd_configs:#- names:# - 'tasks.cadvisor'# type: 'A'# port: 8080#static_configs:# - targets: ['10.33.70.218:8080']- job_name: 'node-exporter'# Override the global default and scrape targets from this job every 5 seconds.scrape_interval: 5sstatic_configs:- targets: ['10.100.10.100:9182']consul_sd_configs:- server: '10.33.70.203:8500'services: ['node-exporter-dev']- job_name: 'mysql-exporter'scrape_interval: 5sstatic_configs:- targets: ['10.33.70.218:9104', '10.33.70.166:9104', '10.33.70.224:9104']- job_name: 'postgres-exporter'scrape_interval: 5sstatic_configs:- targets: ['123.57.190.129:9187']- job_name: 'vsphere-exporter'scrape_interval: 5sstatic_configs:- targets: ['10.33.70.22:9272']- job_name: 'es-exporter'scrape_interval: 5sstatic_configs:- targets: ['123.57.216.51:9114']- job_name: 'pushgateway'scrape_interval: 30sstatic_configs:- targets: ['39.104.94.83:19091']labels:instance: pushgatewayhonor_labels: true- job_name: "cadvisor"scrape_interval: 10smetrics_path: '/metrics'static_configs:- targets: ["47.93.21.11:8080]#- job_name: 'kafka-exporter'# scrape_interval: 5s# static_configs:# - targets: [ '10.100.7.1:9308']# - job_name: 'pushgateway'
# scrape_interval: 10s
# dns_sd_configs:
# - names:
# - 'tasks.pushgateway'
# type: 'A'
# port: 9091# static_configs:
# - targets: ['node-exporter:9100']
node_alert.rules
groups:
- name: zqaalertrules:- alert: 机器宕机expr: up == 0for: 2mlabels:severity: criticalannotations:summary: "Instance {{ $labels.instance }} down"description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 2 minutes."- alert: 负载率expr: node_load1 > 8for: 5mlabels:severity: warningannotations:summary: "Instance {{ $labels.instance }} under high load"description: "{{ $labels.instance }} of job {{ $labels.job }} is under high load."- alert: 可用内存小于5%expr: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100 < 5for: 10mlabels:severity: warningannotations:summary: Host out of memory (instance {{ $labels.instance }})description: "节点内存告警 (< 5% left)\n VALUE = {{ $value }}\n LABELS = {{ $labels }}"- alert: 磁盘使用率expr: (100 - ((node_filesystem_avail_bytes{device!~'rootfs'} * 100) / node_filesystem_size_bytes{device!~'rootfs'}) > 90)for: 5mlabels:severity: Highannotations:summary: "{{$labels.instance}}: High Disk usage detected"description: "{{$labels.instance}}: 硬盘使用率大于 90% (当前值:{{ $value }})"- alert: Cpu使用率expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[10m])) * 100) > 95for: 10mlabels:severity: warningannotations:summary: "{{$labels.instance}}: High Cpu usage detected"description: "{{$labels.instance}}: CPU 使用率大于 95% (current value is:{{ $value }})"# - alert: 进程恢复# expr: ceil(time() - max by(instance, groupname) (namedprocess_namegroup_oldest_start_time_seconds)) < 60# for: 0s# labels:# severity: warning# annotations:# summary: "进程重启"# description: "进程{{ $labels.groupname }}在{{ $value }}秒前重启过"- alert: 进程退出告警# expr: max by(instance, groupname) (rate(namedprocess_namegroup_oldest_start_time_seconds{groupname=~"^vsftpd.*|^proxy.*|^goproxy.*|^lizhu_monitor*|^lizhu_agent*|^lizhurunner*"}[5m])) < 0expr: namedprocess_namegroup_num_procs{groupname=~"^vsftpd.*|^proxy.*|^goproxy.*|^lizhu_monitor*|^lizhu_agent*|^lizhurunner*"} == 0for: 30slabels:severity: warningannotations:summary: "进程退出"description: "进程{{ $labels.groupname }}退出了" # - alert: 进程退出告警
# expr: max_over_time(namedprocess_namegroup_oldest_start_time_seconds{groupname=~"^vsftpd.*|^proxy.*|^goproxy.*|^lizhu_monitor.*|^lizhu_agent.*|^lizhurunner.*"}[1d]) < (time() - 10*60)
# for: 1s
# labels:
# severity: warning
# annotations:
# description: 进程组 {{ $labels.groupname }} 中的进程在最近10分钟内退出了
# summary: 进程退出#- alert: 机器硬盘读取速率# expr: sum by (instance) (rate(node_disk_read_bytes_total[2m])) / 1024 / 1024 > 200# for: 5m# labels:# severity: warning# annotations:# summary: Host unusual disk read rate (instance {{ $labels.instance }})# description: "Disk is probably reading too much data (> 50 MB/s)\n VALUE = {{ $value }}\n LABELS = {{ $labels }}"#- alert: 机器硬盘写入速率# expr: sum by (instance) (rate(node_disk_written_bytes_total[2m])) / 1024 / 1024 > 120# for: 2m# labels:# severity: warning# annotations:# summary: Host unusual disk write rate (instance {{ $labels.instance }})# description: "Disk is probably writing too much data VALUE = {{ $value }}\n LABELS = {{ $labels }}"- alert: HostOomKillDetectedexpr: increase(node_vmstat_oom_kill[1m]) > 0for: 0mlabels:severity: warningannotations:summary: Host OOM kill detected (instance {{ $labels.instance }})description: "OOM kill detected\n VALUE = {{ $value }}\n LABELS = {{ $labels }}"- alert: Esxi主机连接丢失expr: vmware_host_power_state != 1for: 1m labels:severity: criticalannotations:summary: "Esxi 物理机IP: {{ $labels.host_name }} 丢失连接"description: "VMware host {{ $labels.host_name }} is not connected to the virtualization platform."
docker_container.rules
groups:
- name: zqaalertrules:- alert: ContainerAbsentexpr: absent(container_last_seen)for: 5mlabels:severity: warningannotations:summary: "无容器 容器:{{$labels.instance }}"description: "5分钟检查容器不存在,当前值为:{{ $value }}"- alert: ContainerCpuUsageexpr: (sum(rate(container_cpu_usage_seconds_total{name!=""}[3m])) BY(instance, name)*100 ) > 300for: 2mlabels:severity: warningannotations:summary: "容器cpu使用率告警,容器:{{$labels.instance }}"description: "容器cpu使用率超过300%,当前值为:{{ $value }}"- alert: ContainerMemoryUsageexpr: (sum(container_memory_working_set_bytes{name!=""})BY (instance, name) /sum(container_spec_memory_limit_bytes > 0) BY (instance, name) * 100 ) > 80for: 2mlabels:severity: warningannotations:summary: "容器内存使用率告警,容器:{{$labels.instance }}"description: "容器内存使用率超过80%,当前值为:{{ $value }}"- alert: ContainerVolumeIOUsageexpr: (sum(container_fs_io_current{name!=""}) BY (instance, name) * 100) >80 for: 2mlabels:severity: warningannotations:summary: "容器存储IO使用率告警,容器:{{$labels.instance }}"description: "容器存储IO使用率超过80%,当前值为:{{ $value }}"- alert: ContainerHighThrottleRateexpr: rate(container_cpus_cfs_throttled_seconds_total[3m]) > 1 for: 2mlabels:severity: warningannotations:summary: "容器限制告警,容器:{{$labels.instance }}"description: "容器被限制,当前值为:{{ $value }}"
mysql_alert.rules
groups:
- name: zqaalertrules:- alert: Mysql 宕机expr: mysql_up == 0for: 1mlabels:severity: criticalannotations:summary: MySQL down (instance {{ $labels.instance }})description: "MySQL instance is down on {{ $labels.instance }}\n VALUE = {{ $value }}\n LABELS = {{ $labels }}"- alert: MysqlTooManyConnections(>80%)expr: max_over_time(mysql_global_status_threads_connected[1m]) / mysql_global_variables_max_connections * 100 > 80for: 2mlabels:severity: warningannotations:summary: MySQL too many connections (> 80%) (instance {{ $labels.instance }})description: "More than 80% of MySQL connections are in use on {{ $labels.instance }}\n VALUE = {{ $value }}\n LABELS = {{ $labels }}"- alert: MysqlHighThreadsRunningexpr: max_over_time(mysql_global_status_threads_running[1m]) / mysql_global_variables_max_connections * 100 > 60for: 2mlabels:severity: warningannotations:summary: MySQL high threads running (instance {{ $labels.instance }})description: "More than 60% of MySQL connections are in running state on {{ $labels.instance }}\n VALUE = {{ $value }}\n LABELS = {{ $labels }}"- alert: Mysql慢查询expr: increase(mysql_global_status_slow_queries[1m]) > 0for: 60mlabels:severity: warningannotations:summary: MySQL slow queries (instance {{ $labels.instance }})description: "MySQL server mysql has some new slow query.\n VALUE = {{ $value }}\n LABELS = {{ $labels }}"
vmware.rules
- name: VMware Host Connection Staterules:- alert: HostDisconnectedexpr: vmware_host_power_state == "connected"for: 5m # 规定主机连接状态必须持续5分钟才会触发警报labels:severity: warningannotations:summary: "VMware host {{ $labels.instance }} disconnected"description: "VMware host {{ $labels.instance }} is not connected to the virtualization platform."
Alertmanager告警规则
通过定义组来监控组内机器
cat vim /opt/monitor/alertmanager/config.yml
global:resolve_timeout: 5msmtp_from: 'ops@xxx.com'smtp_smarthost: 'smtp.feishu.cn:465'smtp_auth_username: 'ops@xxx.com'smtp_auth_password: 'ydWhsFDk3pF50TZg'smtp_require_tls: falsesmtp_hello: 'ZQA监控告警'route:group_by: ['zqaalert']group_wait: 60s # 在触发第一个警报后，等待相同分组内的所有警报的最长时间group_interval: 10m # 系统每隔10分钟检查一次是否有新的警报需要处理repeat_interval: 60m # 在发送警报通知后，在重复发送通知之间等待的时间。设置为1小时意味着如果同一组内的警报在 1小时再次触发receiver: 'web.hook'
receivers:
#- name: 'web.hook.prometheusalert'
- name: 'web.hook'webhook_configs:- url: 'http://10.33.70.22:9094/prometheusalert?type=fs&tpl=prometheus-fs&fsurl=https://open.feishu.cn/open-apis/bot/v2/hook/7fe7f42d-242b-42eb-837c-028cfc84adb8'
consoul注册服务
* */1 * * * ip addr | awk '/^[0-9]+: / {}; /inet.*global/ {print gensub(/(.*)\/(.*)/, "\\1", "g", $2)}' |grep "10.33"|head -1|xargs -i curl -X PUT -d '{"id": "node-exporter-{}","name": "node-exporter-dev","address": "{}","port": 9100,"tags": ["env-dev"],"checks": [{"http": "http://{}:9100/metrics", "interval": "5s"}]}' http://consul.intra.xxx.net/v1/agent/service/register有现成的consoul容器，运行即可
Dashboard JSON文件
以下是我认为比较好用的 grafana 的 dashboards文件
Grafana dashboards | Grafana Labs
