企业监控大盘Grafana

Grafana简述

Grafana 是一个开源的度量分析与可视化工具。提供查询、可视化、报警和指标展示等功能，能灵活创建图表、仪表盘等可视化界面

主要功能：

可视化: 提供多种可选择的不同类型的图形，能够灵活绘制不同样式，且还提供很多插件。
动态仪表盘: 提供以模板和变量的方式来创建动态且可重复使用的仪表盘，可以灵活调整。
浏览指标: 通过瞬时查询和动态变化等方式展示数据，可以根据不同的时间范围拆分视图。
警报: 可以直观地根据重要的指标定义警报规则。Grafana 将不断评估并向 Slack，邮件，快消息等系统发送通知。
混合数据源: 在同一图中混合不同的数据源，可以基于每个查询指定不同数据源。

在这里插入图片描述

部署Grafana到Kubernetes

数据持久化

apiVersion: v1
kind: PersistentVolumeClaim
metadata:name: grafana-data-pvcnamespace: monitor
spec:accessModes:- ReadWriteManystorageClassName: "nfs-storage"resources:requests:storage: 10Gi

配置grafana-config

apiVersion: v1
kind: ConfigMap
metadata:name: grafana-confignamespace: monitor
data:grafana.ini: |[server]root_url = http://grafana.kubernets.cn[smtp]enabled = truehost = smtp.exmail.qq.com:465user = 1440350254@qq.compassword = xxxskip_verify = truefrom_address = 1440350254@qq.com[alerting]enabled = trueexecute_alerts = true

配置grafana-SVC

apiVersion: v1
kind: Service
metadata:name: grafananamespace: monitorlabels:app: grafanacomponent: core
spec:type: ClusterIPports:- port: 3000selector:app: grafanacomponent: core

部署grafana-DP

apiVersion: apps/v1
kind: Deployment
metadata:name: grafana-corenamespace: monitorlabels:app: grafanacomponent: core
spec:replicas: 1selector:matchLabels:app: grafanatemplate:metadata:labels:app: grafanacomponent: corespec:containers:- name: grafana-coreimage: grafana/grafana:latestimagePullPolicy: IfNotPresentvolumeMounts:- name: storagesubPath: grafanamountPath: /var/lib/grafana# env:resources:# keep request = limit to keep this container in guaranteed classlimits:cpu: 500mmemory: 1Girequests:cpu: 100mmemory: 500Mienv:            #配置环境变量,设置Grafana 的默认管理员用户名/密码# The following env variables set up basic auth twith the default admin user and admin password.- name: GF_AUTH_BASIC_ENABLEDvalue: "true"- name: GF_AUTH_ANONYMOUS_ENABLEDvalue: "false"# - name: GF_AUTH_ANONYMOUS_ORG_ROLE#   value: Admin# does not really work, because of template variables in exported dashboards:# - name: GF_DASHBOARDS_JSON_ENABLED#   value: "true"readinessProbe:httpGet:path: /loginport: 3000# initialDelaySeconds: 30# timeoutSeconds: 1volumeMounts:- name: datasubPath: grafanamountPath: /var/lib/grafana- name: grafana-configmountPath: /etc/grafanareadOnly: truesecurityContext:       #容器安全策略,设置运行容器使用的归属组与用户fsGroup: 472runAsUser: 472volumes:- name: datapersistentVolumeClaim:claimName: grafana-data-pvc- name: grafana-configconfigMap:name: grafana-config

部署grafana-ING

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:name: grafana-ingressnamespace: monitorannotations:prometheus.io/http_probe: "true"
spec:ingressClassName: nginxrules:- host: grafana.kubernets.cnhttp:paths:- pathType: Prefixbackend:service:name: grafanaport:number: 3000path: /

Grafana安装插件

# kubectl exec -it -n monitor grafana-58ffb4db5d-c4wlz bashbash-5.0$ grafana-cli plugins install grafana-piechart-panel
bash-5.0$ grafana-cli plugins install camptocamp-prometheus-alertmanager-datasource

验证测试Grafana

$ curl grafana.kubernets.cn
<a href="/login">Found</a>.

配置数据源

Grafana官方提供了对：Graphite, InfluxDB, OpenTSDB, Prometheus, Elasticsearch, CloudWatch的支持。

添加数据源：Configuration --> Data Sources --> Prometheus

HTTP URL：http://prometheus.monitor:9090 Save & test

企业级监控大盘

创建不同维度的大盘：Create --> New dashboard folder --> 集群层面

官方大盘指引：Dashboards | Grafana Labs

监控指标说明：监控指标说明 | KubeSphere Documents

集群层面监控

导入大盘：Create --> Import

集群：

Kubernetes / Views / Global ：15757
- node_cpu_core_throttles_total ignores second thread of hyperthreading systems · Issue #1472 · prometheus/node_exporter (github.com)
Kubernetes / Views / Nodes ：15759
Kubernetes / Views / Namespaces ：15758
Kubernetes / Views / Pods ：15760
Kubernetes Deployment Statefulset Daemonset metrics ：8588