k8s部署es和skywalking

使用k8s部署es和skywalking

skywalking介绍

skywalking架构

在这里插入图片描述

整个架构，分成上、下、左、右四部分：

上部分
Agent ：负责从应用中，收集链路信息，发送给 SkyWalking OAP 服务器。目前支持 SkyWalking、Zikpin、Jaeger 等提供的 Tracing 数据信息。而我们目前采用的是，SkyWalking Agent 收集 SkyWalking Tracing 数据，传递给服务器
下部分
SkyWalking OAP ：负责接收 Agent 发送的 Tracing 数据信息，然后进行分析(Analysis Core) ，存储到外部存储器( Storage )，最终提供查询( Query )功能
右部分
Storage ：Tracing 数据存储。目前支持 ES、MySQL、Sharding Sphere、TiDB、H2 多种存储器。而我们目前采用的是 ES ，主要考虑是 SkyWalking 开发团队自己的生产环境采用 ES 为主
左部分
SkyWalking UI ：负责提供控台，查看链路等等
部署elasticsearch
skywalking的持久化选择的是elasticsearch，请大家根据自己自己的情况来选择持久化的方式。

部署storageclass

我不太喜欢使用helm一键部署的方式来进行部署服务，helm的方式有点像黑盒。我喜欢使用yaml文件一个服务一个服务来部署。下面是部署的yaml 文件。

storageclass.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:name: nfs-storagenamespace: skywalking 
provisioner: fuseim.pri/ifs # or choose another name, must match deployment's env PROVISIONER_NAME'
parameters:archiveOnDelete: "false"
reclaimPolicy: Retain

部署nfs-provisioner

由于在k8s 1.20以上版本弃用了selfLink，我这里就不用nfs-client-provisioner镜像了，使用dyrnq/nfs-subdir-external-provisioner。

apiVersion: apps/v1
kind: Deployment
metadata:name: nfs-client-provisioner
spec:replicas: 1strategy:type: Recreateselector:matchLabels:app: nfs-client-provisionertemplate:metadata:labels:app: nfs-client-provisionerspec:serviceAccountName: nfs-client-provisionercontainers:- name: nfs-client-provisionerimage: dyrnq/nfs-subdir-external-provisioner:v4.0.2imagePullPolicy: IfNotPresentvolumeMounts:- name: nfs-client-rootmountPath: /persistentvolumesenv:- name: PROVISIONER_NAMEvalue: fuseim.pri/ifs- name: NFS_SERVERvalue: 192.168.5.166- name: NFS_PATHvalue: /mnt/jf/skywalkingvolumes:- name: nfs-client-rootnfs:server: 192.168.5.166path: /mnt/jf/skywalking
---
apiVersion: v1
kind: ServiceAccount
metadata:name: nfs-client-provisioner
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:name: nfs-client-provisioner-runner
rules:- apiGroups: [""]resources: ["persistentvolumes"]verbs: ["get", "list", "watch", "create", "delete"]- apiGroups: [""]resources: ["persistentvolumeclaims"]verbs: ["get", "list", "watch", "update"]- apiGroups: ["storage.k8s.io"]resources: ["storageclasses"]verbs: ["get", "list", "watch"]- apiGroups: [""]resources: ["events"]verbs: ["create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:name: run-nfs-client-provisioner
subjects:- kind: ServiceAccountname: nfs-client-provisionernamespace: skywalking
roleRef:kind: ClusterRolename: nfs-client-provisioner-runnerapiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:name: leader-locking-nfs-client-provisioner
rules:- apiGroups: [""]resources: ["endpoints"]verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:name: leader-locking-nfs-client-provisioner
subjects:- kind: ServiceAccountname: nfs-client-provisioner
roleRef:kind: Rolename: leader-locking-nfs-client-provisionerapiGroup: rbac.authorization.k8s.io

部署elasticsearch

由于es属于有状态的服务，我这里选择使用statefulset方式来部署
es-statefulset.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:name: esnamespace: skywalking
spec:serviceName: elasticsearchreplicas: 3selector:matchLabels:app: elasticsearchtemplate:metadata:labels: app: elasticsearchspec:imagePullSecrets:- name: harborsecretinitContainers:- name: increase-vm-max-mapimage: busybox:latestcommand: ["sysctl", "-w", "vm.max_map_count=262144"]securityContext:privileged: true- name: increase-fd-ulimitimage: busybox:latestcommand: ["sh", "-c", "ulimit -n 65536"]securityContext:privileged: truecontainers:- name: elasticsearchimage: elasticsearch:7.17.14ports:- name: restcontainerPort: 9200- name: intercontainerPort: 9300resources:limits:cpu: 1000mrequests:cpu: 1000mvolumeMounts:- name: datamountPath: /usr/share/elasticsearch/dataenv:- name: cluster.namevalue: k8s-logs- name: node.namevalueFrom:fieldRef:fieldPath: metadata.name- name: cluster.initial_master_nodesvalue: "es-0,es-1,es-2"- name: discovery.seed_hostsvalue: "elasticsearch"- name: ES_JAVA_OPTSvalue: "-Xms512m -Xmx512m"- name: network.hostvalue: "0.0.0.0"volumeClaimTemplates:- metadata:name: datalabels:app: elasticsearchspec:accessModes: [ "ReadWriteOnce" ]storageClassName: nfs-storageresources:requests:storage: 300Gi service.yaml
kind: Service
apiVersion: v1
metadata:name: elasticsearchnamespace: skywalkinglabels:app: elasticsearch
spec:selector:app: elasticsearchclusterIP: Noneports:- port: 9200name: rest- port: 9300name: inter-node
---
kind: Service
apiVersion: v1
metadata:name: elasticsearch-clientnamespace: skywalkinglabels:app: elasticsearch
spec:selector:app: elasticsearchports:- port: 9200name: rest- port: 9300name: inter-node

部署完es以后，通过访问IP:9200进行访问。测试es是否部署成功。
在浏览器访问出现如下界面，证书部署成功：
在这里插入图片描述

部署skywalking

部署RBAC

RBAC.yaml

apiVersion: v1
kind: ServiceAccount
metadata:labels:app: skywalkingname: skywalking-oapnamespace: skywalking
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:name: skywalkingnamespace: skywalking  labels:app: skywalking
rules:- apiGroups: [""]resources: ["pods", "endpoints", "services", "nodes"]verbs: ["get", "watch", "list"]- apiGroups: ["extensions"]resources: ["deployments", "replicasets"]verbs: ["get", "watch", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:name: skywalkingnamespace: skywalking  labels:app: skywalking
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: skywalking
subjects:- kind: ServiceAccountname: skywalking-oapnamespace: skywalking

部署数据初始化Job

apiVersion: batch/v1
kind: Job
metadata:name: "skywalking-es-init"namespace: skywalking  labels:app: skywalking-job
spec:template:metadata:name: "skywalking-es-init"labels:app: skywalking-jobspec:serviceAccountName: skywalking-oaprestartPolicy: NeverinitContainers:- name: wait-for-elasticsearchimage: busybox:1.30imagePullPolicy: IfNotPresentcommand: ['sh', '-c', 'for i in $(seq 1 60); do nc -z -w3 elasticsearch 9200 && exit 0 || sleep 5; done; exit 1']containers:- name: oapimage: skywalking.docker.scarf.sh/apache/skywalking-oap-server:8.9.0imagePullPolicy: IfNotPresentenv:- name: JAVA_OPTSvalue: "-Xmx2g -Xms2g -Dmode=init"- name: SW_STORAGEvalue: elasticsearch- name: SW_STORAGE_ES_CLUSTER_NODESvalue: "elasticsearch:9200"volumeMounts:volumes:

部署OAP

oap.yaml

apiVersion: v1
kind: Service
metadata:name: oap-svcnamespace: skywalking  labels:app: oap
spec:type: ClusterIPports:- port: 11800name: grpc- port: 12800name: restselector:app: oap
---
apiVersion: apps/v1
kind: Deployment
metadata:labels:app: oapname: oapnamespace: skywalking  
spec:replicas: 1selector:matchLabels:app: oaptemplate:metadata:labels:app: oapspec:serviceAccountName: skywalking-oapaffinity:podAntiAffinity:preferredDuringSchedulingIgnoredDuringExecution:- weight: 1podAffinityTerm:topologyKey: kubernetes.io/hostnamelabelSelector:matchLabels:app: "skywalking"initContainers:- name: wait-for-elasticsearchimage: busybox:1.30imagePullPolicy: IfNotPresentcommand: ['sh', '-c', 'for i in $(seq 1 60); do nc -z -w3 elasticsearch 9200 && exit 0 || sleep 5; done; exit 1']containers:- name: oapimage: skywalking.docker.scarf.sh/apache/skywalking-oap-server:8.9.0imagePullPolicy: IfNotPresentlivenessProbe:tcpSocket:port: 12800initialDelaySeconds: 15periodSeconds: 20readinessProbe:tcpSocket:port: 12800initialDelaySeconds: 15periodSeconds: 20ports:- containerPort: 11800name: grpc- containerPort: 12800name: restenv:- name: JAVA_OPTSvalue: "-Dmode=no-init -Xmx2g -Xms2g"- name: SW_CLUSTERvalue: kubernetes- name: SW_CLUSTER_K8S_NAMESPACEvalue: "skywalking"- name: SW_CLUSTER_K8S_LABELvalue: "app=skywalking,release=skywalking,component=oap"# 记录数据- name: SW_CORE_RECORD_DATA_TTLvalue: "2"# Metrics数据  - name: SW_CORE_METRICS_DATA_TTLvalue: "2"- name: SKYWALKING_COLLECTOR_UIDvalueFrom:fieldRef:fieldPath: metadata.uid- name: SW_STORAGEvalue: elasticsearch- name: SW_STORAGE_ES_CLUSTER_NODESvalue: "elasticsearch:9200"

部署UI

skywalking的UI界面
ui.yaml

apiVersion: v1
kind: Service
metadata:labels:app: uiname: ui-svcnamespace: skywalking  
spec:type: ClusterIPports:- port: 80targetPort: 8080protocol: TCPselector:app: ui
---
apiVersion: apps/v1
kind: Deployment
metadata:name: uinamespace: skywalking  labels:app: ui
spec:replicas: 1selector:matchLabels:app: uitemplate:metadata:labels:app: uispec:affinity:containers:- name: uiimage: skywalking.docker.scarf.sh/apache/skywalking-ui:8.9.0imagePullPolicy: IfNotPresentports:- containerPort: 8080name: pageenv:- name: SW_OAP_ADDRESSvalue: http://elasticsearch:12800  #根据oap的svc一致

查看服务

测试部署项目接入agent

编写entrypoint.sh

在jvm启动参数中添加skywalking相关的启动参数

java -javaagent:/skywalking-agent/skywalking-agent.jar \
-Dskywalking.agent.service_name=${appName} \
-Dskywalking.collector.backend_service=skywalking的service名称:11800

添加java的agent

根据你后端语言选择不同的agent进行下载，下载地址，我这边使用的是java agent

下载java-agnet

可以选择将agent下载到指定的服务器

wget https://archive.apache.org/dist/skywalking/java-agent/8.10.0/apache-skywalking-java-agent-8.10.0.tgz
tar xf apache-skywalking-java-agent-8.10.0.tgz 
ls
apache-skywalking-java-agent-8.10.0.tgz  skywalking-agent

我这直接将java的agent添加到dockerfile中。

# 添加以下指令将SkyWalking Agent文件拷贝到容器内
ADD https://archive.apache.org/dist/skywalking/java-agent/8.10.0/apache-skywalking-java-agent-8.10.0.tgz $PROJECT_ROOT/conf/skywalking-agent.tgz
# 解压Agent文件
RUN tar -xzf $PROJECT_ROOT/conf/skywalking-agent.tgz -C $PROJECT_ROOT/conf/

调用链忽略（取消跟踪）

cd skywalking-agent
cp optional-plugins/apm-trace-ignore-plugin-8.10.0.jar plugins/
vim config/apm-trace-ignore-plugin.config 
#添加下面这段
trace.ignore_path=${SW_AGENT_TRACE_IGNORE_PATH:GET:/actuator/**,Redisson/**,Mysql/**,HikariCP/**,Lettuce/**,/xxl-job/**,UndertowDispatch/**}