【k8s部署elasticsearch】k8s环境下安装elasticsearch集群和kibana

文章目录

- - 简介
  - 一.条件及环境说明
  - 二.需求说明
  - 三.实现原理及说明
  - 四.详细步骤
  - - 4.1.规划节点标签
    - 4.2.创建三个statefulset和service headless配置
    - 4.3.创建service配置
  - 五.安装kibana
  - 六.调整索引分区
  - 七.安装说明

简介

k8s集群中搭建有elasticsearch服务一般都会用到pvc，但是考虑到有些自建k8s环境下，搭建的共享存储可能会存在稳定性及性能问题，所以这次是通过采用节点亲和性和hostpath来实现，目前的operator的基本都是采用共享存储的方法。本文将根据现有环境及不同需求将elasticsearch集群的搭建采用hostpath+亲和性的权重+多个副本分区的方式来实现数据持久化和高可用。

一.条件及环境说明

k8s版本k8s-1.29.4，环境搭建在电信机房,六个worker节点，每个节点有一块非系统盘的ssd盘挂载到/data/路径，不采用ceph或nas之类的共享存储，也未采用operator。

二.需求说明

搭建三个节点的elasticsearch集群，并配置三个主分区，2个副本分区。
高可用：有一个k8s节点长期故障后或重启pod之后也不影响使用。
高性能：读写本地磁盘实现高性能的io，共享存储如果资源及硬件性能不够的话，io将会是性能瓶颈。
数据安全：在发生切换,节点故障以后，数据要尽可能的保证完整
安装简单、管理维护容易

三.实现原理及说明

四.详细步骤

4.1.规划节点标签

节点名称	角色	标签
k8s-worker-120-81	主	storage-selector=node-a-master
k8s-worker-120-82	主	storage-selector=node-b-master
k8s-worker-120-83	主	storage-selector=node-c-master
k8s-worker-120-84	备	storage-selector=node-a-slve
k8s-worker-120-85	备	storage-selector=node-a-slve
k8s-worker-120-86	备	storage-selector=node-a-slve

注：这里的角色划分是指：redis由于是一主两从，每个redis的pod分别落在三个主的节点上，当主节点发生故障是，就切换到备节点，例如：名称为autopocket-es-a-0的pod，默认落在标签为storage-selector=node-a-master的节点上，当该节点发生故障是，就会根据权重匹配罗在storage-selector=node-a-slve的节点上。

4.2.创建三个statefulset和service headless配置

1.elasticsearch_a.yaml 节点a的statuflset配置和无头服务配置

apiVersion: apps/v1
kind: StatefulSet
metadata:name: autopocket-es-anamespace: zx-applabels:app: autopocket-es-a appname: pcauto-pocketappgroup: autopocket-es
spec:serviceName: "autopocket-es-a"replicas: 1 selector:matchLabels:app: autopocket-es-aappgroup: autopocket-estemplate:metadata:labels:app: autopocket-es-aappgroup: autopocket-esspec:affinity:nodeAffinity:preferredDuringSchedulingIgnoredDuringExecution:- weight: 80preference:matchExpressions:- key: storage-selector operator: Invalues:- node-a-master- weight: 20preference:matchExpressions:- key: storage-selector operator: Invalues:- node-a-slavecontainers:- name: elasticsearchimage: "xxx.cn-shenzhen.cr.aliyuncs.com/public/elasticsearch:7.17.22"command: ["/bin/sh","-c"]args:- mkdir -p /home/elastic/${POD_NAME}/data /home/elastic/${POD_NAME}/logs;chown elasticsearch.elasticsearch -R /home/elastic;su elasticsearch -c "/usr/share/elasticsearch/bin/elasticsearch"env:- name: cluster.namevalue: autopocket-es - name: K8S_SERVICE_NAMEvalue: autopocket-es-a - name: POD_NAMEvalueFrom:fieldRef:fieldPath: metadata.name- name: POD_NAMESPACEvalueFrom:fieldRef:fieldPath: metadata.namespace- name: discovery.seed_hostsvalue: "autopocket-es-a-0.autopocket-es-a.$(POD_NAMESPACE).svc.cluster.local,autopocket-es-b-0.autopocket-es-b.$(POD_NAMESPACE).svc.cluster.local,autopocket-es-c-0.autopocket-es-c.$(POD_NAMESPACE).svc.cluster.local"- name: cluster.initial_master_nodesvalue: "autopocket-es-a-0.autopocket-es-a.$(POD_NAMESPACE).svc.cluster.local"- name: node.namevalue: $(POD_NAME).$(K8S_SERVICE_NAME).$(POD_NAMESPACE).svc.cluster.local - name: path.datavalue: /home/elastic/$(POD_NAME)/data- name: path.logsvalue: /home/elastic/$(POD_NAME)/logs- name: xpack.security.enabledvalue: "false"ports:- name: restcontainerPort: 9200- name: inter-nodecontainerPort: 9300volumeMounts:- name: elasticsearch-datamountPath: /home/elasticrestartPolicy: AlwaysterminationGracePeriodSeconds: 30volumes:- name: elasticsearch-data hostPath:path: /data/es_datatype: ""tolerations:- key: node.kubernetes.io/not-readyoperator: Existseffect: NoExecutetolerationSeconds: 3600- key: node.kubernetes.io/unreachableoperator: Existseffect: NoExecutetolerationSeconds: 3600
---
apiVersion: v1
kind: Service
metadata:name: autopocket-es-anamespace: zx-applabels:appname: pcauto-pocketapp: autopocket-es-a
spec:ports:- port: 9200name: rest- port: 9300name: inter-nodeclusterIP: Noneselector:app: autopocket-es-a

2.elasticsearch_b.yaml 节点b的statuflset配置和无头服务配置

apiVersion: apps/v1
kind: StatefulSet
metadata:name: autopocket-es-bnamespace: zx-applabels:app: autopocket-es-b appname: pcauto-pocketappgroup: autopocket-es
spec:serviceName: "autopocket-es-b"replicas: 1 selector:matchLabels:app: autopocket-es-bappgroup: autopocket-estemplate:metadata:labels:app: autopocket-es-bappgroup: autopocket-esspec:affinity:nodeAffinity:preferredDuringSchedulingIgnoredDuringExecution:- weight: 80preference:matchExpressions:- key: storage-selector operator: Invalues:- node-b-master- weight: 20preference:matchExpressions:- key: storage-selector operator: Invalues:- node-b-slavecontainers:- name: elasticsearchimage: "xxx.cn-shenzhen.cr.aliyuncs.com/public/elasticsearch:7.17.22"command: ["/bin/sh","-c"]args:- mkdir -p /home/elastic/${POD_NAME}/data /home/elastic/${POD_NAME}/logs;chown elasticsearch.elasticsearch -R /home/elastic;su elasticsearch -c "/usr/share/elasticsearch/bin/elasticsearch"env:- name: cluster.namevalue: autopocket-es - name: K8S_SERVICE_NAMEvalue: autopocket-es-b - name: POD_NAMEvalueFrom:fieldRef:fieldPath: metadata.name- name: POD_NAMESPACEvalueFrom:fieldRef:fieldPath: metadata.namespace- name: discovery.seed_hostsvalue: "autopocket-es-a-0.autopocket-es-a.$(POD_NAMESPACE).svc.cluster.local,autopocket-es-b-0.autopocket-es-b.$(POD_NAMESPACE).svc.cluster.local,autopocket-es-c-0.autopocket-es-c.$(POD_NAMESPACE).svc.cluster.local"- name: cluster.initial_master_nodesvalue: "autopocket-es-a-0.autopocket-es-a.$(POD_NAMESPACE).svc.cluster.local"- name: node.namevalue: $(POD_NAME).$(K8S_SERVICE_NAME).$(POD_NAMESPACE).svc.cluster.local - name: path.datavalue: /home/elastic/$(POD_NAME)/data- name: path.logsvalue: /home/elastic/$(POD_NAME)/logs- name: xpack.security.enabledvalue: "false"ports:- name: restcontainerPort: 9200- name: inter-nodecontainerPort: 9300volumeMounts:- name: elasticsearch-datamountPath: /home/elasticrestartPolicy: AlwaysterminationGracePeriodSeconds: 30volumes:- name: elasticsearch-data hostPath:path: /data/es_datatype: ""tolerations:- key: node.kubernetes.io/not-readyoperator: Existseffect: NoExecutetolerationSeconds: 3600- key: node.kubernetes.io/unreachableoperator: Existseffect: NoExecutetolerationSeconds: 3600
---
apiVersion: v1
kind: Service
metadata:name: autopocket-es-bnamespace: zx-applabels:appname: pcauto-pocketapp: autopocket-es-b
spec:ports:- port: 9200name: rest- port: 9300name: inter-nodeclusterIP: Noneselector:app: autopocket-es-b

3.elasticsearch_c.yaml 节点c的statuflset配置和无头服务配置

apiVersion: apps/v1
kind: StatefulSet
metadata:name: autopocket-es-cnamespace: zx-applabels:app: autopocket-es-c appname: pcauto-pocketappgroup: autopocket-es
spec:serviceName: "autopocket-es-c"replicas: 1 selector:matchLabels:app: autopocket-es-cappgroup: autopocket-estemplate:metadata:labels:app: autopocket-es-cappgroup: autopocket-esspec:affinity:nodeAffinity:preferredDuringSchedulingIgnoredDuringExecution:- weight: 80preference:matchExpressions:- key: storage-selector operator: Invalues:- node-c-master- weight: 20preference:matchExpressions:- key: storage-selector operator: Invalues:- node-c-slavecontainers:- name: elasticsearchimage: "xxx.cn-shenzhen.cr.aliyuncs.com/public/elasticsearch:7.17.22"command: ["/bin/sh","-c"]args:- mkdir -p /home/elastic/${POD_NAME}/data /home/elastic/${POD_NAME}/logs;chown elasticsearch.elasticsearch -R /home/elastic;su elasticsearch -c "/usr/share/elasticsearch/bin/elasticsearch"env:- name: cluster.namevalue: autopocket-es - name: K8S_SERVICE_NAMEvalue: autopocket-es-c - name: POD_NAMEvalueFrom:fieldRef:fieldPath: metadata.name- name: POD_NAMESPACEvalueFrom:fieldRef:fieldPath: metadata.namespace- name: discovery.seed_hostsvalue: "autopocket-es-a-0.autopocket-es-a.$(POD_NAMESPACE).svc.cluster.local,autopocket-es-b-0.autopocket-es-b.$(POD_NAMESPACE).svc.cluster.local,autopocket-es-c-0.autopocket-es-c.$(POD_NAMESPACE).svc.cluster.local"- name: cluster.initial_master_nodesvalue: "autopocket-es-a-0.autopocket-es-a.$(POD_NAMESPACE).svc.cluster.local"- name: node.namevalue: $(POD_NAME).$(K8S_SERVICE_NAME).$(POD_NAMESPACE).svc.cluster.local - name: path.datavalue: /home/elastic/$(POD_NAME)/data- name: path.logsvalue: /home/elastic/$(POD_NAME)/logs- name: xpack.security.enabledvalue: "false"ports:- name: restcontainerPort: 9200- name: inter-nodecontainerPort: 9300volumeMounts:- name: elasticsearch-datamountPath: /home/elasticrestartPolicy: AlwaysterminationGracePeriodSeconds: 30volumes:- name: elasticsearch-data hostPath:path: /data/es_datatype: ""tolerations:- key: node.kubernetes.io/not-readyoperator: Existseffect: NoExecutetolerationSeconds: 3600- key: node.kubernetes.io/unreachableoperator: Existseffect: NoExecutetolerationSeconds: 3600
---
apiVersion: v1
kind: Service
metadata:name: autopocket-es-cnamespace: zx-applabels:appname: pcauto-pocketapp: autopocket-es-c
spec:ports:- port: 9200name: rest- port: 9300name: inter-nodeclusterIP: Noneselector:app: autopocket-es-c

4.3.创建service配置

给9200 9300配置一个service的代理服务，注意其中的selector是指向到appgroup,这是将三个statefulset定义了一个appgroup的标签。

apiVersion: v1
kind: Service
metadata:name: autopocket-esnamespace: zx-applabels:app: autopocket-esappname: pcauto-zx
spec:ports:- port: 9200name: rest- port: 9300name: inter-nodeselector:appgroup: autopocket-estype: LoadBalancer

五.安装kibana

apiVersion: apps/v1
kind: Deployment
metadata:name: autopocket-es-kibananamespace: zx-applabels:app: autopocket-es-kibanaappname: pcauto-pocket
spec:replicas: 1selector:matchLabels:app: autopocket-es-kibanatemplate:metadata:labels:app: autopocket-es-kibanaspec:containers:- name: kibanaimage: "xxx.cn-shenzhen.cr.aliyuncs.com/public/kibana:7.17.22"ports:- containerPort: 5601env:- name: ELASTICSEARCH_HOSTSvalue: '["http://autopocket-es-a-0.autopocket-es-a:9200","http://autopocket-es-b-0.autopocket-es-b:9200","http://autopocket-es-c-0.autopocket-es-c:9200"]'
---
apiVersion: v1
kind: Service
metadata:name: autopocket-es-kibananamespace: zx-applabels:app: autopocket-es-kibanaappname: pcauto-pocket
spec:selector:app: autopocket-es-kibanaports:- port: 5601targetPort: 5601type: LoadBalancer

查看集群健康情况：http://10.16.xx.xx:9200/_cat/health?v

elasticsearch集群状态查看：http://10.16.1xx.xx:9200/_cat/nodes?v

索引状态查看：http://10.16.1xx.xx:9200/_cat/indices?v

kibana地址登录：http://10.16.1xx.xx:5601/app/home#/

六.调整索引分区

调整已存在索引的副本分区：curl -XPUT -H "Content-Type:application/json" -d '{"settings":{"number_of_replicas":2}}' 'http://10.16.xx.xx:9200/testindex/_settings'

调整集群最大分片数：curl -XPUT -H "Content-Type:application/json" http://10.16.xx.xx:9200/_cluster/settings -d '{"transient":{"cluster":{"max_shards_per_node":10000}}}'

调整默认主分片和副本分片数: curl -X PUT -H "Content-Type:application/json" -d '{"index_patterns":["*"],"settings":{"number_of_shards":3,"number_of_replicas":2}}' http://10.16.xx.xx:9200/_template/template_http_request_record

七.安装说明

需要先在k8s的节点创建/data/es_data的路径来存放数据。
使用配置前将配置中的autopocket替换成自己需要的命名,配置的命名空间为：zx-app，修改成自己的命名空间，appname为pcauto-pocket，修改成自己的应用名称，该标签也可以删除。
配置中的镜像地址采用了私有的镜像地址，镜像是dockerhub上下载的elasticsearch:7.17.22 镜像到私有镜像仓库的，如果能直通外网的k8s可以直接用elasticsearch:7.17.22,kibana的镜像是kibana:7.17.22。否则配置一个可以通的镜像地址。
默认没有配置用户名和密码。
节点标签的命名规则若有更改需要在每个statefulset中的affinity: 中的标签名称进行修改。