1、ELK架构
1.1、部署ES集群
https://mirrors.tuna.tsinghua.edu.cn/elasticstack/apt/7.x/pool/main/e/elasticsearch/
1、下载软件包
root@es-server1:~# wget https://mirrors.tuna.tsinghua.edu.cn/elasticstack/apt/7.x/pool/main/e/elasticsearch/elasticsearch-7.12.0-amd64.deb2、安装es服务（es服务依赖java环境，从清华源下载的包默认包含了java环境）
root@es-server1:~# dpkg -i elasticsearch-7.12.0-amd64.deb3、修改es服务的配置文件
root@es-server1:~# vim /etc/elasticsearch/elasticsearch.yml17 cluster.name: log-cluster #集群的名称（这个必须唯一，每个es节点的集群名称必须保持一致）23 node.name: node1 #es节点的名称（每个es节点名称不能一样）37 path.logs: /data/log/elasticsearch #日志数据存放的路径56 network.host: 172.17.1.11 #es节点的IP61 http.port: 9200 #es的服务端口70 discovery.seed_hosts: ["172.17.1.11", "172.17.1.12","172.17.1.13"] #es集群节点的IP74 cluster.initial_master_nodes: ["172.17.1.11", "172.17.1.12","172.17.1.13"] #参与选用leader的es节点IP82 action.destructive_requires_name: true #用于提高集群操作的安全性。当这个参数被设置为true时，它要求所有具有破坏性的操作（如删除索引）必须指定明确的名称，而不允许使用通配符或者_全部_选项来执行操作4、设置es开机自启动
root@es-server1:~# systemctl enable elasticsearch.service
root@es-server1:~# systemctl start elasticsearch.service5、查看端口是否启动
root@es-server1:~# ss -ntl|grep 9200
LISTEN 0 4096 [::ffff:172.17.1.11]:9200 *:*注意：其余es的节点只需更改es配置文件里的network.host和node.name参数
1.2、部署kibana
https://mirrors.tuna.tsinghua.edu.cn/elasticstack/apt/7.x/pool/main/k/kibana/kibana-7.12.0-amd64.deb
（1）、下载安装包（包的版本要和es的版本保持一致）
root@es-server1:~# wget https://mirrors.tuna.tsinghua.edu.cn/elasticstack/apt/7.x/pool/main/k/kibana/kibana-7.12.0-amd64.deb
（2）、安装kibana服务
root@es-server1:~# dpkg -i kibana-7.12.0-amd64.deb
（3）、修改kibana配置文件
root@es-server1:/etc/kibana# vim kibana.yml
2 server.port: 5601 #kibana的服务端口
7 server.host: "172.17.1.11" #kibana的服务IP
32 elasticsearch.hosts: ["http://172.17.1.11:9200","http://172.17.1.12:9200","http://172.17.1.13:9200"] #kibana连接es集群的IP
111 i18n.locale: "zh-CN" #kibana的语言设置为中文
（4）设置kibana服务开机自启动
root@es-server1:/etc/kibana# systemctl enable kibana.service
root@es-server1:/etc/kibana# systemctl start kibana.service
root@es-server1:/etc/kibana# ss -ntl|grep 5601
LISTEN 0 511 172.17.1.11:5601 0.0.0.0:*
1.3、部署zookeeper
（1）下载zookeeper安装包
root@zk-kfk-1:~# wget https://archive.apache.org/dist/zookeeper/zookeeper-3.6.4/apache-zookeeper-3.6.4-bin.tar.gz
（2）解压安装包到指定目录
root@zk-kfk-1:~# mkdir /app
root@zk-kfk-1:~# tar -xvf apache-zookeeper-3.6.4-bin.tar.gz
（3)创建软链接
root@zk-kfk-1:~# ln -sv apache-zookeeper-3.6.4-bin zookeeper
（4)修改zookeeper配置文件
root@zk-kfk-1:~# cd /app/zookeeper/conf/
root@zk-kfk-1:/app/zookeeper/conf# cp zoo_sample.cfg zoo.cfg
root@zk-kfk-1:/app/zookeeper/conf# vim zoo.cfg
dataDir=/data/zookeeper #数据目录
clientPort=2181 #客户端连接zookeeper的端口
server.1=172.17.1.14:2888:3888
server.2=172.17.1.15:2888:3888
server.3=172.17.1.16:2888:3888（4）设置zookeeper节点id号
root@zk-kfk-1:/app/zookeeper/conf# echo "1" > /data/zookeeper/myid
(5)创建zookeeper启动文件
root@zk-kfk-1:/app/kafka/config# cat /etc/systemd/system/zookeeper.service
[Unit]
Description=Apache ZooKeeper Server
Documentation=http://zookeeper.apache.org
Requires=network.target
After=network.target[Service]
Type=forking
User=root
Group=root
ExecStart=/app/zookeeper/bin/zkServer.sh start
ExecStop=/app/zookeeper/bin/zkServer.sh stop
ExecRestart=/app/zookeeper/bin/zkServer.sh restart
WorkingDirectory=/data/zookeeper
Restart=on-abnormal[Install]
WantedBy=multi-user.target(6)设置zookeeper开机自启动
root@zk-kfk-1:/app/zookeeper/conf# systemctl enable zookeeper.service
root@zk-kfk-1:/app/zookeeper/conf# systemctl start zookeeper.service
(7)验证
root@zk-kfk-1:/app/zookeeper/conf# ss -ntl|grep 2181
LISTEN 0 50 *:2181 *:*1.4、部署kafka
（1）下载kafka安装包
root@zk-kfk-1:/app# wget https://archive.apache.org/dist/kafka/3.6.0/kafka_2.12-3.6.0.tgz
(2) 解压到指定路径
root@zk-kfk-1:/app# tar -xvf kafka_2.12-3.6.0.tgz -C /app
(3)创建软链接
root@zk-kfk-1:/app#ln -sv kafka_2.12-3.6.0 kafka
(4)修改kafka配置文件
root@zk-kfk-1:/app/kafka/config# vim server.properties
broker.id=14 #kafka的id号，每个节点的ID号必须保证不一致
listeners=PLAINTEXT://172.17.1.14:9092 #监听地址
log.dirs=/data/kafka-logs #存放日志的路径
zookeeper.connect=172.17.1.14:2181,172.17.1.15:2181,172.17.1.16:2181 #zookeeper集群IP
（5）安装java环境
root@zk-kfk-1:/app/kafka/config# apt -y install openjdk-11-jdk
（6）创建kafka启动文件service
root@zk-kfk-1:/app/kafka/config# cat /etc/systemd/system/kafka.service
[Unit]
Description=Apache Kafka Server
Documentation=http://kafka.apache.org/documentation.html
Requires=zookeeper.service
After=zookeeper.service[Service]
Type=simple
User=root
Group=root
Environment="JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64"
ExecStart=/app/kafka/bin/kafka-server-start.sh /app/kafka/config/server.properties
ExecStop=/app/kafka/bin/kafka-server-stop.sh
Restart=on-failure[Install]
WantedBy=multi-user.target
1.5、部署logstash
(1)下载安装包
root@logstach:~# wget https://mirrors.tuna.tsinghua.edu.cn/elasticstack/apt/7.x/pool/main/l/logstash/logstash-7.12.0-amd64.deb
(2)安装java环境和logstash服务
root@logstach:~# dpkg -i logstash-7.12.0-amd64.deb
root@logstach:~# apt -y install openjdk-11-jdk
2、日志收集的方法
2.1基于daemonset收集日志
基于daemonset运行日志收集服务，主要收集以下类型日志：
- node节点收集，基于daemonset部署日志收集进程，实现json-file类型（标准输出/dev/stdout、错误输出/dev/stderr)日志收集，即应用程序产生的标准输出和错误输出的日志
- 宿主机系统日志等以日志文件形式保存的日志
日志收集架构图：
日志收集的流程：
- pod里的应用容器将自身的json格式日志写入到node宿主机上的某个文件（如json-file1)
- 部署在node节点上的logstash容器收集该node宿主机上存储的应用容器的日志文件（json-file1)
- logstash容器将收集到的应用容器的日志转存到kafka集群
- logstash组件从kafka集群获取到pod应用容器的日志并将其转存到es集群，es集群进行日志的分析
- kibana从es上查询获取到的日志信息，然后展示出来，用户从kibana的web界面获取日志
注意：这里需要将node上存储的应用容器的日志目录挂载到logstash容器上，使用hostpath类型的存储卷，便于logstash收集容器日志
2.1.1dockerfile编写logstash镜像
root@k8s-harbor:~/dockerfile/web/daemonset-logstash# ll
total 16
drwxr-xr-x 2 root root 89 Mar 11 21:23 ./
drwxrwxr-x 11 root root 145 Mar 11 21:00 ../
-rw-r--r-- 1 root root 219 Mar 11 21:11 Dockerfile
-rwxr-xr-x 1 root root 169 Mar 11 21:23 build-commond.sh*
-rw-r--r-- 1 root root 805 Mar 11 21:07 logstash.conf
-rw-r--r-- 1 root root 92 Mar 11 21:07 logstash.yml1、logstash配置文件
root@k8s-harbor:~/dockerfile/web/daemonset-logstash# cat logstash.conf
input { #定义来源的日志文件 file {#path => "/var/lib/docker/containers/*/*-json.log" #dockerpath => "/var/log/pods/*/*/*.log" #表示Logstash会监视匹配该路径下的所有日志文件start_position => "beginning" #表示Logstash启动时从日志文件的开始位置读取日志，而不是仅读新追加的内容type => "jsonfile-daemonset-applog" #给输入的日志流赋予一个类型标签，以便后续处理时进行区分（相当于es的一个索引）}file {path => "/var/log/*.log"start_position => "beginning"type => "jsonfile-daemonset-syslog"}
}output { #定义日志文件输出到kafkaif [type] == "jsonfile-daemonset-applog" { kafka { #使用kafka作为日志输出的目的地bootstrap_servers => "${KAFKA_SERVER}" #kafka集群的地址，从环境变量获取，避免后续需手动更改topic_id => "${TOPIC_ID}" #kafka的主题IDbatch_size => 16384 #logstash每次向ES传输的数据量大小,单位为字节codec => "${CODEC}" #数据的编码格式} }if [type] == "jsonfile-daemonset-syslog" {kafka {bootstrap_servers => "${KAFKA_SERVER}"topic_id => "${TOPIC_ID}"batch_size => 16384codec => "${CODEC}" #系统日志不是json格式}}
}
2、logstash的yml文件
root@k8s-harbor:~/dockerfile/web/daemonset-logstash# cat logstash.yml
http.host: "0.0.0.0"
#xpack.monitoring.elasticsearch.hosts: [ "http://elasticsearch:9200" ] #关闭xpack功能，该功能收费，如果不关闭将无法正常使用logstash3、编写Logstash的Dockerfile镜像文件
root@k8s-harbor:~/dockerfile/web/daemonset-logstash# cat Dockerfile
FROM logstash:7.12.0
USER root
WORKDIR /usr/share/logstash
#RUN rm -rf config/logstash-sample.conf
ADD logstash.yml /usr/share/logstash/config/logstash.yml
ADD logstash.conf /usr/share/logstash/pipeline/logstash.conf4、构造镜像文件脚本
root@k8s-harbor:~/dockerfile/web/daemonset-logstash# cat build-commond.sh
#!/bin/bash
docker build -t harbor.qiange.com/baseimages/logstash:v7.12.0-json-file-log-v4 .
docker push harbor.qiange.com/baseimages/logstash:v7.12.0-json-file-log-v4
2.1.2通过daemonset部署logstash容器
root@k8s-master1:/app/yaml/daemonset-logstash# cat DaemonSet-logstash.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:name: logstash-elasticsearchnamespace: kube-systemlabels:k8s-app: logstash-logging
spec:selector:matchLabels:name: logstash-elasticsearchtemplate:metadata:labels:name: logstash-elasticsearchspec:tolerations:# this toleration is to have the daemonset runnable on master nodes# remove it if your masters can't run pods- key: node-role.kubernetes.io/masteroperator: Existseffect: NoSchedulecontainers:- name: logstash-elasticsearchimage: harbor.qiange.com/baseimages/logstash:v7.12.0-json-file-log-v4env:- name: "KAFKA_SERVER"value: "172.17.1.14:9092,172.17.1.15:9092,172.17.1.16:9092"- name: "TOPIC_ID"value: "jsonfile-log-topic"- name: "CODEC"value: "json"
# resources:
# limits:
# cpu: 1000m
# memory: 1024Mi
# requests:
# cpu: 500m
# memory: 1024MivolumeMounts:- name: varlog #定义宿主机系统日志挂载路径mountPath: /var/log #宿主机系统日志挂载点- name: varlibdockercontainers #定义容器日志挂载路径,和logstash配置文件中的收集路径保持一直#mountPath: /var/lib/docker/containers #docker挂载路径mountPath: /var/log/pods #containerd挂载路径,此路径与logstash的日志收集路径必须一致readOnly: falseterminationGracePeriodSeconds: 30volumes:- name: varloghostPath:path: /var/log #宿主机系统日志- name: varlibdockercontainershostPath:path: /var/lib/docker/containers #docker的宿主机日志路径path: /var/log/pods #containerd的宿主机日志路径#验证
root@k8s-master1:/app/yaml/daemonset-logstash# kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
calico-kube-controllers-59df8b6856-tx7h2 1/1 Running 22 (46m ago) 50d
calico-node-45vdl 1/1 Running 0 19m
calico-node-5dqjz 1/1 Running 0 17m
calico-node-9z6cn 1/1 Running 0 17m
calico-node-bt6zr 1/1 Running 0 16m
calico-node-ct56r 1/1 Running 0 17m
calico-node-dnbwt 1/1 Running 0 16m
coredns-67cb59d684-9jrbw 1/1 Running 14 (46m ago) 46d
logstash-elasticsearch-457h6 1/1 Running 0 7m56s
logstash-elasticsearch-b6g75 1/1 Running 0 7m56s
logstash-elasticsearch-bc5dd 1/1 Running 0 7m56s
logstash-elasticsearch-bhc85 1/1 Running 0 7m56s
logstash-elasticsearch-ctt5r 1/1 Running 0 7m56s
logstash-elasticsearch-kt6bh 1/1 Running 0 7m56s
2.1.3配置logstash服务，使其将数据转存到es集群上
root@logstach:~# cd /etc/logstash/conf.d/
root@logstach:/etc/logstash/conf.d# cat logsatsh-daemonset-jsonfile-kafka-to-es.conf
input {kafka {bootstrap_servers => "172.17.1.14:9092,172.17.1.15:9092,172.17.1.16:9092"topics => ["jsonfile-log-topic"]codec => "json"}
}
output {#if [fields][type] == "app1-access-log" {if [type] == "jsonfile-daemonset-applog" {elasticsearch {hosts => ["172.17.1.11:9200","172.17.1.12:9200","172.17.1.13:9200"]index => "jsonfile-daemonset-applog-%{+YYYY.MM.dd}"}}if [type] == "jsonfile-daemonset-syslog" {elasticsearch {hosts => ["172.17.1.11:9200","172.17.1.12:9200","172.17.1.13:9200"]index => "jsonfile-daemonset-syslog-%{+YYYY.MM.dd}"}}}
2.2基于sidecar模式收集日志
使用sidecar容器（一个pod多容器）收集当前pod内一个或者多个业务容器的日志（通常基于emptyDir实现业务容器与sidcar之间的日志共享）
架构图
日志收集流程
- 应用容器将自身的日志写入emptydir存储卷中，同一个pod中的sidecar容器通过挂载emptydir存储卷到容器内获取应用容器的日志
- 然后将日志写入kafka，logstash从kafka获取日志
- kafka获取到日志后将其转存到es集群，es集群对日志进行分析
- 用户从kibana上获取日志，kibana从es上查询获取日志信息，然后展示出来
2.2.1 编写sidecar容器镜像
root@k8s-harbor:~/dockerfile/web/sidecar-logstash# ll
total 16
drwxr-xr-x 2 root root 89 Mar 13 12:26 ./
drwxrwxr-x 12 root root 169 Mar 12 17:20 ../
-rw-r--r-- 1 root root 221 Mar 12 17:26 Dockerfile
-rwxr-xr-x 1 root root 299 Mar 13 10:01 build-commond.sh*
-rw-r--r-- 1 root root 742 Mar 13 09:18 logstash.conf
-rw-r--r-- 1 root root 92 May 23 2022 logstash.yml#编写lonstash配置文件
root@k8s-harbor:~/dockerfile/web/sidecar-logstash# cat logstash.conf
input {file {path => "/var/log/applogs/catalina.out"start_position => "beginning"type => "app1-sidecar-catalina-log" }file {path => "/var/log/applogs/localhost_access_log.*.txt"start_position => "beginning"type => "app1-sidecar-access-log"}
}output {if [type] == "app1-sidecar-catalina-log" {kafka {bootstrap_servers => "${KAFKA_SERVER}"topic_id => "${TOPIC_ID}"batch_size => 16384 #logstash每次向ES传输的数据量大小,单位为字节codec => "${CODEC}"} }if [type] == "app1-sidecar-access-log" {kafka {bootstrap_servers => "${KAFKA_SERVER}"topic_id => "${TOPIC_ID}"batch_size => 16384codec => "${CODEC}"}}
}#配置logstash.yml文件
root@k8s-harbor:~/dockerfile/web/sidecar-logstash# cat logstash.yml
http.host: "0.0.0.0"
#xpack.monitoring.elasticsearch.hosts: [ "http://elasticsearch:9200" ]#配置logstash容器的镜像文件
root@k8s-harbor:~/dockerfile/web/sidecar-logstash# cat Dockerfile
FROM logstash:7.12.0
USER root
WORKDIR /usr/share/logstash
#RUN rm -rf config/logstash-sample.conf
ADD logstash.yml /usr/share/logstash/config/logstash.yml
ADD logstash.conf /usr/share/logstash/pipeline/logstash.conf#配置编写镜像的脚本文件
root@k8s-harbor:~/dockerfile/web/sidecar-logstash# cat build-commond.sh
#!/bin/bash
docker build -t harbor.qiange.com/baseimages/logstash:v7.12.0-sidecar-1 .
docker push harbor.qiange.com/baseimages/logstash:v7.12.0-sidecar-1
#nerdctl build -t harbor.magedu.net/baseimages/logstash:v7.12.1-sidecar .
#nerdctl push harbor.magedu.net/baseimages/logstash:v7.12.1-sidecar
2.2.2 通过deployment部署应用容器和sidecar容器
root@k8s-master1:/app/yaml/sidecar-logstash# cat tomcat-app1.yaml
kind: Deployment
#apiVersion: extensions/v1beta1
apiVersion: apps/v1
metadata:labels:app: magedu-tomcat-app1-deployment-labelname: magedu-tomcat-app1-deployment #当前版本的deployment 名称namespace: qiange
spec:replicas: 3selector:matchLabels:app: magedu-tomcat-app1-selectortemplate:metadata:labels:app: magedu-tomcat-app1-selectorspec:containers:- name: sidecar-containerimage: harbor.qiange.com/baseimages/logstash:v7.12.0-sidecar-1imagePullPolicy: IfNotPresent#imagePullPolicy: Alwaysenv:- name: "KAFKA_SERVER"value: "172.17.1.14:9092,172.17.1.15:9092,172.17.1.16:9092"- name: "TOPIC_ID"value: "tomcat-app-topic"- name: "CODEC"value: "json"volumeMounts:- name: applogsmountPath: /var/log/applogs- name: magedu-tomcat-app1-containerimage: harbor.qiange.com/tomcat/tomcat-app1:v1imagePullPolicy: IfNotPresent#imagePullPolicy: Alwaysports:- containerPort: 8080protocol: TCPname: httpenv:- name: "password"value: "123456"- name: "age"value: "18"resources:limits:cpu: 1memory: "512Mi"requests:cpu: 500mmemory: "512Mi"volumeMounts:- name: applogsmountPath: /apps/tomcat/logsstartupProbe:httpGet:path: /app1/index.jspport: 8080initialDelaySeconds: 5 #首次检测延迟5sfailureThreshold: 3 #从成功转为失败的次数periodSeconds: 3 #探测间隔周期readinessProbe:httpGet:#path: /monitor/monitor.htmlpath: /app1/index.jspport: 8080initialDelaySeconds: 5periodSeconds: 3timeoutSeconds: 5successThreshold: 1failureThreshold: 3livenessProbe:httpGet:#path: /monitor/monitor.htmlpath: /app1/index.jspport: 8080initialDelaySeconds: 5periodSeconds: 3timeoutSeconds: 5successThreshold: 1failureThreshold: 3volumes:- name: applogs #定义通过emptyDir实现业务容器与sidecar容器的日志共享，以让sidecar收集业务容器中的日志emptyDir: {}
2.2.3配置访问业务的service文件
root@k8s-master1:/app/yaml/sidecar-logstash# cat tomcat-service.yaml
---
kind: Service
apiVersion: v1
metadata:labels:app: magedu-tomcat-app1-service-labelname: magedu-tomcat-app1-servicenamespace: qiange
spec:type: NodePortports:- name: httpport: 80protocol: TCPtargetPort: 8080nodePort: 30080selector:app: magedu-tomcat-app1-selector
2.2.4配置logstash服务，使其将数据转存到es集群上
root@logstach:/etc/logstash/conf.d# cat logsatsh-sidecar-kafka-to-es.conf
input {kafka {bootstrap_servers => "172.17.1.14:9092,172.17.1.15:9092,172.17.1.16:9092"topics => ["tomcat-app-topic"]codec => "json"}
}
output {#if [fields][type] == "app1-access-log" {if [type] == "app1-sidecar-access-log" {elasticsearch {hosts => ["172.17.1.11:9200","172.17.1.12:9200","172.17.1.13:9200"]index => "sidecar-app1-accesslog-%{+YYYY.MM.dd}"}}#if [fields][type] == "app1-catalina-log" {if [type] == "app1-sidecar-catalina-log" {elasticsearch {hosts => ["172.17.1.11:9200","172.17.1.12:9200","172.17.1.13:9200"]index => "sidecar-app1-catalinalog-%{+YYYY.MM.dd}"}}
# stdout {
# codec => rubydebug
# }
}
root@logstach:/etc/logstash/conf.d# systemctl restart logrotate.service
2.2.5验证
2.2.5.1pod正常启动
2.2.5.2kafka获取到日志信息
2.2.5.3 es获取到日志数据
注意：如果kafka获取到数据但是es上没有获取到数据，可以访问一下业务或者调整一下pod的副本数
业务日志
系统日志
2.3基于应用容器内置filebeat收取日志
2.3.1 编写filebeat镜像文件
root@k8s-harbor:~/dockerfile/web/filebeat-tomcat# ll
total 31856
drwxr-xr-x 3 root root 172 Mar 14 15:14 ./
drwxrwxr-x 14 root root 208 Mar 13 13:16 ../
-rw-rw-r-- 1 root root 408 Mar 14 15:14 Dockerfile
drwxr-xr-x 2 root root 23 Feb 20 10:38 app1/
-rw-r--r-- 1 root root 193 Feb 20 10:52 app1.tar.gz
-rwxrwxr-x 1 root root 155 Mar 13 13:26 build-command.sh*
-rw-r--r-- 1 root root 32590134 Mar 13 21:54 filebeat-7.12.0-x86_64.rpm
-rw-r--r-- 1 root root 727 Mar 13 22:06 filebeat.yml
-rw-rw-r-- 1 root root 265 Mar 13 14:10 run_tomcat.sh
-rw-rw-r-- 1 root root 7593 Jan 19 10:21 server.xml#dockerfile文件
root@k8s-harbor:~/dockerfile/web/filebeat-tomcat# cat Dockerfile
FROM harbor.qiange.com/tomcat/tomcat-app1:v1
ADD run_tomcat.sh /apps/tomcat/bin
ADD filebeat-7.12.0-x86_64.rpm /root
RUN yum -y remove filebeat && rpm -ivh /root/filebeat-7.12.0-x86_64.rpm
RUN rm -f /etc/filebeat/filebeat.yml
ADD filebeat.yml /etc/filebeat
RUN chmod 777 /apps/tomcat/bin/run_tomcat.sh && chown www.www /apps/tomcat/bin/run_tomcat.sh
CMD ["/bin/sh", "-c", "/apps/tomcat/bin/run_tomcat.sh"]#filebeat的配置文件
root@k8s-harbor:~/dockerfile/web/filebeat-tomcat# cat filebeat.yml
filebeat.inputs:
- type: logenabled: truepaths:- /apps/tomcat/logs/catalina.outfields:type: filebeat-tomcat-catalina
- type: logenabled: truepaths:- /apps/tomcat/logs/localhost_access_log.*.txtfields:type: filebeat-tomcat-accesslog
filebeat.config.modules:path: ${path.config}/modules.d/*.ymlreload.enabled: false
setup.template.settings:index.number_of_shards: 1
setup.kibana:output.kafka:hosts: ["172.17.1.14:9092","172.17.1.15:9092","172.17.1.16:9092"]required_acks: 1topic: "filebeat-magedu-app1"compression: gzipmax_message_bytes: 1000000
#output.redis:
# hosts: ["172.31.2.105:6379"]
# key: "k8s-magedu-app1"
# db: 1
# timeout: 5
# password: "123456"#服务启动脚本
root@k8s-harbor:~/dockerfile/web/filebeat-tomcat# cat run_tomcat.sh
#!/bin/bash
/usr/share/filebeat/bin/filebeat -e -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat &
su - root -c "/apps/tomcat/bin/catalina.sh start"
tail -f /etc/hosts#构建镜像脚本
root@k8s-harbor:~/dockerfile/web/filebeat-tomcat# cat build-command.sh
#!/bin/bash
docker build -t harbor.qiange.com/tomcat/tomcat-app1-filebeat-7.12.0:v1 .
docker push harbor.qiange.com/tomcat/tomcat-app1-filebeat-7.12.0:v1
2.3.2 使用Deployment控制器部署应用
root@k8s-master1:/app/yaml/filebeat-log# cat tomcat-app2.yaml
kind: Deployment
#apiVersion: extensions/v1beta1
apiVersion: apps/v1
metadata:labels:app: magedu-tomcat-app1-filebeat-deployment-labelname: magedu-tomcat-app1-filebeat-deploymentnamespace: qiange
spec:replicas: 3selector:matchLabels:app: magedu-tomcat-app1-filebeat-selectortemplate:metadata:labels:app: magedu-tomcat-app1-filebeat-selectorspec:containers:- name: magedu-tomcat-app1-filebeat-containerimage: harbor.qiange.com/tomcat/tomcat-app1-filebeat-7.12.0:v1imagePullPolicy: IfNotPresent#imagePullPolicy: Alwaysports:- containerPort: 8080protocol: TCPname: httpenv:- name: "password"value: "123456"- name: "age"value: "18"resources:limits:cpu: 1memory: "512Mi"requests:cpu: 500mmemory: "512Mi"2.3.3 部署service
root@k8s-master1:/app/yaml/filebeat-log# cat tomcat-service.yaml
---
kind: Service
apiVersion: v1
metadata:labels:app: magedu-tomcat-app1-filebeat-service-labelname: magedu-tomcat-app1-filebeat-servicenamespace: qiange
spec:type: NodePortports:- name: httpport: 81protocol: TCPtargetPort: 8080nodePort: 30092selector:app: magedu-tomcat-app1-filebeat-selector
2.3.4 配置logstash服务，使其将数据转存到es集群上
root@logstach:~# cd /etc/logstash/conf.d/
root@logstach:/etc/logstash/conf.d# cat logstash-filebeat-process-kafka-to-es.conf
input {kafka {bootstrap_servers => "172.17.1.14:9092,172.17.1.15:9092,172.17.1.16:9092"topics => ["filebeat-magedu-app1"]codec => "json"}
}output {if [fields][type] == "filebeat-tomcat-catalina" {elasticsearch {hosts => ["172.17.1.11:9200","172.17.1.12:9200","172.17.1.13:9200"]index => "filebeat-tomcat-catalina-%{+YYYY.MM.dd}"}}if [fields][type] == "filebeat-tomcat-accesslog" {elasticsearch {hosts => ["172.17.1.11:9200","172.17.1.12:9200","172.17.1.13:9200"]index => "filebeat-tomcat-accesslog-%{+YYYY.MM.dd}"}}}
2.3.5验证
2.3.5.1 pod启动正常
2.3.5.2 kafka获取日志数据
2.3.5.3 es获取到来自kafka转存的数据
业务日志：
系统日志：
