Microk8s Ingress实现七层负载均衡

Microk8s Ingress是什么

Ingress是k8s的一种资源对象，用于管理外部对集群内服务的访问, 它通过提供一个统一的入口点，将外部流量路由到集群内部的不同服务。

Microk8s Ingress用于解决什么问题

k8s集群中服务默认只能在集群内访问。如果需要从外部访问服务，通常需要使用NodePort或LoadBalancer类型服务，这两个服务都存在一些问题

NodePort会占用节点端口，可能导致端口冲突
LoadBalancer需要云提供商支持, 不适合本地环境
Ingress提供一种灵活的方式暴露服务，允许通过域名或路径规则将流量路由到不同的服务

Microk8s Ingress基本原理

Microk8s内置了一个Nginx的Ingress Controller, 负责监听k8s中的Ingress资源，当检测到Ingress资源更新时, 动态更新Nginx配置文件
外部流量先到达Nginx, 再基于域名和URL将请求转发到Service, Service再将流量分发到Pod

# kubectl -n ingress get pod nginx-ingress-microk8s-controller-nrftt
NAME                                      READY   STATUS    RESTARTS   AGE
nginx-ingress-microk8s-controller-nrftt   1/1     Running   0          15m# ps axf 860646 ?        Sl     0:00 /var/lib/snapd/snap/microk8s/7665/bin/containerd-shim-runc-v2 -namespace k8s.io -id 8fb71f54235bf26664240c8b9272b623a70d860668 ?        Ss     0:00  \_ /pause860700 ?        Ss     0:00  \_ /usr/bin/dumb-init -- /nginx-ingress-controller --configmap=ingress/nginx-load-balancer-microk8s-conf --tcp-services-configmap=ingress/nginx-ingress-tcp-microk8s-conf --udp-services-configmap=ingress/nginx-ingress-udp-microk8s-conf --ingress-class=public   --publish-status-address=127.0.0.1 --default-ssl-certificate=default/example-tls860712 ?        Ssl    0:00      \_ /nginx-ingress-controller --configmap=ingress/nginx-load-balancer-microk8s-conf --tcp-services-configmap=ingress/nginx-ingress-tcp-microk8s-conf --udp-services-configmap=ingress/nginx-ingress-udp-microk8s-conf --ingress-class=public   --publish-status-address=127.0.0.1 --default-ssl-certificate=default/example-tls860782 ?        S      0:00          \_ nginx: master process /usr/bin/nginx -c /etc/nginx/nginx.conf860786 ?        Sl     0:00              \_ nginx: worker process860787 ?        Sl     0:00              \_ nginx: worker process860788 ?        Sl     0:00              \_ nginx: worker process860789 ?        Sl     0:00              \_ nginx: worker process860790 ?        S      0:00              \_ nginx: cache manager process

实践: 使用Microk8s Ingress配置HTTP/TCP负载均衡

首先安装并启动Microk8s和Ingress插件, 参考: 在RockyLinux9.4上安装Microk8s
演示案例如下图:
请添加图片描述

支持HTTP负载均衡

用Flask写两个简单的HTTP服务

app.py

from flask import Flaskapp = Flask(__name__)@app.route('/foo/')
def hello_world():return "Hello, foo!"if __name__ == '__main__':app.run(host='0.0.0.0', port=8080)

Dockerfile

FROM python:3.9-slim
WORKDIR /app
COPY . /app
RUN pip install flask
EXPOSE 8080
CMD ["python", "app.py"]

构建docker镜像

docker build -t foo:1.0 -f Dockerfile .
docker save foo:1.0 > foo.tar

创建k8s资源, 创建Deployment

apiVersion: apps/v1
kind: Deployment
metadata:name: foo-deployment
spec:replicas: 1selector:matchLabels:app: footemplate:metadata:labels:app: foospec:containers:- name: fooimage: foo:1.0ports:- containerPort: 8080

创建Service

apiVersion: v1
kind: Service
metadata:name: foo-service
spec:selector:app: fooports:- protocol: TCPport: 80targetPort: 8080type: ClusterIP

在Microk8s环境上导入镜像, 创建yaml, 先在集群内部测试一下服务OK

microk8s.ctr image *.tar
kubectl create -f *.yaml# kubectl get svc
default       bar-service   ClusterIP   10.152.183.208   <none>        80/TCP                   11s
default       foo-service   ClusterIP   10.152.183.189   <none>        80/TCP                   25m
# curl 10.152.183.189:80/foo/
Hello Foo!
# curl 10.152.183.208/bar/
Hello Bar!

配置Ingress, 支持集群外访问
默认情况下, Ingress Controller是通过NodePort类型暴露的, 这里我们改成监听宿主机的80端口, 通过修改Ingress的DaemonSet实现:
创建nginx-ingress-microk8s-controller-patch.yaml

apiVersion: apps/v1
kind: DaemonSet
metadata:name: nginx-ingress-microk8s-controllernamespace: ingress
spec:template:spec:hostNetwork: truecontainers:- name: nginx-ingress-microk8sargs:# must list all others here?- /nginx-ingress-controller- --configmap=$(POD_NAMESPACE)/nginx-load-balancer-microk8s-conf- --tcp-services-configmap=$(POD_NAMESPACE)/nginx-ingress-tcp-microk8s-conf- --udp-services-configmap=$(POD_NAMESPACE)/nginx-ingress-udp-microk8s-conf- --ingress-class=public- ' '- --publish-status-address=127.0.0.1

修改Ingress的DaemonSet

kubectl -n ingress patch ds nginx-ingress-microk8s-controller --patch-file nginx-ingress-microk8s-controller-patch.yaml

确认宿主机80端口已经LISTEN

ss -antp | grep ":*80"
LISTEN    0      4096                  0.0.0.0:80                   0.0.0.0:*     users:(("nginx",pid=741444,fd=17),("nginx",pid=741437,fd=17))

创建Ingress资源, 支持从集群外访问两个HTTP服务

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:name: example-ingress
spec:rules:- host: foo.example.comhttp:paths:- path: /foo/pathType: Prefixbackend:service:name: foo-serviceport:number: 80- host: bar.example.comhttp:paths:- path: /bar/pathType: Prefixbackend:service:name: bar-serviceport:number: 80

测试, 从集群外部访问成功

curl foo.example.com/foo/
Hello Foo!
curl bar.example.com/bar/
Hello Bar!

支持TCP负载均衡

Ingress同时支持四层(TCP,UDP)的负载均衡, 例如:

创建一个TCP服务, 监听8888端口 (tcp-service.yaml)

apiVersion: apps/v1
kind: Deployment
metadata:name: tcp-servernamespace: service-foo
spec:replicas: 1selector:matchLabels:app: tcp-servertemplate:metadata:labels:app: tcp-serverspec:containers:- name: tcp-containerimage: busyboxcommand: ["nc", "-lk", "8888"]ports:- containerPort: 8888
---
apiVersion: v1
kind: Service
metadata:name: tcp-servicenamespace: service-foo
spec:selector:app: tcp-serverports:- protocol: TCPport: 8888targetPort: 8888type: ClusterIP

配置Ingress的ConfigMap, 以支持TCP的暴露

创建nginx-ingress-tcp-microk8s-conf-patch.yaml

kind: ConfigMap
apiVersion: v1
metadata:name: nginx-ingress-tcp-microk8s-confnamespace: ingress
data:"8888": tcp-service:8888

修改Ingress的ConfigMap

kubectl -n ingress patch cm nginx-ingress-tcp-microk8s-conf --patch-file nginx-ingress-tcp-microk8s-conf-patch.yaml

从集群外部测试, 访问成功

# telnet foo.example.com 8888
Connected to foo.example.com.
Escape character is '^]'

Ingress 支持HTTPS

生成证书和私钥

openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=example.com/O=Example Company"

创建kubernetes secret

microk8s kubectl create secret tls example-tls --cert=tls.crt --key=tls.key -n default

改一下Ingress的DaemonSet, 添加一行--default-ssl-certificate

apiVersion: apps/v1
kind: DaemonSet
metadata:name: nginx-ingress-microk8s-controllernamespace: ingress
spec:template:spec:hostNetwork: truecontainers:- name: nginx-ingress-microk8sargs:# must list all others here?- /nginx-ingress-controller- --configmap=$(POD_NAMESPACE)/nginx-load-balancer-microk8s-conf- --tcp-services-configmap=$(POD_NAMESPACE)/nginx-ingress-tcp-microk8s-conf- --udp-services-configmap=$(POD_NAMESPACE)/nginx-ingress-udp-microk8s-conf- --ingress-class=public- ' '- --publish-status-address=127.0.0.1- --default-ssl-certificate=default/example-tls

kubectl -n ingress patch ds nginx-ingress-microk8s-controller --patch-file nginx-ingress-microk8s-controller-patch.yaml

从集群外部测试, HTTP和HTTPS都可以访问

# curl foo.example.com/foo/
Hello Foo!
# curl https://bar.example.com/bar/ -k
Hello Bar!

问题

Q: 可能遇到导入Microk8s镜像失败, microk8s.ctr image list查询镜像类型为text/html, 只有几百k
A: 原因一般是之前导入镜像前就创建了pod, Microk8s尝试从网络获取镜像失败; 解决方法是先microk8s.ctr image rm删除旧的镜像, 再重新导入即可