Kubernetes 安装配置ingress controller

> 对于Kubernetes的Service，无论是Cluster-Ip和NodePort均是四层的负载，集群内的服务如何实现七层的负载均衡，这就需要借助于Ingress，Ingress控制器的实现方式有很多，比如nginx, Contour, Haproxy, trafik, Istio。

几种常用的ingress功能对比和选型：https://www.kubernetes.org.cn/5948.html

实现逻辑

Ingress-nginx是7层的负载均衡器，负责统一管理外部对k8s cluster中Service的请求。主要包含：

ingress-nginx-controller：根据用户编写的ingress规则（创建的ingress的yaml文件），动态的去更改nginx服务的配置文件，并且reload重载使其生效（是自动化的，通过lua脚本来实现）；
Ingress资源对象：将Nginx的配置抽象成一个Ingress对象

ingress controller通过和kubernetes api交互，动态的去感知集群中ingress规则变化
然后读取ingress规则(规则就是写明了哪个域名对应哪个service)，按照自定义的规则，生成一段nginx配置
再写到nginx-ingress-controller的pod里，这个Ingress controller的pod里运行着一个Nginx服务，控制器把生成的nginx配置写入/etc/nginx/nginx.conf文件中
然后reload一下使配置生效。以此达到域名分别配置和动态更新的问题。

安装

官方文档：https://kubernetes.github.io/ingress-nginx/deploy/
部署文档：https://github.com/kubernetes/ingress-nginx/blob/nginx-0.30.0/docs/deploy/index.md
K8S Annotations文档： https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#enable-cors

Support Versions

**官网地址：**https://github.com/kubernetes/ingress-nginx

Ingress-NGINX version	k8s supported version	Alpine Version	Nginx Version
v1.2.1	1.23, 1.22, 1.21, 1.20, 1.19	3.14.6	1.19.10†
v1.2.0	1.23, 1.22, 1.21, 1.20, 1.19	3.14.6	1.19.10†
v1.1.3	1.23, 1.22, 1.21, 1.20, 1.19	3.14.4	1.19.10†
v1.1.2	1.23, 1.22, 1.21, 1.20, 1.19	3.14.2	1.19.9†
v1.1.1	1.23, 1.22, 1.21, 1.20, 1.19	3.14.2	1.19.9†
v1.1.0	1.22, 1.21, 1.20, 1.19	3.14.2	1.19.9†
v1.0.5	1.22, 1.21, 1.20, 1.19	3.14.2	1.19.9†
v1.0.4	1.22, 1.21, 1.20, 1.19	3.14.2	1.19.9†
v1.0.3	1.22, 1.21, 1.20, 1.19	3.14.2	1.19.9†
v1.0.2	1.22, 1.21, 1.20, 1.19	3.14.2	1.19.9†
v1.0.1	1.22, 1.21, 1.20, 1.19	3.14.2	1.19.9†
v1.0.0	1.22, 1.21, 1.20, 1.19	3.13.5	1.20.1
v0.51.0	1.21, 1.20, 1.19	3.14.4	1.19.10†
v0.49.3	1.21, 1.20, 1.19	3.14.2	1.19.9†
v0.49.2	1.21, 1.20, 1.19	3.14.2	1.19.9†
v0.49.1	1.21, 1.20, 1.19	3.14.2	1.19.9†
v0.49.0	1.21, 1.20, 1.19	3.13.5	1.20.1
v0.48.1	1.21, 1.20, 1.19	3.13.5	1.20.1

Helm 部署（推荐）

部署文档：https://github.com/bitnami/charts/tree/master/bitnami/nginx-ingress-controller/#installing-the-chart

1. 安装helm

#Linux 直接运行curl 命令安装
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

2. 下载Chart

#下载Chart
git clone https://github.com/bitnami/charts.git
cd charts/bitnami/nginx-ingress-controller/

3. 配置Chart

#为安装ingress的node节点添加label
#高可用模式可以在多个node上打标签，host模式不会调度在同一个机器
kubectl label node k8s-master ingress=true
kubectl label node k8s-node01 ingress=true#查询包含指定label（ingress=true）的node节点
kubectl get nodes --show-labels -l "ingress=true"

##配置Chart
$ vim values.yaml81 ##NGINX 的自定义configmap配置选项
82 ##官方文档: https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/configmap/
83 ##添加下面三行，获取客户端真实IP
84 #config: {}
85 config:
86   compute-full-forwarded-for: "true"
87   forwarded-for-header: "X-Forwarded-For"
88   use-forwarded-headers: "true"##如果使用hostNetwork=true，设置reportNodeInternalIp=true，会将标志“report-node-internal-ip-address”传递给Nginx 入口控制器
104 reportNodeInternalIp: true##配置控制器所需的 pod 数量
222 replicaCount: 2##添加为host模式，启用主机网络，通过宿主机ip+port访问。
358 hostNetwork: true##选择器，决定将ingress部署在哪些机器
402 #nodeSelector: {}
403 nodeSelector:
404   ingress: "true"##禁用基于 NGINX 的默认后端
464 defaultBackend:
465   ## @param defaultBackend.enabled Enable a default backend based on NGINX
466   ##
467   enabled: false      ##设置为false757 service:
758   ## @param service.type Kubernetes Service type for Controller
759   ##
760   #type: LoadBalancer
761   type: NodePort   ## 修改服务类型为 NodePort

4. 安装Chart

#添加chart仓库
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update
helm search repo bitnami#安装Chart
helm upgrade --install -f values.yaml --timeout 20m --create-namespace --namespace nginx-ingress-controller k8s-nginx bitnami/nginx-ingress-controller --wait --wait-for-jobs --debug

查看发布版本列表

helm list -n nginx-ingress-controller

升级Chart

##因服务类型为 NodePort，需要--set replicaCount=1 将pod数量改为1，否则端口被占用
helm upgrade -f values.yaml --timeout 30m --namespace nginx-ingress-controller k8s-nginx bitnami/nginx-ingress-controller --atomic --wait --wait-for-jobs --debug --set image.tag='1.3.1-debian-11-r0' --set ingressClassResource.name="nginx-new" --set replicaCount=1##查看deployment信息
[ec2-user@eks ~]$ kubectl get deploy -n nginx-ingress-controller
NAME                                 READY   UP-TO-DATE   AVAILABLE   AGE
k8s-nginx-nginx-ingress-controller   1/1     1            1           39h##手动对pod进行扩容
kubectl scale --replicas 2 deployment k8s-nginx-nginx-ingress-controller -n nginx-ingress-controller

卸载Chart

helm uninstall k8s-nginx -n nginx-ingress-controller
#helm delete k8s-nginx -n nginx-ingress-controllerhelm list -n nginx-ingress-controller

自定义yaml文件部署

安装ingress-controller

hostNetwork: true 修改为host模式，使用node节点ip访问

1.下载yaml文件

wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.30.0/deploy/static/mandatory.yaml -O ./ingress-nginx.yaml#wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.2.1/deploy/static/provider/cloud/deploy.yaml -O ./ingress-nginx.yaml

或者使用myblog/deployment/ingress/mandatory.yaml

2.修改yaml配置文件

$ grep -n5 nodeSelector ingress-nginx.yaml
199-    replicas: 2   #设置副本数，host模式不会被调度到同一个node
212-    spec:
213-      hostNetwork: true #添加为host模式
214-      # wait up to five minutes for the drain of connections
215-      terminationGracePeriodSeconds: 300
216-      serviceAccountName: nginx-ingress-serviceaccount
217:      nodeSelector:
218-        ingress: "true"        #替换此处，来决定将ingress部署在哪些机器
219-      containers:
220-        - name: nginx-ingress-controller
221-          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.30.0
222-          args:

4.为安装ingress的node节点添加label

#为node节点添加label
#高可用模式可以在多个node上打标签，host模式不会调度在同一个机器
kubectl label node k8s-master ingress=true
kubectl label node k8s-node01 ingress=true#查询包含指定label（ingress=true）的node节点
kubectl get nodes --show-labels -l "ingress=true"

5.创建ingress-controller

kubectl apply -f ingress-nginx.yaml

6.检查运行状态

$ kubectl get pods -n ingress-nginx
NAME                                        READY   STATUS    RESTARTS   AGE
nginx-ingress-controller-55dd6f8d7b-tkx77   1/1     Running   0          83d
nginx-ingress-controller-55dd6f8d7b-vkvqp   1/1     Running   0          83d

AWS 中部署

官方文档：https://docs.amazonaws.cn/eks/latest/userguide/alb-ingress.html
在 AWS 中，我们使用网络负载均衡器 (NLB) 在服务类型为LoadBalancer 之后暴露 NGINX 入口控制器。

AWS 负载均衡器 (NLB) 中的 TLS 终止

默认情况下，TLS 在入口控制器中终止。但也可以在负载均衡器中终止 TLS。本节介绍如何使用 NLB 在 AWS 上执行此操作。

1.下载 deploy.yaml模板

wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.2.1/deploy/static/provider/aws/nlb-with-tls-termination/deploy.yaml

2.编辑文件并更改 Kubernetes 集群使用的 VPC CIDR

proxy-real-ip-cidr: XXX.XXX.XXX/XX

3.同时更改 AWS Certificate Manager (ACM) ID

service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:us-west-2:XXXXXXXX:certificate/XXXXXX-XXXXXXX-XXXXXXX-XXXXXXXX

4.部署清单

kubectl apply -f deploy.yaml

Ingress controller配置

透传client源IP

配置X-Forwarded-For 获取客户端真实IP，实现IP白名单和透传client源IP的功能

##查看configmap 信息
[ec2-user@eks ~]$ kubectl get configmap -n nginx-ingress-controller
NAME                                 DATA   AGE
k8s-nginx-nginx-ingress-controller   3      19m##编辑configmap 
[ec2-user@eks ~]$ kubectl edit cm/k8s-nginx-nginx-ingress-controller -n nginx-ingress-controller

修改内容如下：

apiVersion: v1
data:#data里，加入以下配置#将远程地址附加到 X-Forwarded-For 标头而不是替换它。启用此选项后，上游应用程序负责根据自己的受信任代理列表提取客户端 IP。compute-full-forwarded-for: "true"#设置用于标识客户端的原始 IP 地址的标头字段。默认值：X-Forwarded-Forforwarded-for-header: X-Forwarded-For#如果为真，NGINX 会将传入的 X-Forwarded-* 标头传递给上游。当 NGINX 在设置这些标头的另一个代理/负载均衡器之后使用此选项；如果为 false，NGINX 会忽略传入的 X-Forwarded-* 标头，并用它看到的请求信息填充它们。use-forwarded-headers: "true"
kind: ConfigMap

验证是否生效：

[ec2-user@eks ~]$ kubectl exec pods/k8s-nginx-nginx-ingress-controller-69c99c6c7-hcbd2 -n nginx-ingress-controller /bin/cat  /etc/nginx/nginx.conf |grep -b5 'real_ip_header'
... ... ...
##生效后，在 nginx-ingress-controller 中 nginx.conf 增加了以下配置：
http {   
#... ... real_ip_header      X-Forwarded-For;real_ip_recursive   on;                   set_real_ip_from    0.0.0.0/0;    
#... ...                                               
}

Ingress配置

Ingress yml配置

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:name: ingress-wildcard-hostnamespace: opsannotations:kubernetes.io/ingress.class: nginxnginx.ingress.kubernetes.io/ssl-redirect: 'false'
spec:rules:- host: "foo.bar.com"http:paths:- pathType: Prefixpath: "/bar"backend:service:name: service1port:number: 80- host: "bar.foo.com"http:paths:- pathType: Prefixpath: "/foo"backend:service:name: service2port:number: 80

检查ingress动态生成的nginx配置

$ kubectl -n ingress-nginx exec -ti nginx-ingress-xxxxxxx bash
# ps aux
# cat /etc/nginx/nginx.conf|grep myblog -A10 -B1

跨域配置

跨域配置1：（直接在ingress 的annation里面加）

  annotations:kubernetes.io/ingress.class: nginxnginx.ingress.kubernetes.io/Access-Control-Allow-Origin: '*'nginx.ingress.kubernetes.io/cors-allow-credentials: 'true'nginx.ingress.kubernetes.io/cors-allow-headers: >-DNT,web-token,app-token,Authorization,Accept,Origin,Keep-Alive,User-Agent,X-Mx-ReqToken,X-Data-Type,X-Auth-Token,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,token,Cookienginx.ingress.kubernetes.io/cors-allow-methods: 'PUT, GET, POST, OPTIONS'nginx.ingress.kubernetes.io/cors-allow-origin: 'https://*.qqq.cn, http://*.qqq.cn, http://localhost:8081'nginx.ingress.kubernetes.io/enable-cors: 'true'nginx.ingress.kubernetes.io/cors-max-age: '600'

跨域配置2：（使用configuration-snippet配置）

    nginx.ingress.kubernetes.io/configuration-snippet: |if ($request_method = 'OPTIONS') { more_set_headers "Access-Control-Allow-Origin: $http_origin"; more_set_headers 'Access-Control-Allow-Credentials: true'; more_set_headers 'Access-Control-Allow-Methods: GET, POST, OPTIONS';more_set_headers 'Access-Control-Allow-Headers: DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type';more_set_headers 'Access-Control-Max-Age: 1200';more_set_headers 'Content-Type: text/plain charset=UTF-8';more_set_headers 'Content-Length: 0'; return 204; }more_set_headers "Access-Control-Allow-Origin: $http_origin"; more_set_headers 'Access-Control-Allow-Credentials' 'true';more_set_headers 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';more_set_headers 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type';

白名单限制

nginx.ingress.kubernetes.io/whitelist-source-range 可以通过注解指定允许的客户端 IP 源范围。该值是一个逗号分隔的CIDR列表，例如10.0.0.0/24,172.10.0.1。此设置为所有 Ingress 规则进行全局配置。

前端没有负载均衡器的情况下

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:annotations:nginx.ingress.kubernetes.io/whitelist-source-range: '172.21.0.0/16,10.0.0.0/16'

前端有负载均衡器的情况下（阿里云 SLB，AWS ALB）

Ingress controller配置透传client源IP（需要在ConfigMap配置中额外增加下面配置，在data 下添加 use-forwarded-headers）

kind: ConfigMap
apiVersion: v1
metadata:name: nginx-configurationnamespace: ingress-nginxlabels:app.kubernetes.io/name: ingress-nginxapp.kubernetes.io/part-of: ingress-nginx
data:allow-backend-server-header: 'false'allow-snippet-annotations: 'true'enable-underscores-in-headers: 'false'generate-request-id: 'true'ignore-invalid-headers: 'true'keep-alive-requests: '100'max-worker-connections: '65536'proxy-body-size: '20m'proxy-connect-timeout: '10'ssl-redirect: 'false'upstream-keepalive-connections: '320'upstream-keepalive-timeout: '900'forwarded-for-header: 'X-Forwarded-For'compute-full-forwarded-for: 'true'use-forwarded-headers: 'true'   # 添加此行

针对某个接口进行IP白名单访问限制

  annotations:nginx.ingress.kubernetes.io/server-snippet: |set $test '';if ($request_uri ~*  (/doc.html|/swagger)){  set $test 1;}if ( $remote_addr !~* 110.110.110.0 ) {set $test "${test}2";}if ( $test = 12 ) {return 403;}

针对IP网段进行白名单访问限制

  annotations:nginx.ingress.kubernetes.io/server-snippet: |geo $allowed_ip {default 0;110.110.110.0/24 1;}if ($request_uri ~* (/doc.html|/swagger)) {set $flag "1";}if ($allowed_ip = 0) {set $flag "${flag}2";}if ($flag = "12") {return 403;}

重定向配置

官方地址：https://kubernetes.github.io/ingress-nginx/examples/rewrite/

域名跳转

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:annotations:kubernetes.io/ingress.class: nginxnginx.ingress.kubernetes.io/rewrite-target: 'https://ops-bbb.qsh.cn/$1'generation: 2name: ops-webnamespace: ops
spec:rules:- host: ops-aaa.qsh.cnhttp:paths:- backend:service:name: ops-webport:number: 8000path: /(.*)pathType: Prefix
status:loadBalancer: {}

路径跳转

$ echo '
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:annotations:nginx.ingress.kubernetes.io/rewrite-target: /$2name: rewritenamespace: default
spec:ingressClassName: nginxrules:- host: rewrite.bar.comhttp:paths:- path: /something(/|$)(.*)pathType: Prefixbackend:service:name: http-svcport: number: 80
' | kubectl create -f -

报错小结

posthog-ingress-nginx-controller-admission not found

Error from server (InternalError): error when creating “ingress-nginx-posthog.yaml”: Internal error occurred: failed calling webhook “validate.nginx.ingress.kubernetes.io”: Post “https://posthog-ingress-nginx-controller-admission.posthog.svc:443/networking/v1/ingresses?timeout=10s”: service “posthog-ingress-nginx-controller-admission” not found

原因分析：
类似问题：https://stackoverflow.com/questions/61616203/nginx-ingress-controller-failed-calling-webhook/61681896#61681896

我删除了它创建的命名空间和clusterrole and clusterrolebinding as noted in the documentation，但这并没有删除安装在清单中的ValidatingWebhookConfiguration，但在默认情况下使用helm 时不会删除。

解决方法1：

kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admission

解决方法2：
1、检查是否有validation webhook 和服务。

[root@ops ~]# kubectl get -A ValidatingWebhookConfiguration
NAME                              WEBHOOKS   AGE
aws-load-balancer-webhook         2          90d
cert-manager-webhook              1          91d
posthog-ingress-nginx-admission   1          5d1h
vpc-resource-validating-webhook   2          107d[root@ops ~]# kubectl get svc -n ingress-nginx
No resources found in ingress-nginx namespace.

2、手动删除posthog-ingress-nginx-admission后恢复正常

[root@ops ~]# kubectl get ValidatingWebhookConfiguration/posthog-ingress-nginx-admission -o yaml >posthog-ingress-nginx-admission.yaml[root@ops ~]# kubectl delete -f posthog-ingress-nginx-admission.yaml

no matches for kind “Role” in version “rbac.authorization.k8s.io/v1beta1”

执行异常信息：
unable to recognize “ingress-nginx.yaml”: no matches for kind “ClusterRole” in version “rbac.authorization.k8s.io/v1beta1”
unable to recognize “ingress-nginx.yaml”: no matches for kind “Role” in version “rbac.authorization.k8s.io/v1beta1”
unable to recognize “ingress-nginx.yaml”: no matches for kind “RoleBinding” in version “rbac.authorization.k8s.io/v1beta1”
unable to recognize “ingress-nginx.yaml”: no matches for kind “ClusterRoleBinding” in version “rbac.authorization.k8s.io/v1beta1”

原因分析：
自 v1.22 起，Ingress 的 extensions/v1beta1 和networking.k8s.io/v1beta1 API 版本不再提供。
官方文档：https://kubernetes.io/docs/reference/using-api/deprecation-guide/#ingress-v122

解决方法：

sed -i 's#rbac.authorization.k8s.io/v1beta1#rbac.authorization.k8s.io/v1#' ingress-nginx.yaml

Unable to continue with install: IngressClass “nginx” in namespace “” exists and cannot be imported into the current release

Error: INSTALLATION FAILED: rendered manifests contain a resource that already exists. Unable to continue with install: IngressClass “nginx” in namespace “” exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: missing key “meta.helm.sh/release-name”: must be set to “k8s-nginx”; annotation validation error: missing key “meta.helm.sh/release-namespace”: must be set to “nginx-ingress-controller”
helm.go:84: [debug] IngressClass “nginx” in namespace “” exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: missing key “meta.helm.sh/release-name”: must be set to “k8s-nginx”; annotation validation error: missing key “meta.helm.sh/release-namespace”: must be set to “nginx-ingress-controller”
rendered manifests contain a resource that already exists. Unable to continue with install

原因分析：
使用 helm 创建nginx-ingress-controller时出错

查看helm chart仓库values.yaml文件

#... ... ...
##查看以下字段 
ingressClassResource:name: nginx  ##修改资源名enabled: truedefault: falsecontrollerClass: "k8s.io/ingress-nginx"parameters: {}
#... ... ...

解决方法：

##创建
helm install k8s-nginx bitnami/nginx-ingress-controller -n nginx-ingress-controller --create-namespace --set ingressClassResource.name="nginx-new"##升级
cd charts/bitnami/nginx-ingress-controller/
helm upgrade --install -f values.yaml --timeout 20m --create-namespace --namespace nginx-ingress-controller k8s-nginx bitnami/nginx-ingress-controller --wait --wait-for-jobs --set ingressClassResource.name="nginx-new" --debug

如果没生效，使用以下命令：

helm install k8s-nginx bitnami/nginx-ingress-controller -n nginx-ingress-controller --create-namespace --set controller.ingressClassResource.name="nginx-new"