Ubuntu24搭建k8s高可用集群

环境信息

主机名	IP	k8s版本	备注
vm-master	192.168.103.250	1.28.2	master1
vm-master2	192.168.103.249	1.28.2	master2
vm-master3	192.168.103.254	1.28.2	master3
vm-node1	192.168.103.251	1.28.2	node1
vm-node2	192.168.103.252	1.28.2	node2

容器进行时：container
网络插件为：flannel

配置中有不清楚的地方可以查看官方文档 https://kubernetes.io/zh-cn/docs/home/

1. 基础配置（所有节点）

配置/etc/hosts文件

192.168.103.253 vip.cluster.local
192.168.103.250 vm-master
192.168.103.249 vm-master2
192.168.103.251 vm-node1
192.168.103.252 vm-node2
192.168.103.254 vm-master3

配置时间同步

sudo apt install chrony -y

sudo vim /etc/chrony/chrony.conf# master节点配置
confdir /etc/chrony/conf.d
pool ntp.aliyun.com iburst maxsources 4
pool time1.cloud.tencent.com iburst maxsources 2
refclock PHC /dev/ptp0 poll 0 dpoll -2 offset 0
local stratum 10
allow 192.168.103.0/24
sourcedir /run/chrony-dhcp
sourcedir /etc/chrony/sources.d
keyfile /etc/chrony/chrony.keys
driftfile /var/lib/chrony/chrony.drift
ntsdumpdir /var/lib/chrony
logdir /var/log/chrony
maxupdateskew 100.0
makestep 1 3
maxdistance 16.0
leapsecmode slew
rtcsync
leapsectz right/UTC

sudo vim /etc/chrony/chrony.conf# node节点配置
confdir /etc/chrony/conf.d
server vm-master iburst
server vm-master2 iburst
sourcedir /run/chrony-dhcp
sourcedir /etc/chrony/sources.d
keyfile /etc/chrony/chrony.keys
driftfile /var/lib/chrony/chrony.drift
ntsdumpdir /var/lib/chrony
logdir /var/log/chrony
maxupdateskew 100.0
makestep 1 3
maxdistance 16.0
leapsecmode slew
rtcsync
leapsectz right/UTC

sudo systemctl restart chrony

更新系统并关闭交换分区

sudo apt-get update
sudo apt-get upgrade -y
sudo swapoff -a
sudo vim /etc/fstab
#注释掉swap所在行
#/swap
#重启查看是否成功，swap显示0就是成功sudo reboot
free -h
#Swap:             0B          0B          0B

修改内核参数

sudo tee /etc/modules-load.d/containerd.conf <<EOF
overlay
br_netfilter
EOFsudo modprobe overlay
sudo modprobe br_netfilter

配置网络参数

# 默认情况下，Linux 内核不允许 IPv4 数据包在接口之间路由。
# 手动启用 IPv4 数据包转发
sudo tee /etc/sysctl.d/kubernetes.conf <<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
# 应用 sysctl 参数而不重新启动
sudo sysctl --system

生产环境需要配置防火墙放行端口

IPVS负载均衡

# 安装
sudo apt install -y ipset ipvsadm# 内核加载ipvs
cat <<EOF | sudo tee /etc/modules-load.d/ipvs.conf
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack
EOF# 加载模块
sudo modprobe ip_vs
sudo modprobe ip_vs_rr
sudo modprobe ip_vs_wrr
sudo modprobe ip_vs_sh
sudo modprobe nf_conntrack

2. 安装组件（所有节点）

容器运行时containerd

sudo apt install -y containerd# 配置containerd文件
sudo mkdir -p /etc/containerd/containerd config default | sudo tee /etc/containerd/config.toml >/dev/null 2>&1sudo sed -i "s#SystemdCgroup\ \=\ false#SystemdCgroup\ \=\ true#g" /etc/containerd/config.toml
sudo cat /etc/containerd/config.toml | grep SystemdCgroup
# 修改沙箱镜像源
sudo sed -i "s#registry.k8s.io/pause#registry.cn-hangzhou.aliyuncs.com/google_containers/pause#g" /etc/containerd/config.toml
sudo cat /etc/containerd/config.toml | grep sandbox_image

安装kubeadm，kubelet，kubectl

# 安装依赖
sudo apt-get update && sudo apt-get install -y apt-transport-https ca-certificates curl gpg# 添加kubernetes的key
curl -fsSL https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg# 添加kubernetes apt仓库，使用阿里云镜像源
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list# 更新apt索引
sudo apt update# 查看版本列表
apt-cache madison kubeadm# 不带版本默认会安装最新版本，本文安装的版本为1.28.2
sudo apt-get install -y kubelet kubeadm kubectl# 锁定版本，不随 apt upgrade 更新
sudo apt-mark hold kubelet kubeadm kubectl# kubectl命令补全
sudo apt install -y bash-completionkubectl completion bash | sudo tee /etc/profile.d/kubectl_completion.sh > /dev/null. /etc/profile.d/kubectl_completion.sh

说明：

kubelet 现在每隔几秒就会重启，因为它陷入了一个等待 kubeadm 指令的死循环。

3. 高可用方案部署keepalived、haproxy（所有master节点）

所有master节点安装配置haproxy

sudo apt install keepalived haproxy -y

sudo vim /etc/haproxy/haproxy.cfg

### 在文件末尾添加
frontend apiserverbind *:16443mode tcpoption tcplogdefault_backend apiserverbackendbackend apiserverbackendoption httpchkhttp-check connect sslhttp-check send meth GET uri /healthzhttp-check expect status 200mode tcpbalance     roundrobinserver vm-master 192.168.103.250:6443 check verify none	#有几个master节点就添加几个server vm-master2 192.168.103.249:6443 check verify none
###

#重启haproxy服务，监听16443端口
sudo systemctl restart haproxy

所有master节点安装配置keepalived

cd /etc/keepalived
sudo cp keepalived.conf.sample keepalived.conf

sudo vim keepalived.conf
###全部替换成以下内容，主备节点略有不同，需要修改注释的地方! Configuration File for keepalivedglobal_defs {router_id LVS_DEVEL
}vrrp_script check_apiserver {script "/etc/keepalived/check_apiserver.sh"interval 3weight -2fall 10rise 2
}vrrp_instance VI_1 {state MASTER	# 主master节点为MASTER，备master节点为BACKUPinterface eth0virtual_router_id 51priority 150	#优先级，主master比备master多50。例如主150，备100advert_int 1authentication {auth_type PASSauth_pass 1111}virtual_ipaddress {192.168.103.253	#虚拟IP，指定的vip，所有节点一致，可添加多个vip}track_script {check_apiserver}
}

#添加检查脚本

sudo vim /etc/keepalived/check_apiserver.sh

#!/bin/sh
errorExit() {echo "*** $*" 1>&2exit 1
}
curl -sfk --max-time 2 https://localhost:16443/healthz -o /dev/null || errorExit "Error GET https://localhost:16443/healthz"

#脚本添加可执行权限
sudo chmod +x /etc/keepalived/check_apiserver.sh

#重启keepalived
sudo systemctl restart keepalived

所有master节点拉取镜像

kubeadm config images listkubeadm config images list --image-repository registry.cn-hangzhou.aliyuncs.com/google_containerskubeadm config images pull --kubernetes-version=v1.28.2 --image-repository registry.aliyuncs.com/google_containers

4. 初始化k8s集群

仅主master节点生成配置文件模板

kubeadm config print init-defaults > init.default.yaml

###内容如下，根据情况修改注释的地方
apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:- system:bootstrappers:kubeadm:default-node-tokentoken: abcdef.0123456789abcdefttl: 24h0m0susages:- signing- authentication
kind: InitConfiguration
localAPIEndpoint:advertiseAddress: 192.168.103.250	#主master节点IPbindPort: 6443
nodeRegistration:criSocket: unix:///var/run/containerd/containerd.sockimagePullPolicy: IfNotPresentname: vm-master	#主master名称taints: null
---
apiServer:timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
controlPlaneEndpoint: vip.cluster.local:16443	#添加高可用配置
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
etcd:local:dataDir: /var/lib/etcd
imageRepository: registry.aliyuncs.com/google_containers	#配置镜像库
kind: ClusterConfiguration
kubernetesVersion: 1.28.2
networking:dnsDomain: cluster.localpodSubnet: 10.200.0.0/16	#pod子网serviceSubnet: 10.96.0.0/12
scheduler: {}

仅主master初始化节点

sudo kubeadm init --config init.default.yaml --upload-certs

###完成后输出结果如下，输出内容中有后续操作说明
Your Kubernetes control-plane has initialized successfully!To start using your cluster, you need to run the following as a regular user:# 创建配置文件操作mkdir -p $HOME/.kubesudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/configsudo chown $(id -u):$(id -g) $HOME/.kube/configAlternatively, if you are the root user, you can run:export KUBECONFIG=/etc/kubernetes/admin.confYou should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:https://kubernetes.io/docs/concepts/cluster-administration/addons/You can now join any number of the control-plane node running the following command on each as root:# 添加其他master节点操作kubeadm join vip.cluster.local:16443 --token abcdef.0123456789abcdef \--discovery-token-ca-cert-hash sha256:a3dbd9108cf5a5b14267013f8e7fa4f6698a4bcc9396de86cfb265c74bea30e4 \--control-plane --certificate-key f01a6c541e26097e4c90557256e376352a71490f149db45516ac542c58eef883Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.Then you can join any number of worker nodes by running the following on each as root:
# 添加worker节点操作
kubeadm join vip.cluster.local:16443 --token abcdef.0123456789abcdef \--discovery-token-ca-cert-hash sha256:a3dbd9108cf5a5b14267013f8e7fa4f6698a4bcc9396de86cfb265c74bea30e4 

#配置kubeconfig文件
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id-u):$(id -g) $HOME/.kube/config

#如果初始化出错其他问题，可以清除初始化操作，重新初始化
sudo kubeadm reset -f

仅主master节点配置网络组件flannel

#先下载配置文件，若需要代理则加上代理参数，我使用的是clash代理
wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml -e "https_proxy=http://127.0.0.1:7890" 

vim kube-flannel.yml #更改子网网段和init.default.yaml中配置的podSubnet一致

### 若container拉取容器时需要配置代理则配置以下操作，不需要代理则跳过下列操作
# 在containerd层面设置代理（所有节点执行）
sudo mkdir -p /etc/systemd/system/containerd.service.d
sudo tee /etc/systemd/system/containerd.service.d/http-proxy.conf <<EOF
[Service]
Environment="HTTP_PROXY=http://127.0.0.1:7890"
Environment="HTTPS_PROXY=http://127.0.0.1:7890"
EOF# 重启服务生效
sudo systemctl daemon-reload
sudo systemctl restart containerd kubelet

#应用并创建flannel，需等待kube自行配置
kubectl apply -f kube-flannel.yml

#查看状态为running则配置成功
例如：
root@vm-master:~# kubectl get pods -n kube-flannel
NAME                    READY   STATUS    RESTARTS   AGE
kube-flannel-ds-4ct2x   1/1     Running   0          81m

备master节点配置加入集群，所有备用master节点都执行

#测试能否ping通虚拟IP
ping vip.cluster.local#加入节点命令（初始化后的信息中有命令）：
kubeadm join vip.cluster.local:16443 --token abcdef.0123456789abcdef \--discovery-token-ca-cert-hash sha256:a3dbd9108cf5a5b14267013f8e7fa4f6698a4bcc9396de86cfb265c74bea30e4 \--control-plane --certificate-key f01a6c541e26097e4c90557256e376352a71490f149db45516ac542c58eef883

成功加入后显示信息

This node has joined the cluster and a new control plane instance was created:

• Certificate signing request was sent to apiserver and approval was received.
• The Kubelet was informed of the new secure connection details.
• Control plane label and taint were applied to the new node.
• The Kubernetes control plane instances scaled up.
• A new etcd member was added to the local/stacked etcd cluster.

To start administering your cluster from this node, you need to run the following as a regular user:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id-u):$(id -g) $HOME/.kube/config

Run ‘kubectl get nodes’ to see this node join the cluster.

执行配置文件操作

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

耐心等待从主节点同步配置

worker节点加入

#执行命令
kubeadm join vip.cluster.local:16443 --token abcdef.0123456789abcdef \--discovery-token-ca-cert-hash sha256:a3dbd9108cf5a5b14267013f8e7fa4f6698a4bcc9396de86cfb265c74bea30e4

查看状态

kubectl get node

5. 添加新的master节点或node节点

如果前期操作时就部署了3个master节点则不用配置此操作。

由于k8s高可用集群最少需要3个master节点，所以如果只配置了两个master节点就需要再添加一个master节点。

新server配置，和之前的配置一样

1. 基础配置（所有节点）
2. 安装组件（所有节点）
3. 高可用方案部署keepalived、haproxy（所有master节点，如果是添加node节点则不用配置）

获取加入节点的证书和token

主master节点操作

# 获取证书
kubeadm init phase upload-certs --upload-certs

# 生成加入集群的token
kubeadm token create --print-join-command

需要新添加的节点上执行

拼接获取的命令，添加master节点需要在token和证书之间用–control-plane --certificate-key连接。

如果只使用token则会添加为node节点

kubeadm join vip.cluster.local:16443 --token x12ydm.5w9y2439fmrfbll3 --discovery-token-ca-cert-hash sha256:a3dbd9108cf5a5b14267013f8e7fa4f6698a4bcc9396de86cfb265c74bea30e4 --control-plane --certificate-key 6ac8c3d9baead5aac03895f2e9ba3f37de4067f2f34b8524f9cc5de0dc063c09

加入集群成功后输出结果如下

添加节点超时故障处理

如果新节点卡住，如图

且主master查看如下，

kubectl get node
kubectl get pods -n kube-flannel

则需要在新节点上检查错误日志

journalctl -u kubelet --since "10 minutes ago" | grep -i error

如果日志显示连接无法访问 registry.k8s.io 的443端口，则需要配置国内仓库或者使用代理，我使用配置代理的方法解决问题

# 在新节点上运行clash代理，
sudo mkdir -p /etc/systemd/system/containerd.service.d
sudo tee /etc/systemd/system/containerd.service.d/http-proxy.conf <<EOF
[Service]
Environment="HTTP_PROXY=http://127.0.0.1:7890"
Environment="HTTPS_PROXY=http://127.0.0.1:7890"
EOF# 重启服务生效
sudo systemctl daemon-reload
sudo systemctl restart containerd kubelet

配置代理后再运行加入节点的命令即可，输出如图则配置完成

至此高可用集群部署完成，测试单点故障后集群仍正常工作