【Kubernetes部署篇】二进制搭建K8s高可用集群1.26.15版本(超详细,可跟做)

文章目录

    • 一、服务器环境信息及部署规划
      • 1、K8S服务器信息及网段规划
      • 2、服务器部署架构规划
      • 3、组件版本信息
      • 4、实验架构图
    • 二、初始化环境操作
      • 1、关闭防火墙
      • 2、配置本地域名解析
      • 3、配置服务器时间保持一致
      • 4、禁用swap交换分区(K8S强制要求禁用)
      • 5、配置主机之间无密码登录
      • 6、修改Linux内核参数,添加网桥过滤器和地址转发功能
      • 7、启用ipvs功能
      • 8、安装Docker容器组件
      • 9、安装cri-dockerd插件
      • 10、安装基础软件包
    • 三、部署etcd高可用集群
      • 1、签发etcd证书
      • 2、搭建etcd高可用集群
      • 3、测试连接集群
    • 四、部署Nginx+Keepalived
      • 1、部署并配置Nginx服务
      • 2、部署并配置Keepalived服务
      • 3、测试VIP故障自动切换
    • 五、部署Kubernetes控制节点相关组件
      • 1、下载二进制包和环境准备相关操作
      • 2、部署api-service组件
      • 3、部署kubectl组件
      • 4、部署controller-manager组件
      • 5、部署kube-scheduler组件
      • 6、验证控制节点相关组件
    • 六、部署Kubernetes工作节点相关组件
      • 1、部署kubelet组件
      • 2、部署kube-proxy组件
      • 3、部署calico网络组件
      • 4、部署coreDNS解析组件
    • 七、K8S集群功能测试
      • 1、测试创建Pod、SVC资源
      • 2、测试coreDNS解析

在这里插入图片描述

一、服务器环境信息及部署规划

1、K8S服务器信息及网段规划

K8S网段规划:

  • Pod网段:10.0.0.0/16
  • Service网段:10.255.0.0/16

服务器资源配置信息:

IP地址操作系统资源备注
16.32.15.115CentOS Linux release 7.94C/4G/50G
16.32.15.200CentOS Linux release 7.94C/4G/50G
16.32.15.201CentOS Linux release 7.94C/4G/50G
16.32.15.202CentOS Linux release 7.94C/4G/50G
16.32.15.210VIP地址

2、服务器部署架构规划

我本次实验工作节点是一个,生产环境中根据需求自行扩容即可!

角色IP地址主机名安装的组件
控制节点16.32.15.115master-1apiserver、controller-manager、scheduler、etcd、docker、keepalived、nginx
控制节点16.32.15.200master-2apiserver、controller-manager、scheduler、etcd、docker、keepalived、nginx
控制节点16.32.15.201master-3apiserver、controller-manager、scheduler、etcd、docker、keepalived、nginx
工作节点16.32.15.202node-1kubelet、kube-proxy、docker、calico、coredns
VIP地址16.32.15.210

3、组件版本信息

序号组件名称版本备注
1Docker20.10.6
2Cri-dockerd0.3.14下载地址:
3Etcd3.4.33下载地址:
4Nginx1.20.1
5Keepalived1.3.5
8Calico3.25.0
9CoreDNS1.9.3
10Kubernetes集群各组件1.26.15下载地址:

4、实验架构图

在这里插入图片描述

二、初始化环境操作

提示:初始化环境操作,所有设计到的服务器中都需要操作!

1、关闭防火墙

1、安装iptable防火墙(先安装 在禁用)

yum install iptables-services -y

2、关闭防火墙限制

systemctl disable firewalld --now
setenforce 0
sed  -i -r 's/SELINUX=[ep].*/SELINUX=disabled/g' /etc/selinux/configservice iptables stop
systemctl disable iptables
iptables -F

2、配置本地域名解析

1、配置本地域名解析

cat  >> /etc/hosts << EOF
16.32.15.115 master-1
16.32.15.200 master-2
16.32.15.201 master-3
16.32.15.202 node-1
EOF

2、修改服务器主机名(对应服务器中执行)

hostnamectl set-hostname master-1 && bash
hostnamectl set-hostname master-2 && bash
hostnamectl set-hostname master-3 && bash
hostnamectl set-hostname node-1 && bash

3、配置服务器时间保持一致

1、设置时区

timedatectl set-timezone Asia/Shanghai

2、同步阿里云时间源

yum -y install ntpdate
ntpdate ntp1.aliyun.com

3、添加定时同步 每天凌晨1点自动同步时间

echo "0 1 * * * ntpdate ntp1.aliyun.com" >> /var/spool/cron/root
crontab -l

4、禁用swap交换分区(K8S强制要求禁用)

1、禁用swap交换分区

swapoff --all

2、禁止开机自启动swap交换分区

sed -i -r '/swap/ s/^/#/' /etc/fstab

5、配置主机之间无密码登录

1、生成ssh密钥对

ssh-keygen -t rsa

2、将本地的ssh公钥文件COPY到远程主机

ssh-copy-id -i ~/.ssh/id_rsa.pub master-1
ssh-copy-id -i ~/.ssh/id_rsa.pub master-2
ssh-copy-id -i ~/.ssh/id_rsa.pub master-3
ssh-copy-id -i ~/.ssh/id_rsa.pub node-1

6、修改Linux内核参数,添加网桥过滤器和地址转发功能

1、添加内核参数

cat >> /etc/sysctl.d/kubernetes.conf <<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOFsysctl -p /etc/sysctl.d/kubernetes.conf

2、加载网桥过滤器模块

modprobe br_netfilter
lsmod | grep br_netfilter # 验证是否生效

7、启用ipvs功能

在K8S中Service有两种代理模型,一种是基于iptables的,一种是基于ipvs,两者对比ipvs的性能要高,如果想要使用ipvs模型,需要手动载入ipvs模块。

1、安装ipvs

yum -y install ipset ipvsadm

2、配置ipvs功能

cat > /etc/sysconfig/modules/ipvs.modules <<EOF
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4  
EOFchmod +x /etc/sysconfig/modules/ipvs.modules
/etc/sysconfig/modules/ipvs.modules

3、验证ipvs模块

lsmod | grep -e ip_vs -e nf_conntrack_ipv4

8、安装Docker容器组件

1、配置阿里安装源,并安装docker

curl -o /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo
wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
yum makecache# yum-utils软件用于提供yum-config-manager程序
yum install -y yum-utils# 使用yum-config-manager创建docker阿里存储库
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repoyum install docker-ce-20.10.6 docker-ce-cli-20.10.6 -y

2、配置docker国内加速器

mkdir /etc/docker
cat <<EOF > /etc/docker/daemon.json
{"registry-mirrors": [
"https://vm1wbfhf.mirror.aliyuncs.com",
"http://f1361db2.m.daocloud.io",
"https://hub-mirror.c.163.com",
"https://docker.mirrors.ustc.edu.cn",
"https://mirror.baidubce.com",
"https://ustc-edu-cn.mirror.aliyuncs.com",
"https://registry.cn-hangzhou.aliyuncs.com",
"https://ccr.ccs.tencentyun.com",
"https://hub.daocloud.io",
"https://docker.shootchat.top",
"https://do.nark.eu.org",
"https://dockerproxy.com",
"https://docker.m.daocloud.io",
"https://dockerhub.timeweb.cloud",
"https://docker.shootchat.top",
"https://do.nark.eu.org"],
"exec-opts": ["native.cgroupdriver=systemd"]
}
EOF

3、启动docker并设置开机自启

systemctl enable docker --now
systemctl status docker

9、安装cri-dockerd插件

Kubernetes 1.24 版本之后不在兼容docker了,如果还需要使用docker那就需要安装cri-dockerd了,或者使用其他容器运行时,比如containerd等。

1、安装cri-dockerd插件

wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.14/cri-dockerd-0.3.14-3.el7.x86_64.rpm
rpm -ivh cri-dockerd-0.3.14-3.el7.x86_64.rpm

2、备份并更新Systemd文件

mv /usr/lib/systemd/system/cri-docker.service{,.default}
cat > /usr/lib/systemd/system/cri-docker.service << EOF
[Unit]
Description=CRI Interface for Docker Application Container Engine
Documentation=https://docs.mirantis.com
After=network-online.target firewalld.service docker.service
Wants=network-online.target
Requires=cri-docker.socket
[Service]
Type=notify
ExecStart=/usr/bin/cri-dockerd --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.7
ExecReload=/bin/kill -s HUP \$MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
StartLimitBurst=3
StartLimitInterval=60s
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
Delegate=yes
KillMode=process
[Install]
WantedBy=multi-user.target
EOF

3、启动并加入开机自启

systemctl daemon-reload
systemctl start cri-docker.service 
systemctl enable cri-docker.service 
systemctl status cri-docker.service 

4、查看sock文件

ll -l /var/run/cri-dockerd.sock

K8S kubelet组件指定cirSocket时,将填写unix:///var/run/cri-dockerd.sock

10、安装基础软件包

平常排查问题使用

yum install -y yum-utils device-mapper-persistent-data lvm2 wget net-tools nfs-utils lrzsz gcc gcc-c++ make cmake libxml2-devel openssl-devel curl curl-devel unzip sudo ntp libaio-devel wget vim ncurses-devel autoconf automake zlib-devel  python-devel epel-release openssh-server socat  ipvsadm conntrack ntpdate telnet rsync openssh-clients

三、部署etcd高可用集群

提示:在master-1、master-2、master-3中部署etcd集群,涉及证书配置可以先在其中一台生成,然后同步到其他master主机中。

1、签发etcd证书

注意:在操作签发证书操作时一定要检查三台master主机服务器时间、时区是否一致,会导致证书不可用!!

1、创建etcd目录(master-1、master-2、master-3中都要先创建)

mkdir /etc/etcd/{ssl,data} -p

2、安装签发证书工具

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo

3、创建CA生成证书签名请求文件

创建工作目录,证书和配置相关文件在此目录进行生成,之后在同步在master主机。

mkdir ~/workdir
cd ~/workdir

创建CA证书签名请求文件

cat > ~/workdir/ca-csr.json << EOF
{"CN": "kubernetes","key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","ST": "Hebei","L": "Handan","O": "k8s","OU": "system"}],"ca": {"expiry": "87600h"}
}
EOF

重要参数解释:

  • CN:证书的公共名称
  • algo:指定使用 RSA 算法
  • size:RSA 密钥的大小,以位为单位
  • expiry:证书过期时间,87600h=10年

4、生成CA根证书

cfssl gencert -initca ca-csr.json  | cfssljson -bare ca

5、创建CA证书配置文件,用于定义证书颁发机构 (CA) 的签名策略和配置

cat > ~/workdir/ca-config.json << EOF
{"signing": {"default": {"expiry": "87600h"},"profiles": {"kubernetes": {"usages": ["signing","key encipherment","server auth","client auth"],"expiry": "87600h"}}}
}
EOF

重要参数解释:

  • usages:定义了证书可以用来做什么。这个配置指定了四种用途

    • "key encipherment": 用于加密密钥
    • "server auth": 用于服务器身份验证
    • "client auth": 用于客户端身份验证
  • expiry:指定了kubernetes 配置文件中定义的证书有效期也是 10 年。

6、创建etcd生成证书签名请求文件

cat > ~/workdir/etcd-csr.json << EOF
{"CN": "etcd","hosts": ["127.0.0.1","16.32.15.115","16.32.15.200","16.32.15.201","16.32.15.210"],"key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","ST": "Hebei","L": "Handan","O": "k8s","OU": "system"}]
}
EOF

注意:上述文件hosts字段中IP为所有etcd节点的集群内部通信IP,可以预留几个,后续做扩容用,就不用在重新配置证书了。

7、签发etcd证书

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes etcd-csr.json | cfssljson  -bare etcd

重要参数解释:

  • gencert:生成证书
  • -ca=ca.pem:指定证书颁发机构(CA)的证书文件 ca.pem,用于签发新证书
  • -ca-key=ca-key.pem:指定 CA 的私钥文件
  • -config=ca-config.json:使用 ca-config.json 文件中定义的配置来生成证书。这些配置包括签名策略和证书有效期等
  • -profile=kubernetes:使用配置文件中的 kubernetes 配置文件作为证书的签名配置
  • -bare etcd:定生成的证书文件名为 etcd

8、同步相关证书文件到/etc/etcd/ssl目录

cp -p ca*.pem /etc/etcd/ssl/
cp -p etcd*.pem /etc/etcd/ssl/

查看证书:

ls -l /etc/etcd/ssl*

9、将相关配置文件同步至其他Master主机中

scp -p /etc/etcd/ssl/* root@master-2:/etc/etcd/ssl
scp -p /etc/etcd/ssl/* root@master-3:/etc/etcd/ssl

2、搭建etcd高可用集群

首先需要下载对应版本etcd二进制包 官网下载地址:

1、解压压缩包并移动etcd相关命令

tar zxf etcd-v3.5.15-linux-amd64.tar.gz
cp -p etcd-v3.5.15-linux-amd64/etcd* /usr/local/bin/

2、创建etcd集群配置文件,各个节点配置文件是不同的,需要单独配置

master-1服务器配置:

cat > /etc/etcd/etcd.conf << EOF
ETCD_NAME="etcd1"
ETCD_DATA_DIR="/etc/etcd/data"
ETCD_LISTEN_PEER_URLS="https://16.32.15.115:2380"
ETCD_LISTEN_CLIENT_URLS="https://16.32.15.115:2379,http://127.0.0.1:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://16.32.15.115:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://16.32.15.115:2379"
ETCD_INITIAL_CLUSTER="etcd1=https://16.32.15.115:2380,etcd2=https://16.32.15.200:2380,etcd3=https://16.32.15.201:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF

master-2服务器配置:

cat > /etc/etcd/etcd.conf << EOF
ETCD_NAME="etcd2"
ETCD_DATA_DIR="/etc/etcd/data"
ETCD_LISTEN_PEER_URLS="https://16.32.15.200:2380"
ETCD_LISTEN_CLIENT_URLS="https://16.32.15.200:2379,http://127.0.0.1:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://16.32.15.200:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://16.32.15.200:2379"
ETCD_INITIAL_CLUSTER="etcd1=https://16.32.15.115:2380,etcd2=https://16.32.15.200:2380,etcd3=https://16.32.15.201:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF

master-3服务器配置:

cat > /etc/etcd/etcd.conf << EOF
ETCD_NAME="etcd3"
ETCD_DATA_DIR="/etc/etcd/data"
ETCD_LISTEN_PEER_URLS="https://16.32.15.201:2380"
ETCD_LISTEN_CLIENT_URLS="https://16.32.15.201:2379,http://127.0.0.1:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://16.32.15.201:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://16.32.15.201:2379"
ETCD_INITIAL_CLUSTER="etcd1=https://16.32.15.115:2380,etcd2=https://16.32.15.200:2380,etcd3=https://16.32.15.201:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF

重要配置项解释:

  • ETCD_NAME:节点名称,集群中唯一
  • ETCD_DATA_DIR:数据目录
  • ETCD_LISTEN_PEER_URLS:集群通信监听地址
  • ETCD_LISTEN_CLIENT_URLS:客户端访问监听地址
  • ETCD_INITIAL_ADVERTISE_PEER_URLS:集群通告地址
  • ETCD_ADVERTISE_CLIENT_URLS:客户端通告地址
  • ETCD_INITIAL_CLUSTER:集群节点地址
  • ETCD_INITIAL_CLUSTER_TOKEN:集群Token
  • ETCD_INITIAL_CLUSTER_STATE:加入集群的当前状态,new是新集群,existing表示加入已有集群

3、创建Systemd管理etcd服务

cat > /usr/lib/systemd/system/etcd.service << EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target[Service]
Type=notify
EnvironmentFile=-/etc/etcd/etcd.conf
WorkingDirectory=/etc/etcd/data/
ExecStart=/usr/local/bin/etcd \\--cert-file=/etc/etcd/ssl/etcd.pem \\--key-file=/etc/etcd/ssl/etcd-key.pem \\--trusted-ca-file=/etc/etcd/ssl/ca.pem \\--peer-cert-file=/etc/etcd/ssl/etcd.pem \\--peer-key-file=/etc/etcd/ssl/etcd-key.pem \\--peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \\--peer-client-cert-auth \\--client-cert-auth
Restart=on-failure
RestartSec=5
LimitNOFILE=65536[Install]
WantedBy=multi-user.target
EOF

将etcd相关命令和Systemd配置同步到其他master主机中:

scp -p /usr/local/bin/etcd* root@master-2:/usr/local/bin/
scp -p /usr/local/bin/etcd* root@master-3:/usr/local/bin/scp -p /usr/lib/systemd/system/etcd.service root@master-2:/usr/lib/systemd/system/
scp -p /usr/lib/systemd/system/etcd.service root@master-3:/usr/lib/systemd/system/

4、启动集群,并加入开机自启动(三台master主机中执行)

systemctl enable etcd
systemctl start etcd
systemctl status etcd

3、测试连接集群

ETCDCTL_API=3/usr/local/bin/etcdctl --write-out=table --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --endpoints=https://16.32.15.115:2379,https://16.32.15.200:2379,https://16.32.15.201:2379  endpoint health+---------------------------+--------+-------------+-------+
|         ENDPOINT          | HEALTH |    TOOK     | ERROR |
+---------------------------+--------+-------------+-------+
| https://16.32.15.115:2379 |   true | 19.402185ms |       |
| https://16.32.15.200:2379 |   true | 21.604449ms |       |
| https://16.32.15.201:2379 |   true | 53.427052ms |       |
+---------------------------+--------+-------------+-------+

四、部署Nginx+Keepalived

1、部署并配置Nginx服务

注意:master-1、master-2主机都需要执行,步骤都是一致的。

1、安装Nginx服务

yum install nginx-1.20.1 nginx-mod-stream -y

2、编辑nginx配置文件

vim /etc/nginx/nginx.confuser nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;include /usr/share/nginx/modules/*.conf;events {worker_connections 1024;
}stream {log_format  main  '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';access_log  /var/log/nginx/k8s-access.log  main;upstream k8s-apiserver {server 16.32.15.115:6443;server 16.32.15.200:6443;server 16.32.15.201:6443;}server {listen 16443; proxy_pass k8s-apiserver;}
}http {log_format  main  '$remote_addr - $remote_user [$time_local] "$request" ''$status $body_bytes_sent "$http_referer" ''"$http_user_agent" "$http_x_forwarded_for"';access_log  /var/log/nginx/access.log  main;sendfile            on;tcp_nopush          on;tcp_nodelay         on;keepalive_timeout   65;types_hash_max_size 2048;include             /etc/nginx/mime.types;default_type        application/octet-stream;
}

3、启动并加入开机自启动

systemctl start nginx
systemctl enable nginx
systemctl status nginx

4、查看端口是否监听

netstat -anput |grep :16443

2、部署并配置Keepalived服务

提示:三台master主机都需要执行,除了keepalived配置内容不一致,其余都一致。

1、安装keepalived

yum -y install keepalived

2、编辑keepalived配置文件

master-1配置如下:

cat > /etc/keepalived/keepalived.conf << EOFglobal_defs {smtp_server 127.0.0.1smtp_connect_timeout 30router_id NGINX_MASTER
}
vrrp_script check_nginx {script "/etc/keepalived/check_nginx.sh"
}vrrp_instance VI_1 {state MASTERinterface ens32      # 网卡名称virtual_router_id 51priority 100advert_int 1         # VRRP心跳包通告间隔时间authentication {auth_type PASSauth_pass 1111}virtual_ipaddress {16.32.15.210/24  # VIP地址}track_script {check_nginx}
}
EOF

master-2配置如下:

cat > /etc/keepalived/keepalived.conf << EOFglobal_defs {smtp_server 127.0.0.1smtp_connect_timeout 30router_id NGINX_MASTER
}
vrrp_script check_nginx {script "/etc/keepalived/check_nginx.sh"
}vrrp_instance VI_1 {state BACKUPinterface eth0     virtual_router_id 51priority 50advert_int 1        authentication {auth_type PASSauth_pass 1111}virtual_ipaddress {16.32.15.210/24 }track_script {check_nginx}
}
EOF

master-3配置如下:

cat > /etc/keepalived/keepalived.conf << EOFglobal_defs {smtp_server 127.0.0.1smtp_connect_timeout 30router_id NGINX_MASTER
}
vrrp_script check_nginx {script "/etc/keepalived/check_nginx.sh"
}vrrp_instance VI_1 {state BACKUPinterface eth0     virtual_router_id 51priority 10advert_int 1         authentication {auth_type PASSauth_pass 1111}virtual_ipaddress {16.32.15.210/24 }track_script {check_nginx}
}
EOF

3、添加Nginx检测脚本

vim /etc/keepalived/check_nginx.sh#!/bin/bash
count=$(ps -ef |grep nginx | grep sbin | egrep -cv "grep|$$")
if [ $count -eq 0 ];thensystemctl stop keepalived
fi

添加执行权限:

chmod +x /etc/keepalived/check_nginx.sh

3、启动并加入开机自启动

systemctl start keepalived
systemctl enable keepalived
systemctl status keepalived

3、测试VIP故障自动切换

停止掉master-1服务器nginx服务,VIP会切换到master-2服务器中。

1、master-1服务器中操作

查看是否有VIP地址,正常是存在的:

hostname -I|grep 210

在这里插入图片描述

停止nginx服务器,这时VIP地址会漂移到master-2主机上面:

systemctl stop nginx

当停止nginx后,会发现master-1服务器中的VIP没有了

2、master-2服务器中操作

查看VIP地址是否切换到master-2服务器中

hostname -I|grep 210

在这里插入图片描述

验证完成!!后面的Kubernetes相关组件连接apiserver地址填写 https://16.32.15.210:16443 即可!

五、部署Kubernetes控制节点相关组件

1、下载二进制包和环境准备相关操作

1、官网下载二进制包

官网下载地址:

在这里插入图片描述

在这里插入图片描述

2、解压缩,并将二进制命令放到/usr/local/bin/目录下

tar zxf kubernetes-server-linux-amd64.tar.gz
cp -p kubernetes/server/bin/{kube-apiserver,kube-controller-manager,kube-scheduler,kubectl} /usr/local/bin/

3、将二进制命令同步到其他K8S主机

scp -p /usr/local/bin/kube* master-2:/usr/local/bin/
scp -p /usr/local/bin/kube* master-3:/usr/local/bin/# kubelet、kube-proxy组件同步至Node节点中
scp -p kubernetes/server/bin/{kubelet,kube-proxy} node-1:/usr/local/bin/

4、创建Kubernetes相关目录(所有主机中都要创建)

mkdir -p /etc/kubernetes/{ssl,logs}

2、部署api-service组件

提供K8S API,是整个系统的对外接口,提供资源操作的唯一入口,供客户端和其它组件调用,提供了K8S各类资源对象(pod,deployment,Service等)的增删改查,是整个系统的数据总线和数据中心,并提供认证、授权、访问控制、API注册和发现等机制,并将操作对象持久化到etcd中。

1、创建token.csv文件

该文件中是一个预设的用户配置,后面kubelet认证所使用的!

# 格式: token,用户名,UID,用户组
cat > /etc/kubernetes/token.csv << EOF
$(head -c 16 /dev/urandom | od -An -t x | tr -d ' '),kubelet-bootstrap,10001,"system:kubelet-bootstrap"
EOF

2、创建apiserver证书签名请求文件

cat > ~/workdir/kube-apiserver-csr.json << EOF
{"CN": "kubernetes","hosts": ["127.0.0.1","16.32.15.115","16.32.15.200","16.32.15.201","16.32.15.202","16.32.15.210","10.255.0.1","kubernetes","kubernetes.default","kubernetes.default.svc","kubernetes.default.svc.cluster","kubernetes.default.svc.cluster.local"],"key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","ST": "Hebei","L": "Handan","O": "k8s","OU": "system"}]
}
EOF

注意:hosts 字段需要指定授权使用该证书的 IP 或域名列表。 由于该证书后续被 K8S Master 集群使用,需要将Master节点的IP都填上,同时还需要填写 Service 网络的首个IP。(一般是 kube-apiserver 指定的 service-cluster-ip-range 网段的第一个IP,如 10.255.0.1)

3、生成证书

注意:这里使用的CA颁发机构和etcd是同一个,保证当前目录下存在ca.pemca-key.pem文件!

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-apiserver-csr.json | cfssljson -bare kube-apiserver

4、移动证书到指定位置

cp -p ca*.pem /etc/kubernetes/ssl/
cp -p kube-apiserver*.pem /etc/kubernetes/ssl/ls -l /etc/kubernetes/ssl/

5、创建kube-apiserver配置文件

Master-1配置:

cat > /etc/kubernetes/kube-apiserver.conf <<EOF
KUBE_APISERVER_OPTS="--enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \--anonymous-auth=false \\--bind-address=16.32.15.115 \\--secure-port=6443 \\--advertise-address=16.32.15.115 \\--authorization-mode=Node,RBAC \\--runtime-config=api/all=true \\--enable-bootstrap-token-auth \\--service-cluster-ip-range=10.255.0.0/16 \\--token-auth-file=/etc/kubernetes/token.csv \\--service-node-port-range=30000-50000 \\--tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem  \\--tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem \\--client-ca-file=/etc/kubernetes/ssl/ca.pem \\--kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem \\--kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem \\--service-account-key-file=/etc/kubernetes/ssl/ca-key.pem \\--service-account-signing-key-file=/etc/kubernetes/ssl/ca-key.pem  \\--service-account-issuer=https://kubernetes.default.svc.cluster.local \\--etcd-cafile=/etc/etcd/ssl/ca.pem \\--etcd-certfile=/etc/etcd/ssl/etcd.pem \\--etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \\--etcd-servers=https://16.32.15.115:2379,https://16.32.15.200:2379,https://16.32.15.201:2379 \\--allow-privileged=true \\--apiserver-count=3 \\--audit-log-maxage=30 \\--audit-log-maxbackup=3 \\--audit-log-maxsize=100 \\--audit-log-path=/etc/kubernetes/logs/kube-apiserver-audit.log \\--logging-format=json \\--event-ttl=1h \\--v=4"
EOF

Master-2配置:

cat > /etc/kubernetes/kube-apiserver.conf << EOF
KUBE_APISERVER_OPTS="--enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \--anonymous-auth=false \\--bind-address=16.32.15.200 \\--secure-port=6443 \\--advertise-address=16.32.15.200 \\--authorization-mode=Node,RBAC \\--runtime-config=api/all=true \\--enable-bootstrap-token-auth \\--service-cluster-ip-range=10.255.0.0/16 \\--token-auth-file=/etc/kubernetes/token.csv \\--service-node-port-range=30000-50000 \\--tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem  \\--tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem \--client-ca-file=/etc/kubernetes/ssl/ca.pem \--kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem \--kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem \--service-account-key-file=/etc/kubernetes/ssl/ca-key.pem \--service-account-signing-key-file=/etc/kubernetes/ssl/ca-key.pem  \--service-account-issuer=https://kubernetes.default.svc.cluster.local \--etcd-cafile=/etc/etcd/ssl/ca.pem \\--etcd-certfile=/etc/etcd/ssl/etcd.pem \\--etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \\--etcd-servers=https://16.32.15.115:2379,https://16.32.15.200:2379,https://16.32.15.201:2379 \\--allow-privileged=true \\--apiserver-count=3 \\--audit-log-maxage=30 \\--audit-log-maxbackup=3 \\--audit-log-maxsize=100 \\--audit-log-path=/etc/kubernetes/logs/kube-apiserver-audit.log \\--logging-format=json \\--event-ttl=1h \\--v=4"
EOF

Master-3配置:

cat > /etc/kubernetes/kube-apiserver.conf << EOF
KUBE_APISERVER_OPTS="--enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \\--anonymous-auth=false \\--bind-address=16.32.15.201 \\--secure-port=6443 \\--advertise-address=16.32.15.201 \\--authorization-mode=Node,RBAC \\--runtime-config=api/all=true \\--enable-bootstrap-token-auth \\--service-cluster-ip-range=10.255.0.0/16 \\--token-auth-file=/etc/kubernetes/token.csv \\--service-node-port-range=30000-50000 \\--tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem  \\--tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem \\--client-ca-file=/etc/kubernetes/ssl/ca.pem \\--kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem \\--kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem \\--service-account-key-file=/etc/kubernetes/ssl/ca-key.pem \\--service-account-signing-key-file=/etc/kubernetes/ssl/ca-key.pem  \\--service-account-issuer=https://kubernetes.default.svc.cluster.local \\--etcd-cafile=/etc/etcd/ssl/ca.pem \\--etcd-certfile=/etc/etcd/ssl/etcd.pem \\--etcd-keyfile=/etc/etcd/ssl/etcd-key.pem \\--etcd-servers=https://16.32.15.115:2379,https://16.32.15.200:2379,https://16.32.15.201:2379 \\--allow-privileged=true \\--apiserver-count=3 \\--audit-log-maxage=30 \\--audit-log-maxbackup=3 \\--audit-log-maxsize=100 \\--audit-log-path=/etc/kubernetes/logs/kube-apiserver-audit.log \\--logging-format=json \\--event-ttl=1h \\--v=4"
EOF

重要配置项解释:

  • enable-admission-plugins:指定启用的 Admission 插件列表;

  • anonymous-auth:禁用匿名用户的认证。默认情况下,Kubernetes 允许匿名用户访问 API;

  • bind-address: 设置 API Server 绑定的 IP 地址;

  • secure-port:指定 API Server 监听的安全端口;

  • advertise-address:设置 API Server 向集群其他组件公开的 IP 地址;

  • authorization-mode:设置授权模式,这里启用了 NodeRBAC 模式;

  • runtime-config:配置 API 的运行时设置,这里设置为 api/all=true,表示启用所有 API 版本;

  • enable-bootstrap-token-auth:启用 Bootstrap Token 认证,这用于集群节点加入时的认证;

  • service-cluster-ip-range: 指定Service服务的 IP 地址范围;

  • token-auth-file:Bootstrap Token 认证文件路径;

  • service-node-port-range:NodePort类型端口暴露;

  • tls-cert-file:指定 API Server 使用的 TLS 证书文件路径,用于加密与客户端的通信;

  • tls-private-key-file:指定 API Server 使用的 TLS 私钥文件路径;

  • client-ca-file:指定客户端证书的 CA 文件路径;

  • etcd-servers:连接etcd地址列表;

  • allow-privileged:是否允许特权容器;

  • apiserver-count: 设置 API Server 实例的数量;

  • logging-format:设置日志格式,这里使用 json 格式。

6、添加Systemd管理apiserver

cat > /usr/lib/systemd/system/kube-apiserver.service << EOF[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=etcd.service
Wants=etcd.service[Service]
EnvironmentFile=-/etc/kubernetes/kube-apiserver.conf
ExecStart=/usr/local/bin/kube-apiserver \$KUBE_APISERVER_OPTS
Restart=on-failure
RestartSec=5
Type=notify
LimitNOFILE=65536[Install]
WantedBy=multi-user.target
EOF

7、同步相关配置到Master主机

scp -p /usr/lib/systemd/system/kube-apiserver.service root@master-2:/usr/lib/systemd/system/
scp -p /usr/lib/systemd/system/kube-apiserver.service root@master-3:/usr/lib/systemd/system/scp -p /etc/kubernetes/ssl/* master-2:/etc/kubernetes/ssl/
scp -p /etc/kubernetes/ssl/* master-3:/etc/kubernetes/ssl/scp -p /etc/kubernetes/token.csv master-2:/etc/kubernetes/
scp -p /etc/kubernetes/token.csv master-3:/etc/kubernetes/

8、启动apiserver,并加入开机自启动(三台master主机中执行)

systemctl enable kube-apiserver
systemctl start kube-apiserver
systemctl status kube-apiserver

9、访问apiserver

curl -k https://16.32.15.210:16443/{"kind": "Status","apiVersion": "v1","metadata": {},"status": "Failure","message": "Unauthorized","reason": "Unauthorized","code": 401
}

正常是无法访问直接apiserver的,所以此处返回401是属于正常的

3、部署kubectl组件

kubectl,是管理K8S的命令行工具,可以操作K8S中的资源对象,如增删改查等。

1、创建证书签名请求文件

cat > ~/workdir/admin-csr.json << EOF
{"CN": "admin","hosts": [],"key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","ST": "Hebei","L": "Handan","O": "system:masters","OU": "system"}]
}
EOF

2、生成证书,使用apiservice的CA证书颁发机构进行生成,这样apiserver就会信任kubectl了

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin

3、将生成的证书同步到指定位置

cp admin*.pem /etc/kubernetes/ssl/
ls -l /etc/kubernetes/ssl/admin*.pem

4、配置安全上下文

创建默认配置文件目录:

mkdir -p $HOME/.kube
chown $(id -u):$(id -g) $HOME/.kube

设置集群参数:

kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://16.32.15.210:16443 --kubeconfig=$HOME/.kube/config

设置客户端认证参数:

kubectl config set-credentials admin --client-certificate=admin.pem --client-key=admin-key.pem --embed-certs=true --kubeconfig=$HOME/.kube/config

设置上下文参数:

kubectl config set-context kubernetes --cluster=kubernetes --user=admin --kubeconfig=$HOME/.kube/config
kubectl config use-context kubernetes --kubeconfig=$HOME/.kube/config

查看当前配置信息:

kubectl config view

授权K8S证书访问kubelet api权限:

kubectl create clusterrolebinding kube-apiserver:kubelet-apis --clusterrole=system:kubelet-api-admin --user kubernetes

5、查看集群组件状态

kubectl cluster-info
kubectl get componentstatuses

6、将相关配置文件同步至其他Master主机中

scp -rp ~/.kube master-2:~/
scp -rp ~/.kube master-3:~/scp -p  /etc/kubernetes/ssl/admin*.pem master-2:/etc/kubernetes/ssl/
scp -p  /etc/kubernetes/ssl/admin*.pem master-3:/etc/kubernetes/ssl/

将配置同步过去,可以直接使用kubectl命令了

kubectl cluster-info

4、部署controller-manager组件

controller-manager,负责管理和维护集群中的各类控制器,它的作用是监控集群状态,并执行相应的操作,确保集群的实际状态与期望状态一致。具体来说,它可以处理资源的创建、更新和删除操作,确保应用程序和服务按照定义运行。

1、创建证书签名请求文件

cat > ~/workdir/kube-controller-manager-csr.json << EOF
{"CN": "system:kube-controller-manager","key": {"algo": "rsa","size": 2048},"hosts": ["127.0.0.1","16.32.15.115","16.32.15.200","16.32.15.201","16.32.15.210"],"names": [{"C": "CN","ST": "Heibei","L": "Handan","O": "system:kube-controller-manager","OU": "system"}]
}
EOF

注意:"O": "system:kube-controller-manager",是K8S内置的 ClusterRoleBindings,system:kube-controller-manager 赋予 kube-controller-manager工作所需的权限!

查看内置system:kube-controller-manager ClusterRoleBindings

kubectl get clusterrolebindings|grep 'system:kube-controller-manager'

2、生成证书

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager

3、将证书同步到指定位置

cp -p kube-controller-manager*.pem /etc/kubernetes/ssl/

4、配置安全上下文

设置集群参数:

kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://16.32.15.210:16443 --kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig

设置客户端认证参数:

kubectl config set-credentials system:kube-controller-manager --client-certificate=kube-controller-manager.pem --client-key=kube-controller-manager-key.pem --embed-certs=true --kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig

设置上下文:

kubectl config set-context system:kube-controller-manager --cluster=kubernetes --user=system:kube-controller-manager --kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig
kubectl config use-context system:kube-controller-manager --kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig

5、创建controller-manager配置文件

cat > /etc/kubernetes/kube-controller-manager.conf << EOF
KUBE_CONTROLLER_MANAGER_OPTS="--secure-port=10257 \\--bind-address=127.0.0.1 \\--kubeconfig=/etc/kubernetes/kube-controller-manager.kubeconfig \\--service-cluster-ip-range=10.255.0.0/16 \\--cluster-name=kubernetes \\--cluster-signing-cert-file=/etc/kubernetes/ssl/ca.pem \\--cluster-signing-key-file=/etc/kubernetes/ssl/ca-key.pem \\--allocate-node-cidrs=true \\--cluster-cidr=10.0.0.0/16 \\--root-ca-file=/etc/kubernetes/ssl/ca.pem \\--service-account-private-key-file=/etc/kubernetes/ssl/ca-key.pem \\--leader-elect=true \\--feature-gates=RotateKubeletServerCertificate=true \\--controllers=*,bootstrapsigner,tokencleaner \\--horizontal-pod-autoscaler-sync-period=10s \\--tls-cert-file=/etc/kubernetes/ssl/kube-controller-manager.pem \\--tls-private-key-file=/etc/kubernetes/ssl/kube-controller-manager-key.pem \\--use-service-account-credentials=true \\--logging-format=json \\--cluster-signing-duration=87600h \\--v=3"
EOF

重要参数解释:

  • secure-port:监听的端口;
  • bind-address:端口绑定地址;
  • kubeconfig:指定用于访问 Kubernetes 集群的配置文件路径;
  • service-cluster-ip-range:Service地址范围;
  • cluster-name:集群名称;
  • allocate-node-cidrs:启用节点 CIDR 分配,用于设置每个节点的 IP 地址范围;
  • cluster-cidr:集群的 CIDR 地址范围;
  • root-ca-file:根 CA 文件路径,用于验证集群中的证书;
  • leader-elect:启用领导者选举,用于在多实例环境中选举一个控制管理器作为主节点;
  • feature-gates:启用实验性功能,RotateKubeletServerCertificate,允许轮换 kubelet 的服务器证书;
  • controllers:指定启用的控制器列表,* 表示启用所有控制器,额外启用了 bootstrapsigner 和 tokencleaner;
  • horizontal-pod-autoscaler-sync-period:设置水平 Pod 自动扩展器的同步周期,这里为 10s;
  • use-service-account-credentials:启用服务账户凭证的使用,设置为 true;
  • logging-format:日志格式,设置为 json;
  • cluster-signing-duration:证书签名的有效时长,这里为 87600h10(年)。

6、添加Systemd管理controller-manager

cat > /usr/lib/systemd/system/kube-controller-manager.service << EOF
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=-/etc/kubernetes/kube-controller-manager.conf
ExecStart=/usr/local/bin/kube-controller-manager \$KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
EOF

7、将相关配置文件同步至其他Master主机中

scp -rp /etc/kubernetes/ssl/kube-controller-manager*.pem master-2:/etc/kubernetes/ssl/
scp -rp /etc/kubernetes/ssl/kube-controller-manager*.pem master-3:/etc/kubernetes/ssl/scp -rp /etc/kubernetes/kube-controller-manager.kubeconfig master-2:/etc/kubernetes/
scp -rp /etc/kubernetes/kube-controller-manager.kubeconfig master-3:/etc/kubernetes/scp -rp /etc/kubernetes/kube-controller-manager.conf master-2:/etc/kubernetes/
scp -rp /etc/kubernetes/kube-controller-manager.conf master-3:/etc/kubernetes/scp -rp /usr/lib/systemd/system/kube-controller-manager.service master-2:/usr/lib/systemd/system/
scp -rp /usr/lib/systemd/system/kube-controller-manager.service master-3:/usr/lib/systemd/system/

8、启动并加入开机自启动(三台master主机中执行)

systemctl enable kube-controller-manager.service
systemctl start kube-controller-manager.service
systemctl status kube-controller-manager.service

5、部署kube-scheduler组件

kube-scheduler,负责将 Pods 调度到合适的节点上,根据预定义的策略和资源的可用性,选择最适合的节点以运行 Pod,kube-scheduler 会考虑各种因素,如节点资源、Pod 需求、亲和性和反亲和性规则等,确保 Pod 在集群中均匀地分布并满足性能需求。

1、创建证书签名请求文件

cat > ~/workdir/kube-scheduler-csr.json << EOF
{"CN": "system:kube-scheduler","hosts": ["127.0.0.1","16.32.15.115","16.32.15.200","16.32.15.201","16.32.15.210"],"key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","ST": "Hebei","L": "Handan","O": "system:kube-scheduler","OU": "system"}]
}
EOF

2、生成证书

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler

3、将证书文件同步到指定位置

cp -p kube-scheduler*.pem /etc/kubernetes/ssl/

4、配置安全上下文

设置集群参数:

kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://16.32.15.210:16443 --kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig

设置客户端认证参数:

kubectl config set-credentials system:kube-scheduler --client-certificate=kube-scheduler.pem --client-key=kube-scheduler-key.pem --embed-certs=true --kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig

设置上下文参数:

kubectl config set-context system:kube-scheduler --cluster=kubernetes --user=system:kube-scheduler --kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig
kubectl config use-context system:kube-scheduler --kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig

5、创建scheduler配置文件

cat > /etc/kubernetes/kube-scheduler.conf  << EOF
KUBE_SCHEDULER_OPTS="--bind-address=127.0.0.1 \\
--kubeconfig=/etc/kubernetes/kube-scheduler.kubeconfig \\
--leader-elect=true \\
--logging-format=json \\
--v=2"
EOF

重要参数解释:

  • bind-address:设置绑定地址;
  • kubeconfig:指定用于访问 Kubernetes 集群的配置文件路径;
  • leader-elect:启用领导选举机制;
  • logging-format:日志格式,设置为 json;

6、添加Systemd管理scheduler

cat > /usr/lib/systemd/system/kube-scheduler.service << EOF
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes
[Service]
EnvironmentFile=-/etc/kubernetes/kube-scheduler.conf
ExecStart=/usr/local/bin/kube-scheduler \$KUBE_SCHEDULER_OPTS
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
EOF

7、将相关配置文件同步至其他Master主机中

scp -rp /etc/kubernetes/ssl/kube-scheduler*.pem master-2:/etc/kubernetes/
scp -rp /etc/kubernetes/ssl/kube-scheduler*.pem master-3:/etc/kubernetes/scp -rp /etc/kubernetes/kube-scheduler.kubeconfig master-2:/etc/kubernetes/
scp -rp /etc/kubernetes/kube-scheduler.kubeconfig master-3:/etc/kubernetes/scp -rp /etc/kubernetes/kube-scheduler.conf master-2:/etc/kubernetes/
scp -rp /etc/kubernetes/kube-scheduler.conf master-3:/etc/kubernetes/scp -rp /usr/lib/systemd/system/kube-scheduler.service master-2:/usr/lib/systemd/system/
scp -rp /usr/lib/systemd/system/kube-scheduler.service master-3:/usr/lib/systemd/system/

8、启动并加入开机自启动(三台master主机中执行)

systemctl enable kube-scheduler.service
systemctl start kube-scheduler.service
systemctl status kube-scheduler.service

6、验证控制节点相关组件

查看控制节点各组件状态,如下图表示正常

kubectl get componentstatuses

外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传

六、部署Kubernetes工作节点相关组件

1、部署kubelet组件

每个Node节点上的kubelet定期就会调用API Server的REST接口报告自身状态,API Server接收这些信息后,将节点状态信息更新到etcd中。kubelet也通过API Server监听Pod信息,从而对Node机器上的Pod进行管理,如创建、删除、更新Pod。

1、获取token.csv文件token值,就是上面apiserver指定的token.csv

BOOTSTRAP_TOKEN=$(awk -F "," '{print $1}' /etc/kubernetes/token.csv)
echo ${BOOTSTRAP_TOKEN}

2、配置安全上下文

设置集群参数:

kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://16.32.15.210:16443 --kubeconfig=kubelet-bootstrap.kubeconfig

设置客户端认证参数:

kubectl config set-credentials kubelet-bootstrap --token=${BOOTSTRAP_TOKEN} --kubeconfig=kubelet-bootstrap.kubeconfig

设置上下文:

kubectl config set-context default --cluster=kubernetes --user=kubelet-bootstrap --kubeconfig=kubelet-bootstrap.kubeconfig
kubectl config use-context default --kubeconfig=kubelet-bootstrap.kubeconfig

绑定角色:

kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap

3、将相关配置同步到Node主机

scp -p kubelet-bootstrap.kubeconfig node-1:/etc/kubernetes/
scp -p /etc/kubernetes/ssl/ca.pem node-1:/etc/kubernetes/ssl/

4、创建kubelet.json 文件

注意:后续操作均在node-1操作,16.32.15.202是node-1宿主机IP地址!

cat > /etc/kubernetes/kubelet.json << EOF
{"kind": "KubeletConfiguration","apiVersion": "kubelet.config.k8s.io/v1beta1","authentication": {"x509": {"clientCAFile": "/etc/kubernetes/ssl/ca.pem"},"webhook": {"enabled": true,"cacheTTL": "2m0s"},"anonymous": {"enabled": false}},"authorization": {"mode": "Webhook","webhook": {"cacheAuthorizedTTL": "5m0s","cacheUnauthorizedTTL": "30s"}},"address": "16.32.15.202","port": 10250,"readOnlyPort": 10255,"cgroupDriver": "systemd","hairpinMode": "promiscuous-bridge","serializeImagePulls": false,"clusterDomain": "cluster.local.","clusterDNS": ["10.255.0.2"]
}
EOF

5、创建kubelet配置文件

cat > /etc/kubernetes/kubelet.conf << EOF
KUBELET_OPTS="--bootstrap-kubeconfig=/etc/kubernetes/kubelet-bootstrap.kubeconfig \\--cert-dir=/etc/kubernetes/ssl \\--kubeconfig=/etc/kubernetes/kubelet.kubeconfig \\--config=/etc/kubernetes/kubelet.json \\--pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.9.3 \\--logging-format=json \\--container-runtime-endpoint=unix:///var/run/cri-dockerd.sock \\--v=2"
EOF

重要参数解释:

  • bootstrap-kubeconfig:首次启动向apiserver申请证书;
  • cert-dir:指定证书会路径,apiserver自动生成证书会放到此目录;
  • config:配置文件路径;
  • pod-infra-container-image:指定 Pod 基础设施容器的镜像;
  • container-runtime-endpoint:指定容器运行时sock路径。

6、添加Systemd管理kubelet

mkdir /var/lib/kubelet
cat > /usr/lib/systemd/system/kubelet.service << EOF
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=docker.service
Requires=docker.service
[Service]
EnvironmentFile=-/etc/kubernetes/kubelet.conf
WorkingDirectory=/var/lib/kubelet
ExecStart=/usr/local/bin/kubelet \$KUBELET_OPTS
Restart=on-failure
RestartSec=5[Install]
WantedBy=multi-user.target
EOF

7、启动并加入开机自启动

systemctl enable kubelet.service
systemctl start kubelet.service
systemctl status kubelet.service

8、kubelet启动后会向apiserver发送一个CSR请求,去申请证书

注意:任意Master主机执行

kubectl get csr

在这里插入图片描述

批准csr请求

kubectl certificate approve node-csr-gCK454vfN_Wlj6z-HRvz3scijAzmkvO0CbgR3EyKI_U

批量请求后,在node主机可以查看到apiserver生成的证书文件,如下图:

ll /etc/kubernetes/ssl/

外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传

2、部署kube-proxy组件

kube-proxy,供网络代理和负载均衡,是实现service的通信与负载均衡机制的重要组件,kube-proxy负责为Pod创建代理服务,从apiserver获取所有service信息,并根据service信息创建代理服务,实现service到Pod的请求路由和转发,从而实现K8s层级的虚拟转发网络,将到service的请求转发到后端的pod上。

1、创建证书签名请求文件

cat > kube-proxy-csr.json << EOF
{"CN": "system:kube-proxy","key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","ST": "Hebei","L": "Handan","O": "k8s","OU": "system"}]
}
EOF

2、生成证书

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy

3、配置安全上下文

设置集群参数:

kubectl config set-cluster kubernetes --certificate-authority=ca.pem --embed-certs=true --server=https://16.32.15.210:16443 --kubeconfig=kube-proxy.kubeconfig

设置客户端认证参数:

kubectl config set-credentials kube-proxy --client-certificate=kube-proxy.pem --client-key=kube-proxy-key.pem --embed-certs=true --kubeconfig=kube-proxy.kubeconfig

设置上下文参数:

kubectl config set-context default --cluster=kubernetes --user=kube-proxy --kubeconfig=kube-proxy.kubeconfig
kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

4、将相关配置同步到Node主机

scp -p kube-proxy.kubeconfig node-1:/etc/kubernetes/
scp -p kube-proxy*.pem node-1:/etc/kubernetes/ssl/

5、创建kube-proxy配置文件

注意:后续操作均在node-1操作,16.32.15.202是node-1宿主机IP地址!

cat >> /etc/kubernetes/kube-proxy.yaml << EOF
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 16.32.15.202
clientConnection:kubeconfig: /etc/kubernetes/kube-proxy.kubeconfig
clusterCIDR: 16.32.15.0/24
healthzBindAddress: 16.32.15.202:10256
kind: KubeProxyConfiguration
metricsBindAddress: 16.32.15.202:10249
mode: "ipvs"
EOF

重要参数解释:

  • apiVersion:配置文件的 API 版本;
  • bindAddress:监听的 IP 地址;
  • clientConnection.kubeconfig:指定用于与 Kubernetes API 服务器通信的 kubeconfig 文件;
  • clusterCIDR:集群中 Pod 网络的 CIDR 范围,用于配置和管理 Pod 网络流量;
  • healthzBindAddress: 健康检查的监听地址和端口。
  • metricsBindAddress:指标暴露地址和端口,供监控系统访问;
  • mode:指定代理模式为 ipvs,用于更高效的流量处理和负载均衡。

6、添加Systemd管理scheduler

mkdir /var/lib/kube-proxy
cat > /usr/lib/systemd/system/kube-proxy.service << EOF
[Unit]
Description=Kubernetes Kube-Proxy Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target
[Service]
WorkingDirectory=/var/lib/kube-proxy
ExecStart=/usr/local/bin/kube-proxy \\--config=/etc/kubernetes/kube-proxy.yaml \\--v=2
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
EOF

8、启动并加入开机自启动

systemctl enable kube-proxy.service
systemctl start kube-proxy.service
systemctl status kube-proxy.service

3、部署calico网络组件

calico,是一套开源的网络和网络安全方案,用于容器、虚拟机、宿主机之前的网络连接,由于calico.yaml文件内容较多,我放到了Gitee上面了,Gitee地址为:https://gitee.com/qinziteng/K8S/blob/master/YMAL/calico_1.26.yaml

1、执行apply

kubectl apply -f calico.yaml

2、查看创建的资源

kubectl get pods  -n kube-system

在这里插入图片描述

4、部署coreDNS解析组件

coreDNS,是一个DNS 服务,而 DNS 作为一种常见的服务发现手段,很多开源项目以及工程师都会使用 CoreDNS 为集群提供服务发现的功能,Kubernetes 就在集群中使用 CoreDNS 解决服务发现的问题。

1、创建coredns.yaml文件

cat > coredns.yaml << EOF
apiVersion: v1
kind: ServiceAccount
metadata:name: corednsnamespace: kube-systemlabels:kubernetes.io/cluster-service: "true"addonmanager.kubernetes.io/mode: Reconcile
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:labels:kubernetes.io/bootstrapping: rbac-defaultsaddonmanager.kubernetes.io/mode: Reconcilename: system:coredns
rules:
- apiGroups:- ""resources:- endpoints- services- pods- namespacesverbs:- list- watch
- apiGroups:- ""resources:- nodesverbs:- get
- apiGroups:- discovery.k8s.ioresources:- endpointslicesverbs:- list- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:annotations:rbac.authorization.kubernetes.io/autoupdate: "true"labels:kubernetes.io/bootstrapping: rbac-defaultsaddonmanager.kubernetes.io/mode: EnsureExistsname: system:coredns
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: system:coredns
subjects:
- kind: ServiceAccountname: corednsnamespace: kube-system
---
apiVersion: v1
kind: ConfigMap
metadata:name: corednsnamespace: kube-systemlabels:addonmanager.kubernetes.io/mode: EnsureExists
data:Corefile: |.:53 {errorshealth {lameduck 5s}readykubernetes cluster.local in-addr.arpa ip6.arpa {pods insecurefallthrough in-addr.arpa ip6.arpattl 30}prometheus :9153forward . /etc/resolv.conf {max_concurrent 1000}cache 30reloadloadbalance}
---
apiVersion: apps/v1
kind: Deployment
metadata:name: corednsnamespace: kube-systemlabels:k8s-app: kube-dnskubernetes.io/cluster-service: "true"addonmanager.kubernetes.io/mode: Reconcilekubernetes.io/name: "CoreDNS"
spec:replicas: 1strategy:type: RollingUpdaterollingUpdate:maxUnavailable: 1selector:matchLabels:k8s-app: kube-dnstemplate:metadata:labels:k8s-app: kube-dnsspec:securityContext:seccompProfile:type: RuntimeDefaultpriorityClassName: system-cluster-criticalserviceAccountName: corednsaffinity:podAntiAffinity:preferredDuringSchedulingIgnoredDuringExecution:- weight: 100podAffinityTerm:labelSelector:matchExpressions:- key: k8s-appoperator: Invalues: ["kube-dns"]topologyKey: kubernetes.io/hostnametolerations:- key: "CriticalAddonsOnly"operator: "Exists"nodeSelector:kubernetes.io/os: linuxcontainers:- name: corednsimage: coredns/coredns:1.9.3imagePullPolicy: IfNotPresentresources:limits:memory: 300Mirequests:cpu: 100mmemory: 70Miargs: [ "-conf", "/etc/coredns/Corefile" ]volumeMounts:- name: config-volumemountPath: /etc/corednsreadOnly: trueports:- containerPort: 53name: dnsprotocol: UDP- containerPort: 53name: dns-tcpprotocol: TCP- containerPort: 9153name: metricsprotocol: TCPlivenessProbe:httpGet:path: /healthport: 8080scheme: HTTPinitialDelaySeconds: 60timeoutSeconds: 5successThreshold: 1failureThreshold: 5readinessProbe:httpGet:path: /readyport: 8181scheme: HTTPsecurityContext:allowPrivilegeEscalation: falsecapabilities:add:- NET_BIND_SERVICEdrop:- allreadOnlyRootFilesystem: truednsPolicy: Defaultvolumes:- name: config-volumeconfigMap:name: corednsitems:- key: Corefilepath: Corefile
---
apiVersion: v1
kind: Service
metadata:name: kube-dnsnamespace: kube-systemannotations:prometheus.io/port: "9153"prometheus.io/scrape: "true"labels:k8s-app: kube-dnskubernetes.io/cluster-service: "true"addonmanager.kubernetes.io/mode: Reconcilekubernetes.io/name: "CoreDNS"
spec:selector:k8s-app: kube-dnsclusterIP: 10.255.0.2ports:- name: dnsport: 53protocol: UDP- name: dns-tcpport: 53protocol: TCP- name: metricsport: 9153protocol: TCP
EOF

2、执行apply

kubectl apply -f coredns.yaml 

3、查看创建的资源

kubectl get pods  -n kube-system

外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传

七、K8S集群功能测试

1、测试创建Pod、SVC资源

1、创建nginx.yaml文件

cat > nginx.yaml << EOF
apiVersion: v1 
kind: Pod
metadata:name: nginxnamespace: defaultlabels:app: nginx env: dev
spec:containers: - name: nginx ports:- containerPort: 80image: nginximagePullPolicy: IfNotPresent
---
apiVersion: v1
kind: Service
metadata:name: nginx-svc 
spec:type: NodePortports:- port: 80nodePort: 30080selector:app: nginxenv: dev
EOF

2、执行apply

kubectl apply -f nginx.yaml

3、查看创建资源

kubectl get pods,svc -o wide

外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传

4、浏览器访问

http://nodeIP:30080

在这里插入图片描述

如果Pod都已经运行起来了,但是页面访问不到,排查一下kube-proxy组件是否正常!

2、测试coreDNS解析

1、创建Pod,版本一定要是1.28,其他版本测试域名解析会有问题

kubectl run busybox --image busybox:1.28 --restart=Never --rm -it busybox -- sh

2、测试解析

nslookup kubernetes.default.svc.cluster.local
nslookup nginx-svc.default.svc.cluster.local

正常解析如下图:

外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传

本文来自互联网用户投稿,该文观点仅代表作者本人,不代表本站立场。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如若转载,请注明出处:http://www.rhkb.cn/news/413732.html

如若内容造成侵权/违法违规/事实不符,请联系长河编程网进行投诉反馈email:809451989@qq.com,一经查实,立即删除!

相关文章

JVM2-JVM组成、字节码文件、类的生命周期、类加载器

Java虚拟机的组成 Java虚拟机主要分为以下几个组成部分&#xff1a; 类加载子系统&#xff1a;核心组件类加载器&#xff0c;负责将字节码文件中的内容加载到内存中运行时数据区&#xff1a;JVM管理的内存&#xff0c;创建出来的对象、类的信息等内容都会放在这块区域中执行引…

RelativeLayout相对布局

activity_relative_layout.xml <?xml version"1.0" encoding"utf-8"?> <RelativeLayout xmlns:android"http://schemas.android.com/apk/res/android"android:layout_width"match_parent"android:layout_height"150dp…

QT QGraphicsView实现预览图片显示缩略图功能

QT QGraphicsView实现预览图片显示缩略图功能QT creator Qt5.15.2 头文件&#xff1a; #ifndef TGRAPHICSVIEW_H #define TGRAPHICSVIEW_H#include <QGraphicsView> #include <QMainWindow> #include <QObject> #include <QWidget>class TGraphicsVie…

Oracle 客户端 PL/SQL Developer 15.0.4 安装与使用

目录 官网下载与安装 切换中文与注册 连接Oracle数据库 tnsnames.ora 文件使用 Oracle 客户端 PL/SQL Developer 12.0.7 安装、数据导出、Oracle 执行/解释计划、for update。 官网下载与安装 1、官网&#xff1a;https://www.allroundautomations.com/products/pl-sql-d…

P01-何谓Java方法

P01-何谓Java方法 一、System.out.println()分析 二、剖析方法 谈到方法&#xff0c;我就突然想到了c函数&#xff1a; 其实&#xff1a;Java 方法和 C 函数在许多方面确实有类似之处&#xff0c;但它们也存在一些显著的差异。下面是它们的一些共同点和不同点&#xff1a; 共同…

DORIS - DORIS简介

前言 本博文基于DORIS的2.1.5版本。apache-doris-2.1.5-bin-x64.tar.gz 是什么&#xff1f; DORIS官网 Apache Doris 是一款基于 MPP 架构的高性能、实时的分析型数据库&#xff0c;以高效、简单、统一的特点被人们所熟知&#xff0c;仅需亚秒级响应时间即可返回海量数据下的…

【第0004页 · 递归】生成括号对

【前言】本文以及之后的一些题解都会陆续整理到目录中&#xff0c;若想了解全部题解整理&#xff0c;请看这里&#xff1a; 第0004页 生成括号对 今天这题有点难绷&#xff0c;从某种程度上来说应该是第二次写这个问题了&#xff0c;但还是卡住了&#xff0c;现在我们来看一下…

安防视频汇聚平台EasyCVR启动后无法访问登录页面是什么原因?

安防视频监控/视频集中存储/云存储/磁盘阵列EasyCVR平台基于云边端一体化架构&#xff0c;兼容性强、支持多协议接入&#xff0c;包括国标GB/T28181协议、部标JT808、GA/T1400协议、RTMP、RTSP/Onvif协议、海康Ehome、海康SDK、大华SDK、华为SDK、宇视SDK、乐橙SDK、萤石云SDK等…

设计模式之适配器模式:软件世界的桥梁建筑师

一、什么是适配器模式 适配器模式&#xff08;Adapter Pattern&#xff09;是一种结构型设计模式&#xff08;Structural Pattern&#xff09;&#xff0c;通过将类的接口转换为客户期望的另一个接口&#xff0c;适配器可以让不兼容的两个类一起协同工作。其核心思想是通过一个…

SQL 语言简明入门:从历史到实践

SQL&#xff08;Structured Query Language&#xff09;是数据库领域的核心语言。自20世纪70年代中期由IBM公司开发以来&#xff0c;SQL已经成为全球最广泛使用的数据库管理语言。 本文将以简洁明了的方式为您介绍SQL的历史、基本结构、核心语言组成以及其独特的特点和书写规则…

Cookie对象的缺陷与应对策略

Cookie对象的缺陷与应对策略 1. 安全性问题&#xff1a;Cookie是明文的2. 存储限制&#xff1a;浏览器对Cookie数量和大小有限制3. 性能影响&#xff1a;Cookie携带过多增加网络流量4. 数据类型限制&#xff1a;Cookie的value值只能是字符串 &#x1f496;The Begin&#x1f4…

华为2024 届秋招招聘——硬件技术工程师-电源方向-机试题(四套)(每套四十题)

华为 2024 届秋招——硬件-电源机试题&#xff08;四套&#xff09;&#xff08;每套四十题&#xff09; 岗位——硬件技术工程师 岗位意向——电源 真题题目分享&#xff0c;完整版带答案(有答案和解析&#xff0c;答案非官方&#xff0c;未仔细校正&#xff0c;仅供参考&am…

bbr 和 inflight 守恒的收敛原理

先看 bbr&#xff0c;以 2 条流 bw 收敛为例&#xff0c;微分方程组如下&#xff1a; { d x d t C ⋅ g ⋅ x g ⋅ x y − x d y d t C ⋅ g ⋅ y g ⋅ y x − y \begin{cases} \dfrac{dx}{dt}C\cdot\dfrac{g\cdot x}{g\cdot xy}-x\\\ \dfrac{dy}{dt}C\cdot\dfrac{g\cdot y…

Python酷库之旅-第三方库Pandas(113)

目录 一、用法精讲 496、pandas.DataFrame.kurtosis方法 496-1、语法 496-2、参数 496-3、功能 496-4、返回值 496-5、说明 496-6、用法 496-6-1、数据准备 496-6-2、代码示例 496-6-3、结果输出 497、pandas.DataFrame.max方法 497-1、语法 497-2、参数 497-3、…

element的el-date-picker组件实现只显示年月日时分,不显示秒

需求&#xff1a;使用element的el-date-picker组件&#xff0c;只显示时分&#xff0c;不消失秒 效果&#xff1a; 解决方法&#xff1a; <el-date-pickerv-model"ruleForm.startTime"type"datetime"placeholder"开始时间"format"yyyy-…

分支和循环(上)

目录 1. if语句 1.1 if ​1.2 else 1.3 分支中包含多条语句 1.4 嵌套if 1.5 悬空else问题 2. 关系操作符 3. 条件操作符 4. 逻辑操作符 4.1 逻辑取反操作符 4.2 逻辑与运算符 4.3 逻辑或运算符 4.4 连续:闰年的判断 4.5 短路 5. switch语句 5.1 if语句和switch…

企业级Mysql 集群技术部署

目录 1.1部署mysql 1.1.1 安装依赖性&#xff1a; 1.1.2 下载并解压源码包 1.1.3 源码编译安装mysql 1.1.4 部署mysql 2.mysql的主从复制 2.1 配置masters 2.2配置slave 2.3 延迟复制 2.4 慢查询日志 2.5并行复制 2.6 原理刨析 2. 7架构缺陷 3.半同步模式 3.1半同…

公务员面试(c语言)

1./ 描述 //公务员面试现场打分。有7位考官&#xff0c;从键盘输入若干组成绩&#xff0c;每组7个分数&#xff08;百分制&#xff09;&#xff0c;去掉一个最高分和一个最低分&#xff0c;输出每组的平均成绩。 //&#xff08;注&#xff1a;本题有多组输入&#xff09; //输入…

C语言:ASCII码表和字符操作

目录 目录 1. 引言 2. ASCII码表 2.1 控制字符 2.2 可显示字符 3. 例子 3.1 相关函数 3.2 打印能够显示的 ASCII码 3.3 字母大小写转换 3.4 数字转数字字符 1. 引言 因为计算机只是认识 0 和 1组成的一串串的二进制数字&#xff0c;为了将人类认识的文…

C++ | Leetcode C++题解之第385题迷你语法分析器

题目&#xff1a; 题解&#xff1a; class Solution { public:int index 0;NestedInteger deserialize(string s) {if (s[index] [) {index;NestedInteger ni;while (s[index] ! ]) {ni.add(deserialize(s));if (s[index] ,) {index;}}index;return ni;} else {bool negati…