K8S二进制安装与部署

一、安装部署步骤  

1.1 初始化配置

1.2 所有 node 节点部署docker引擎

1.3 准备cfssl证书生成工具

1.4 生成Etcd证书

1.5 部署 Master 组件

1.6 部署 Worker Node 组件

1.7 部署 CNI 网络组件-部署 flannel

1.8 部署 CoreDNS

 1.9 master02 节点部署

 1.10 负载均衡部署


一、安装部署步骤  

节点名称IP地址相关组件
master01192.168.133.80kube-apiserver kube-controller-manager kube-scheduler etcd
master02192.168.133.20kube-apiserver kube-controller-manager kube-scheduler etcd
node01192.168.133.18kubelet kube-proxy docker etcd
node02192.168.133.19kubelet kube-proxy docker etcd
master192.168.133.21nginx+keepalive
backup192.168.133.22nginx+keepalive


1.1 初始化配置

#关闭防火墙
systemctl stop firewalld
systemctl disable firewalld
iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X#关闭selinux
setenforce 0
sed -i 's/enforcing/disabled/' /etc/selinux/config#关闭swap
swapoff -a
sed -ri 's/.*swap.*/#&/' /etc/fstab #根据规划设置主机名
hostnamectl set-hostname master01
hostnamectl set-hostname node01
hostnamectl set-hostname node02#在master添加hosts
cat >> /etc/hosts << EOF
192.168.133.80 master01
192.168.133.20 master02
192.168.133.18 node01
192.168.133.19 node02
EOF#调整内核参数
cat > /etc/sysctl.d/k8s.conf << EOF
#开启网桥模式,可将网桥的流量传递给iptables链
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
#关闭ipv6协议
net.ipv6.conf.all.disable_ipv6=1
net.ipv4.ip_forward=1
EOFsysctl --system#时间同步
yum install ntpdate -y
ntpdate time.windows.com

1.2 所有 node 节点部署docker引擎

yum install -y yum-utils device-mapper-persistent-data lvm2 
yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo 
yum install -y docker-ce docker-ce-cli containerd.iosystemctl start docker.service
systemctl enable docker.service 

1.3 准备cfssl证书生成工具

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -O /usr/local/bin/cfssl
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -O /usr/local/bin/cfssljson
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -O /usr/local/bin/cfssl-certinfochmod +x /usr/local/bin/cfssl*

1.4 生成Etcd证书

mkdir /opt/k8s
cd /opt/k8s/

 (1)etcd-cert.sh

#!/bin/bash
#配置证书生成策略,让 CA 软件知道颁发有什么功能的证书,生成用来签发其他组件证书的根证书
cat > ca-config.json <<EOF
{"signing": {"default": {"expiry": "87600h"},"profiles": {"www": {"expiry": "87600h","usages": ["signing","key encipherment","server auth","client auth"]}}}
}
EOF#ca-config.json:可以定义多个 profiles,分别指定不同的过期时间、使用场景等参数;
#后续在签名证书时会使用某个 profile;此实例只有一个 www 模板。
#expiry:指定了证书的有效期,87600h 为10年,如果用默认值一年的话,证书到期后集群会立即宕掉
#signing:表示该证书可用于签名其它证书;生成的 ca.pem 证书中 CA=TRUE;
#key encipherment:表示使用非对称密钥加密,如 RSA 加密;
#server auth:表示client可以用该 CA 对 server 提供的证书进行验证;
#client auth:表示server可以用该 CA 对 client 提供的证书进行验证;
#注意标点符号,最后一个字段一般是没有逗号的。#-----------------------
#生成CA证书和私钥(根证书和私钥)
#特别说明: cfssl和openssl有一些区别,openssl需要先生成私钥,然后用私钥生成请求文件,最后生成签名的证书和私钥等,但是cfssl可以直接得到请求文件。
cat > ca-csr.json <<EOF
{"CN": "etcd","key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","L": "Beijing","ST": "Beijing"}]
}
EOF#CN:Common Name,浏览器使用该字段验证网站或机构是否合法,一般写的是域名 
#key:指定了加密算法,一般使用rsa(size:2048)
#C:Country,国家
#ST:State,州,省
#L:Locality,地区,城市
#O: Organization Name,组织名称,公司名称
#OU: Organization Unit Name,组织单位名称,公司部门cfssl gencert -initca ca-csr.json | cfssljson -bare ca#生成的文件:
#ca-key.pem:根证书私钥
#ca.pem:根证书
#ca.csr:根证书签发请求文件#cfssl gencert -initca <CSRJSON>:使用 CSRJSON 文件生成生成新的证书和私钥。如果不添加管道符号,会直接把所有证书内容输出到屏幕。
#注意:CSRJSON 文件用的是相对路径,所以 cfssl 的时候需要 csr 文件的路径下执行,也可以指定为绝对路径。
#cfssljson 将 cfssl 生成的证书(json格式)变为文件承载式证书,-bare 用于命名生成的证书文件。#-----------------------
#生成 etcd 服务器证书和私钥
cat > server-csr.json <<EOF
{"CN": "etcd","hosts": ["192.168.80.10","192.168.80.11","192.168.80.12"],"key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","L": "BeiJing","ST": "BeiJing"}]
}
EOF#hosts:将所有 etcd 集群节点添加到 host 列表,需要指定所有 etcd 集群的节点 ip 或主机名不能使用网段,新增 etcd 服务器需要重新签发证书。cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server#生成的文件:
#server.csr:服务器的证书请求文件
#server-key.pem:服务器的私钥
#server.pem:服务器的数字签名证书#-config:引用证书生成策略文件 ca-config.json
#-profile:指定证书生成策略文件中的的使用场景,比如 ca-config.json 中的 www

(2) etcd.sh

#!/bin/bash
#example: ./etcd.sh etcd01 192.168.80.10 etcd02=https://192.168.80.11:2380,etcd03=https://192.168.80.12:2380#创建etcd配置文件/opt/etcd/cfg/etcd
ETCD_NAME=$1
ETCD_IP=$2
ETCD_CLUSTER=$3WORK_DIR=/opt/etcdcat > $WORK_DIR/cfg/etcd  <<EOF
#[Member]
ETCD_NAME="${ETCD_NAME}"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://${ETCD_IP}:2380"
ETCD_LISTEN_CLIENT_URLS="https://${ETCD_IP}:2379"#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://${ETCD_IP}:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://${ETCD_IP}:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://${ETCD_IP}:2380,${ETCD_CLUSTER}"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF#Member:成员配置
#ETCD_NAME:节点名称,集群中唯一。成员名字,集群中必须具备唯一性,如etcd01
#ETCD_DATA_DIR:数据目录。指定节点的数据存储目录,这些数据包括节点ID,集群ID,集群初始化配置,Snapshot文件,若未指定-wal-dir,还会存储WAL文件;如果不指定会用缺省目录
#ETCD_LISTEN_PEER_URLS:集群通信监听地址。用于监听其他member发送信息的地址。ip为全0代表监听本机所有接口
#ETCD_LISTEN_CLIENT_URLS:客户端访问监听地址。用于监听etcd客户发送信息的地址。ip为全0代表监听本机所有接口#Clustering:集群配置
#ETCD_INITIAL_ADVERTISE_PEER_URLS:集群通告地址。其他member使用,其他member通过该地址与本member交互信息。一定要保证从其他member能可访问该地址。静态配置方式下,该参数的value一定要同时在--initial-cluster参数中存在
#ETCD_ADVERTISE_CLIENT_URLS:客户端通告地址。etcd客户端使用,客户端通过该地址与本member交互信息。一定要保证从客户侧能可访问该地址
#ETCD_INITIAL_CLUSTER:集群节点地址。本member使用。描述集群中所有节点的信息,本member根据此信息去联系其他member
#ETCD_INITIAL_CLUSTER_TOKEN:集群Token。用于区分不同集群。本地如有多个集群要设为不同
#ETCD_INITIAL_CLUSTER_STATE:加入集群的当前状态,new是新集群,existing表示加入已有集群。#创建etcd.service服务管理文件
cat > /usr/lib/systemd/system/etcd.service <<EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target[Service]
Type=notify
EnvironmentFile=${WORK_DIR}/cfg/etcd
ExecStart=${WORK_DIR}/bin/etcd \
--cert-file=${WORK_DIR}/ssl/server.pem \
--key-file=${WORK_DIR}/ssl/server-key.pem \
--trusted-ca-file=${WORK_DIR}/ssl/ca.pem \
--peer-cert-file=${WORK_DIR}/ssl/server.pem \
--peer-key-file=${WORK_DIR}/ssl/server-key.pem \
--peer-trusted-ca-file=${WORK_DIR}/ssl/ca.pem \
--logger=zap \
--enable-v2
Restart=on-failure
LimitNOFILE=65536[Install]
WantedBy=multi-user.target
EOF#--enable-v2:开启 etcd v2 API 接口。当前 flannel 版本不支持 etcd v3 通信
#--logger=zap:使用 zap 日志框架。zap.Logger 是go语言中相对日志库中性能最高的
#--peer开头的配置项用于指定集群内部TLS相关证书(peer 证书),这里全部都使用同一套证书认证
#不带--peer开头的的参数是指定 etcd 服务器TLS相关证书(server 证书),这里全部都使用同一套证书认证systemctl daemon-reload
systemctl enable etcd
systemctl restart etcd

 

chmod +x etcd-cert.sh etcd.sh
mkdir /opt/k8s/etcd-cert
mv etcd-cert.sh etcd-cert/
cd /opt/k8s/etcd-cert/
./etcd-cert.sh	

cd /opt/k8s/
tar zxvf etcd-v3.4.9-linux-amd64.tar.gz
mkdir -p /opt/etcd/{cfg,bin,ssl}
cd /opt/k8s/etcd-v3.4.9-linux-amd64/
mv etcd etcdctl /opt/etcd/bin/
cp /opt/k8s/etcd-cert/*.pem /opt/etcd/ssl/
cd /opt/k8s/
./etcd.sh etcd01 192.168.133.80 etcd02=https://192.168.133.18:2380,etcd03=https://192.168.133.19:2380
#可另外打开一个窗口查看etcd进程是否正常
ps -ef | grep etcd

 

scp -r /opt/etcd/ root@192.168.133.18:/opt/
scp -r /opt/etcd/ root@192.168.133.19:/opt/
scp /usr/lib/systemd/system/etcd.service root@192.168.133.18:/usr/lib/systemd/system/
scp /usr/lib/systemd/system/etcd.service root@192.168.133.19:/usr/lib/systemd/system/

在 node01 节点和node02 节点上分别操作

vim /opt/etcd/cfg/etcd
systemctl start etcd
systemctl enable etcd

 

 node01

 node02

 在master01检查etcd群集状态以及etcd集群成员列表

ETCDCTL_API=3   /opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints="https://192.168.133.80:2379,https://192.168.133.18:2379,https://192.168.133.19:2379" endpoint health --write-out=tableETCDCTL_API=3 /opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints="https://192.168.133.80:2379,https://192.168.133.18:2379,https://192.168.133.19:2379" endpoint status --write-out=tableETCDCTL_API=3 /opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints="https://192.168.133.80:2379,https://192.168.133.18:2379,https://192.168.133.19:2379" --write-out=table member list

1.5 部署 Master 组件

在 master01 节点上操作

#上传 master.zip 和 k8s-cert.sh 到 /opt/k8s 目录中,解压 master.zip 压缩包
cd /opt/k8s/
unzip master.zip
chmod +x *.sh

admin.sh 

#!/bin/bash
mkdir /root/.kube
KUBE_CONFIG="/root/.kube/config"
KUBE_APISERVER="https://192.168.133.80:6443"cd /opt/k8s/k8s-cert/kubectl config set-cluster kubernetes \--certificate-authority=/opt/kubernetes/ssl/ca.pem \--embed-certs=true \--server=${KUBE_APISERVER} \--kubeconfig=${KUBE_CONFIG}
kubectl config set-credentials cluster-admin \--client-certificate=./admin.pem \--client-key=./admin-key.pem \--embed-certs=true \--kubeconfig=${KUBE_CONFIG}
kubectl config set-context default \--cluster=kubernetes \--user=cluster-admin \--kubeconfig=${KUBE_CONFIG}
kubectl config use-context default --kubeconfig=${KUBE_CONFIG}

 apiserver.sh

#!/bin/bash
#example: apiserver.sh 192.168.133.80 https://192.168.133.80:2379,https://192.168.133.18:2379,https://192.168.133.19:2379
#创建 kube-apiserver 启动参数配置文件
MASTER_ADDRESS=$1
ETCD_SERVERS=$2cat >/opt/kubernetes/cfg/kube-apiserver <<EOF
KUBE_APISERVER_OPTS="--logtostderr=false  \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--etcd-servers=${ETCD_SERVERS} \\
--bind-address=${MASTER_ADDRESS} \\
--secure-port=6443 \\
--advertise-address=${MASTER_ADDRESS} \\
--allow-privileged=true \\
--service-cluster-ip-range=10.0.0.0/24 \\
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \\
--authorization-mode=RBAC,Node \\
--enable-bootstrap-token-auth=true \\
--token-auth-file=/opt/kubernetes/cfg/token.csv \\
--service-node-port-range=30000-50000 \\
--kubelet-client-certificate=/opt/kubernetes/ssl/apiserver.pem \\
--kubelet-client-key=/opt/kubernetes/ssl/apiserver-key.pem \\
--tls-cert-file=/opt/kubernetes/ssl/apiserver.pem  \\
--tls-private-key-file=/opt/kubernetes/ssl/apiserver-key.pem \\
--client-ca-file=/opt/kubernetes/ssl/ca.pem \\
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \\
--service-account-issuer=api \\
--service-account-signing-key-file=/opt/kubernetes/ssl/apiserver-key.pem \\
--etcd-cafile=/opt/etcd/ssl/ca.pem \\
--etcd-certfile=/opt/etcd/ssl/server.pem \\
--etcd-keyfile=/opt/etcd/ssl/server-key.pem \\
--requestheader-client-ca-file=/opt/kubernetes/ssl/ca.pem \\
--proxy-client-cert-file=/opt/kubernetes/ssl/apiserver.pem \\
--proxy-client-key-file=/opt/kubernetes/ssl/apiserver-key.pem \\
--requestheader-allowed-names=kubernetes \\
--requestheader-extra-headers-prefix=X-Remote-Extra- \\
--requestheader-group-headers=X-Remote-Group \\
--requestheader-username-headers=X-Remote-User \\
--enable-aggregator-routing=true \\
--audit-log-maxage=30 \\
--audit-log-maxbackup=3 \\
--audit-log-maxsize=100 \\
--audit-log-path=/opt/kubernetes/logs/k8s-audit.log"
EOF#--logtostderr=true:启用日志。输出日志到标准错误控制台,不输出到文件
#--v=4:日志等级。指定输出日志的级别,v=4为调试级别详细输出
#--etcd-servers:etcd集群地址。指定etcd服务器列表(格式://ip:port),逗号分隔
#--bind-address:监听地址。指定 HTTPS 安全接口的监听地址,默认值0.0.0.0
#--secure-port:https安全端口。指定 HTTPS 安全接口的监听端口,默认值6443
#--advertise-address:集群通告地址。通过该 ip 地址向集群其他节点公布 api server 的信息,必须能够被其他节点访问
#--allow-privileged=true:启用授权。允许拥有系统特权的容器运行,默认值false
#--service-cluster-ip-range:Service虚拟IP地址段。指定 Service Cluster IP 地址段
#--enable-admission-plugins:准入控制模块。kuberneres集群的准入控制机制,各控制模块以插件的形式依次生效,集群时必须包含ServiceAccount,运行在认证(Authentication)、授权(Authorization)之后,Admission Control是权限认证链上的最后一环, 对请求API资源对象进行修改和校验
#--authorization-mode:认证授权,启用RBAC授权和节点自管理。在安全端口使用RBAC,Node授权模式,未通过授权的请求拒绝,默认值AlwaysAllow。RBAC是用户通过角色与权限进行关联的模式;Node模式(节点授权)是一种特殊用途的授权模式,专门授权由kubelet发出的API请求,在进行认证时,先通过用户名、用户分组验证是否是集群中的Node节点,只有是Node节点的请求才能使用Node模式授权
#--enable-bootstrap-token-auth:启用TLS bootstrap机制。在apiserver上启用Bootstrap Token 认证
#--token-auth-file=/opt/kubernetes/cfg/token.csv:指定bootstrap token认证文件路径
#--service-node-port-range:指定 Service  NodePort 的端口范围,默认值30000-32767
#–-kubelet-client-xxx:apiserver访问kubelet客户端证书
#--tls-xxx-file:apiserver https证书
#1.20版本必须加的参数:–-service-account-issuer,–-service-account-signing-key-file
#--etcd-xxxfile:连接Etcd集群证书
#–-audit-log-xxx:审计日志
#启动聚合层相关配置:–requestheader-client-ca-file,–proxy-client-cert-file,–proxy-client-key-file,–requestheader-allowed-names,–requestheader-extra-headers-prefix,–requestheader-group-headers,–requestheader-username-headers,–enable-aggregator-routing#创建 kube-apiserver.service 服务管理文件
cat >/usr/lib/systemd/system/kube-apiserver.service <<EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-apiserver
ExecStart=/opt/kubernetes/bin/kube-apiserver \$KUBE_APISERVER_OPTS
Restart=on-failure[Install]
WantedBy=multi-user.target
EOFsystemctl daemon-reload
systemctl enable kube-apiserver
systemctl restart kube-apiserver

 controller-manager.sh

#!/bin/bash
##创建 kube-controller-manager 启动参数配置文件
MASTER_ADDRESS=$1cat >/opt/kubernetes/cfg/kube-controller-manager <<EOF
KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--leader-elect=true \\
--kubeconfig=/opt/kubernetes/cfg/kube-controller-manager.kubeconfig \\
--bind-address=127.0.0.1 \\
--allocate-node-cidrs=true \\
--cluster-cidr=10.244.0.0/16 \\
--service-cluster-ip-range=10.0.0.0/24 \\
--cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \\
--cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem  \\
--root-ca-file=/opt/kubernetes/ssl/ca.pem \\
--service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \\
--cluster-signing-duration=87600h0m0s"
EOF#––leader-elect:当该组件启动多个时,自动选举(HA)
#-–kubeconfig:连接 apiserver 用的配置文件,用于识别 k8s 集群
#--cluster-cidr=10.244.0.0/16:pod资源的网段,需与pod网络插件的值设置一致。通常,Flannel网络插件的默认为10.244.0.0/16,Calico插件的默认值为192.168.0.0/16
#--cluster-signing-cert-file/–-cluster-signing-key-file:自动为kubelet颁发证书的CA,与apiserver保持一致。指定签名的CA机构根证书,用来签名为 TLS BootStrapping 创建的证书和私钥
#--root-ca-file:指定根CA证书文件路径,用来对 kube-apiserver 证书进行校验,指定该参数后,才会在 Pod 容器的 ServiceAccount 中放置该 CA 证书文件
#--experimental-cluster-signing-duration:设置为 TLS BootStrapping 签署的证书有效时间为10年,默认为1年##生成kube-controller-manager证书
cd /opt/k8s/k8s-cert/
#创建证书请求文件
cat > kube-controller-manager-csr.json << EOF
{"CN": "system:kube-controller-manager","hosts": [],"key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","L": "BeiJing", "ST": "BeiJing","O": "system:masters","OU": "System"}]
}
EOF#生成证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager#生成kubeconfig文件
KUBE_CONFIG="/opt/kubernetes/cfg/kube-controller-manager.kubeconfig"
KUBE_APISERVER="https://192.168.133.80:6443"kubectl config set-cluster kubernetes \--certificate-authority=/opt/kubernetes/ssl/ca.pem \--embed-certs=true \--server=${KUBE_APISERVER} \--kubeconfig=${KUBE_CONFIG}
kubectl config set-credentials kube-controller-manager \--client-certificate=./kube-controller-manager.pem \--client-key=./kube-controller-manager-key.pem \--embed-certs=true \--kubeconfig=${KUBE_CONFIG}
kubectl config set-context default \--cluster=kubernetes \--user=kube-controller-manager \--kubeconfig=${KUBE_CONFIG}
kubectl config use-context default --kubeconfig=${KUBE_CONFIG}##创建 kube-controller-manager.service 服务管理文件
cat >/usr/lib/systemd/system/kube-controller-manager.service <<EOF
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-controller-manager
ExecStart=/opt/kubernetes/bin/kube-controller-manager \$KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure[Install]
WantedBy=multi-user.target
EOFsystemctl daemon-reload
systemctl enable kube-controller-manager
systemctl restart kube-controller-manager

scheduler.sh

#!/bin/bash
##创建 kube-scheduler 启动参数配置文件
MASTER_ADDRESS=$1cat >/opt/kubernetes/cfg/kube-scheduler <<EOF
KUBE_SCHEDULER_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--leader-elect=true \\
--kubeconfig=/opt/kubernetes/cfg/kube-scheduler.kubeconfig \\
--bind-address=127.0.0.1"
EOF#-–kubeconfig:连接 apiserver 用的配置文件,用于识别 k8s 集群
#--leader-elect=true:当该组件启动多个时,自动启动 leader 选举
#k8s中Controller-Manager和Scheduler的选主逻辑:k8s中的etcd是整个集群所有状态信息的存储,涉及数据的读写和多个etcd之间数据的同步,对数据的一致性要求严格,所以使用较复杂的 raft 算法来选择用于提交数据的主节点。而 apiserver 作为集群入口,本身是无状态的web服务器,多个 apiserver 服务之间直接负载请求并不需要做选主。Controller-Manager 和 Scheduler 作为任务类型的组件,比如 controller-manager 内置的 k8s 各种资源对象的控制器实时的 watch apiserver 获取对象最新的变化事件做期望状态和实际状态调整,调度器watch未绑定节点的pod做节点选择,显然多个这些任务同时工作是完全没有必要的,所以 controller-manager 和 scheduler 也是需要选主的,但是选主逻辑和 etcd 不一样的,这里只需要保证从多个 controller-manager 和 scheduler 之间选出一个 leader 进入工作状态即可,而无需考虑它们之间的数据一致和同步。##生成kube-scheduler证书
cd /opt/k8s/k8s-cert/
#创建证书请求文件
cat > kube-scheduler-csr.json << EOF
{"CN": "system:kube-scheduler","hosts": [],"key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","L": "BeiJing","ST": "BeiJing","O": "system:masters","OU": "System"}]
}
EOF#生成证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-scheduler-csr.json | cfssljson -bare kube-scheduler#生成kubeconfig文件
KUBE_CONFIG="/opt/kubernetes/cfg/kube-scheduler.kubeconfig"
KUBE_APISERVER="https://192.168.133.80:6443"kubectl config set-cluster kubernetes \--certificate-authority=/opt/kubernetes/ssl/ca.pem \--embed-certs=true \--server=${KUBE_APISERVER} \--kubeconfig=${KUBE_CONFIG}
kubectl config set-credentials kube-scheduler \--client-certificate=./kube-scheduler.pem \--client-key=./kube-scheduler-key.pem \--embed-certs=true \--kubeconfig=${KUBE_CONFIG}
kubectl config set-context default \--cluster=kubernetes \--user=kube-scheduler \--kubeconfig=${KUBE_CONFIG}
kubectl config use-context default --kubeconfig=${KUBE_CONFIG}##创建 kube-scheduler.service 服务管理文件
cat >/usr/lib/systemd/system/kube-scheduler.service <<EOF
[Unit]
Description=Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-scheduler
ExecStart=/opt/kubernetes/bin/kube-scheduler \$KUBE_SCHEDULER_OPTS
Restart=on-failure[Install]
WantedBy=multi-user.target
EOFsystemctl daemon-reload
systemctl enable kube-scheduler
systemctl restart kube-scheduler
#创建kubernetes工作目录
mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs}#创建用于生成CA证书、相关组件的证书和私钥的目录
mkdir /opt/k8s/k8s-cert
mv /opt/k8s/k8s-cert.sh /opt/k8s/k8s-cert
cd /opt/k8s/k8s-cert/
chmod +x *.sh
./k8s-cert.sh				#生成CA证书、相关组件的证书和私钥

 k8s-cert.sh    

#!/bin/bash
#配置证书生成策略,让 CA 软件知道颁发有什么功能的证书,生成用来签发其他组件证书的根证书
cat > ca-config.json <<EOF
{"signing": {"default": {"expiry": "87600h"},"profiles": {"kubernetes": {"expiry": "87600h","usages": ["signing","key encipherment","server auth","client auth"]}}}
}
EOF#生成CA证书和私钥(根证书和私钥)
cat > ca-csr.json <<EOF
{"CN": "kubernetes","key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","L": "Beijing","ST": "Beijing","O": "k8s","OU": "System"}]
}
EOFcfssl gencert -initca ca-csr.json | cfssljson -bare ca -#-----------------------
#生成 apiserver 的证书和私钥(apiserver和其它k8s组件通信使用)
#hosts中将所有可能作为 apiserver 的 ip 添加进去,后面 keepalived 使用的 VIP 也要加入
cat > apiserver-csr.json <<EOF
{"CN": "kubernetes","hosts": ["10.0.0.1","127.0.0.1","192.168.133.80",		#master01"192.168.133.20",		#master02"192.168.133.100",	#vip,后面 keepalived 使用"192.168.133.21",		#load balancer01(master)"192.168.133.22",		#load balancer02(backup)"kubernetes","kubernetes.default","kubernetes.default.svc","kubernetes.default.svc.cluster","kubernetes.default.svc.cluster.local"],"key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","L": "BeiJing","ST": "BeiJing","O": "k8s","OU": "System"}]
}
EOFcfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes apiserver-csr.json | cfssljson -bare apiserver#-----------------------
#生成 kubectl 连接集群的证书和私钥,具有admin权限
cat > admin-csr.json <<EOF
{"CN": "admin","hosts": [],"key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","L": "BeiJing","ST": "BeiJing","O": "system:masters","OU": "System"}]
}
EOFcfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes admin-csr.json | cfssljson -bare admin#-----------------------
#生成 kube-proxy 的证书和私钥
cat > kube-proxy-csr.json <<EOF
{"CN": "system:kube-proxy","hosts": [],"key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","L": "BeiJing","ST": "BeiJing","O": "k8s","OU": "System"}]
}
EOFcfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-proxy-csr.json | cfssljson -bare kube-proxy

 

cp ca*pem apiserver*pem /opt/kubernetes/ssl/
cd /opt/k8s/
tar zxvf kubernetes-server-linux-amd64.tar.gz
#复制master组件的关键命令文件到 kubernetes工作目录的 bin 子目录中
cd /opt/k8s/kubernetes/server/bin
cp kube-apiserver kubectl kube-controller-manager kube-scheduler /opt/kubernetes/bin/
ln -s /opt/kubernetes/bin/* /usr/local/bin/
#创建 bootstrap token 认证文件,apiserver 启动时会调用,然后就相当于在集群内创建了一个这个用户,接下来就可以用 RBAC 给他授权
cd /opt/k8s/
vim token.sh
#!/bin/bash
#获取随机数前16个字节内容,以十六进制格式输出,并删除其中空格
BOOTSTRAP_TOKEN=$(head -c 16 /dev/urandom | od -An -t x | tr -d ' ') 
#生成 token.csv 文件,按照 Token序列号,用户名,UID,用户组 的格式生成
cat > /opt/kubernetes/cfg/token.csv <<EOF
${BOOTSTRAP_TOKEN},kubelet-bootstrap,10001,"system:kubelet-bootstrap"
EOFchmod +x token.sh
./token.shcat /opt/kubernetes/cfg/token.csv

#二进制文件、token、证书都准备好后,开启 apiserver 服务
cd /opt/k8s/
./apiserver.sh 192.168.133.80 https://192.168.133.80:2379,https://192.168.133.18:2379,https://192.168.133.19:2379#检查进程是否启动成功
ps aux | grep kube-apiservernetstat -natp | grep 6443   #安全端口6443用于接收HTTPS请求,用于基于Token文件或客户端证书等认证#启动 scheduler 服务
cd /opt/k8s/
./scheduler.sh
ps aux | grep kube-scheduler#启动 controller-manager 服务
./controller-manager.sh
ps aux | grep kube-controller-manager#生成kubectl连接集群的kubeconfig文件
./admin.sh#通过kubectl工具查看当前集群组件状态
kubectl get cs

1.6 部署 Worker Node 组件

在所有 node 节点上操作

#创建kubernetes工作目录
mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs}
#上传 node.zip 到 /opt 目录中,解压 node.zip 压缩包,获得kubelet.sh、proxy.sh
cd /opt/
unzip node.zip
chmod +x kubelet.sh proxy.sh

kubelet.sh

#!/bin/bashNODE_ADDRESS=$1
DNS_SERVER_IP=${2:-"10.0.0.2"}#创建 kubelet 启动参数配置文件
cat >/opt/kubernetes/cfg/kubelet <<EOF
KUBELET_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--hostname-override=${NODE_ADDRESS} \\
--network-plugin=cni \\
--kubeconfig=/opt/kubernetes/cfg/kubelet.kubeconfig \\
--bootstrap-kubeconfig=/opt/kubernetes/cfg/bootstrap.kubeconfig \\
--config=/opt/kubernetes/cfg/kubelet.config \\
--cert-dir=/opt/kubernetes/ssl \\
--pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google-containers/pause-amd64:3.0"
EOF#--hostname-override:指定kubelet节点在集群中显示的主机名或IP地址,默认使用主机hostname;kube-proxy和kubelet的此项参数设置必须完全一致
#--network-plugin:启用CNI
#--kubeconfig:指定kubelet.kubeconfig文件位置,当前为空路径,会自动生成,用于如何连接到apiserver,里面含有kubelet证书,master授权完成后会在node节点上生成 kubelet.kubeconfig 文件
#--bootstrap-kubeconfig:指定连接 apiserver 的 bootstrap.kubeconfig 文件
#--config:指定kubelet配置文件的路径,启动kubelet时将从此文件加载其配置
#--cert-dir:指定master颁发的kubelet证书生成目录
#--pod-infra-container-image:指定Pod基础容器(Pause容器)的镜像。Pod启动的时候都会启动一个这样的容器,每个pod之间相互通信需要Pause的支持,启动Pause需要Pause基础镜像#----------------------
#创建kubelet配置文件(该文件实际上就是一个yml文件,语法非常严格,不能出现tab键,冒号后面必须要有空格,每行结尾也不能有空格)
cat >/opt/kubernetes/cfg/kubelet.config <<EOF
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: ${NODE_ADDRESS}
port: 10250
readOnlyPort: 10255
cgroupDriver: cgroupfs
clusterDNS:
- ${DNS_SERVER_IP} 
clusterDomain: cluster.local
failSwapOn: false
authentication:anonymous:enabled: true
EOF#PS:当命令行参数与此配置文件(kubelet.config)有相同的值时,就会覆盖配置文件中的该值。#----------------------
#创建 kubelet.service 服务管理文件
cat >/usr/lib/systemd/system/kubelet.service <<EOF
[Unit]
Description=Kubernetes Kubelet
After=docker.service
Requires=docker.service[Service]
EnvironmentFile=/opt/kubernetes/cfg/kubelet
ExecStart=/opt/kubernetes/bin/kubelet \$KUBELET_OPTS
Restart=on-failure
KillMode=process[Install]
WantedBy=multi-user.target
EOFsystemctl daemon-reload
systemctl enable kubelet
systemctl restart kubelet

 proxy.sh

#!/bin/bashNODE_ADDRESS=$1#创建 kube-proxy 启动参数配置文件
cat >/opt/kubernetes/cfg/kube-proxy <<EOF
KUBE_PROXY_OPTS="--logtostderr=true \\
--v=4 \\
--hostname-override=${NODE_ADDRESS} \\
--cluster-cidr=172.17.0.0/16 \\
--proxy-mode=ipvs \\
--kubeconfig=/opt/kubernetes/cfg/kube-proxy.kubeconfig"
EOF#--hostnameOverride: 参数值必须与 kubelet 的值一致,否则 kube-proxy 启动后会找不到该 Node,从而不会创建任何 ipvs 规则
#--cluster-cidr:指定 Pod 网络使用的聚合网段,Pod 使用的网段和 apiserver 中指定的 service 的 cluster ip 网段不是同一个网段。 kube-proxy 根据 --cluster-cidr 判断集群内部和外部流量,指定 --cluster-cidr 选项后 kube-proxy 才会对访问 Service IP 的请求做 SNAT,即来自非 Pod 网络的流量被当成外部流量,访问 Service 时需要做 SNAT。
#--proxy-mode:指定流量调度模式为ipvs模式,可添加--ipvs-scheduler选项指定ipvs调度算法(rr|wrr|lc|wlc|lblc|lblcr|dh|sh|sed|nq)
#--kubeconfig: 指定连接 apiserver 的 kubeconfig 文件	#----------------------
#创建 kube-proxy.service 服务管理文件
cat >/usr/lib/systemd/system/kube-proxy.service <<EOF
[Unit]
Description=Kubernetes Proxy
After=network.target[Service]
EnvironmentFile=-/opt/kubernetes/cfg/kube-proxy
ExecStart=/opt/kubernetes/bin/kube-proxy \$KUBE_PROXY_OPTS
Restart=on-failure[Install]
WantedBy=multi-user.target
EOFsystemctl daemon-reload
systemctl enable kube-proxy
systemctl restart kube-proxy

在 master01 节点上操作
 

#把 kubelet、kube-proxy 拷贝到 node 节点
cd /opt/k8s/kubernetes/server/bin
scp kubelet kube-proxy root@192.168.133.18:/opt/kubernetes/bin/
scp kubelet kube-proxy root@192.168.133.19:/opt/kubernetes/bin/

 

mkdir /opt/k8s/kubeconfig
cd /opt/k8s/kubeconfig
chmod +x kubeconfig.sh
./kubeconfig.sh 192.168.133.80 /opt/k8s/k8s-cert/

 kubeconfig.sh

#!/bin/bash
#example: kubeconfig 192.168.133.80 /opt/k8s/k8s-cert/
#创建bootstrap.kubeconfig文件
#该文件中内置了 token.csv 中用户的 Token,以及 apiserver CA 证书;kubelet 首次启动会加载此文件,使用 apiserver CA 证书建立与 apiserver 的 TLS 通讯,使用其中的用户 Token 作为身份标识向 apiserver 发起 CSR 请求BOOTSTRAP_TOKEN=$(awk -F ',' '{print $1}' /opt/kubernetes/cfg/token.csv)
APISERVER=$1
SSL_DIR=$2export KUBE_APISERVER="https://$APISERVER:6443"# 设置集群参数
kubectl config set-cluster kubernetes \--certificate-authority=$SSL_DIR/ca.pem \--embed-certs=true \--server=${KUBE_APISERVER} \--kubeconfig=bootstrap.kubeconfig
#--embed-certs=true:表示将ca.pem证书写入到生成的bootstrap.kubeconfig文件中# 设置客户端认证参数,kubelet 使用 bootstrap token 认证
kubectl config set-credentials kubelet-bootstrap \--token=${BOOTSTRAP_TOKEN} \--kubeconfig=bootstrap.kubeconfig# 设置上下文参数
kubectl config set-context default \--cluster=kubernetes \--user=kubelet-bootstrap \--kubeconfig=bootstrap.kubeconfig# 使用上下文参数生成 bootstrap.kubeconfig 文件
kubectl config use-context default --kubeconfig=bootstrap.kubeconfig#----------------------#创建kube-proxy.kubeconfig文件
# 设置集群参数
kubectl config set-cluster kubernetes \--certificate-authority=$SSL_DIR/ca.pem \--embed-certs=true \--server=${KUBE_APISERVER} \--kubeconfig=kube-proxy.kubeconfig# 设置客户端认证参数,kube-proxy 使用 TLS 证书认证
kubectl config set-credentials kube-proxy \--client-certificate=$SSL_DIR/kube-proxy.pem \--client-key=$SSL_DIR/kube-proxy-key.pem \--embed-certs=true \--kubeconfig=kube-proxy.kubeconfig# 设置上下文参数
kubectl config set-context default \--cluster=kubernetes \--user=kube-proxy \--kubeconfig=kube-proxy.kubeconfig# 使用上下文参数生成 kube-proxy.kubeconfig 文件
kubectl config use-context default --kubeconfig=kube-proxy.kubeconfig

 

#把配置文件 bootstrap.kubeconfig、kube-proxy.kubeconfig 拷贝到 node 节点
scp bootstrap.kubeconfig kube-proxy.kubeconfig root@192.168.133.18:/opt/kubernetes/cfg/
scp bootstrap.kubeconfig kube-proxy.kubeconfig root@192.168.133.19:/opt/kubernetes/cfg/#RBAC授权,使用户 kubelet-bootstrap 能够有权限发起 CSR 请求证书
kubectl create clusterrolebinding kubelet-bootstrap --clusterrole=system:node-bootstrapper --user=kubelet-bootstrap若执行失败,可先给kubectl绑定默认cluster-admin管理员集群角色,授权集群操作权限
kubectl create clusterrolebinding cluster-system-anonymous --clusterrole=cluster-admin --user=system:anonymous

在 node01 节点上操作
 

#启动 kubelet 服务
cd /opt/
./kubelet.sh 192.168.133.18
ps aux | grep kubelet

在 master01 节点上操作,通过 CSR 请求
 

#检查到 node01 节点的 kubelet 发起的 CSR 请求,Pending 表示等待集群给该节点签发证书
kubectl get csr

kubectl certificate approve node-csr-K84Eomk2ahAzGHNu0QC7lTb0oIKC7PScyeCJFdcrXqs

 在 node01 节点上操作

#加载 ip_vs 模块
for i in $(ls /usr/lib/modules/$(uname -r)/kernel/net/netfilter/ipvs|grep -o "^[^.]*");do echo $i; /sbin/modinfo -F filename $i >/dev/null 2>&1 && /sbin/modprobe $i;done#启动proxy服务
cd /opt/
./proxy.sh 192.168.133.18
ps aux | grep kube-proxy

同理在node2上操作 

1.7 部署 CNI 网络组件-部署 flannel

在 node01 节点和node02 节点上操作

#上传 cni-plugins-linux-amd64-v0.8.6.tgz 和 flannel.tar 到 /opt 目录中
cd /opt/
docker load -i flannel.tarmkdir /opt/cni/bin
tar zxvf cni-plugins-linux-amd64-v0.8.6.tgz -C /opt/cni/bin

在 master01 节点上操作
 

#上传 kube-flannel.yml 文件到 /opt/k8s 目录中,部署 CNI 网络
cd /opt/k8s
kubectl apply -f kube-flannel.yml 
kubectl get pods -n kube-system
kubectl get nodes

 kube-flannel.yml 

---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:name: psp.flannel.unprivilegedannotations:seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/defaultseccomp.security.alpha.kubernetes.io/defaultProfileName: docker/defaultapparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/defaultapparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
spec:privileged: falsevolumes:- configMap- secret- emptyDir- hostPathallowedHostPaths:- pathPrefix: "/etc/cni/net.d"- pathPrefix: "/etc/kube-flannel"- pathPrefix: "/run/flannel"readOnlyRootFilesystem: false# Users and groupsrunAsUser:rule: RunAsAnysupplementalGroups:rule: RunAsAnyfsGroup:rule: RunAsAny# Privilege EscalationallowPrivilegeEscalation: falsedefaultAllowPrivilegeEscalation: false# CapabilitiesallowedCapabilities: ['NET_ADMIN', 'NET_RAW']defaultAddCapabilities: []requiredDropCapabilities: []# Host namespaceshostPID: falsehostIPC: falsehostNetwork: truehostPorts:- min: 0max: 65535# SELinuxseLinux:# SELinux is unused in CaaSPrule: 'RunAsAny'
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:name: flannel
rules:
- apiGroups: ['extensions']resources: ['podsecuritypolicies']verbs: ['use']resourceNames: ['psp.flannel.unprivileged']
- apiGroups:- ""resources:- podsverbs:- get
- apiGroups:- ""resources:- nodesverbs:- list- watch
- apiGroups:- ""resources:- nodes/statusverbs:- patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:name: flannel
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: flannel
subjects:
- kind: ServiceAccountname: flannelnamespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:name: flannelnamespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:name: kube-flannel-cfgnamespace: kube-systemlabels:tier: nodeapp: flannel
data:cni-conf.json: |{"name": "cbr0","cniVersion": "0.3.1","plugins": [{"type": "flannel","delegate": {"hairpinMode": true,"isDefaultGateway": true}},{"type": "portmap","capabilities": {"portMappings": true}}]}net-conf.json: |{"Network": "10.244.0.0/16","Backend": {"Type": "vxlan"}}
---
apiVersion: apps/v1
kind: DaemonSet
metadata:name: kube-flannel-dsnamespace: kube-systemlabels:tier: nodeapp: flannel
spec:selector:matchLabels:app: flanneltemplate:metadata:labels:tier: nodeapp: flannelspec:affinity:nodeAffinity:requiredDuringSchedulingIgnoredDuringExecution:nodeSelectorTerms:- matchExpressions:- key: kubernetes.io/osoperator: Invalues:- linuxhostNetwork: truepriorityClassName: system-node-criticaltolerations:- operator: Existseffect: NoScheduleserviceAccountName: flannelinitContainers:- name: install-cniimage: quay.io/coreos/flannel:v0.14.0command:- cpargs:- -f- /etc/kube-flannel/cni-conf.json- /etc/cni/net.d/10-flannel.conflistvolumeMounts:- name: cnimountPath: /etc/cni/net.d- name: flannel-cfgmountPath: /etc/kube-flannel/containers:- name: kube-flannelimage: quay.io/coreos/flannel:v0.14.0command:- /opt/bin/flanneldargs:- --ip-masq- --kube-subnet-mgrresources:requests:cpu: "100m"memory: "50Mi"limits:cpu: "100m"memory: "50Mi"securityContext:privileged: falsecapabilities:add: ["NET_ADMIN", "NET_RAW"]env:- name: POD_NAMEvalueFrom:fieldRef:fieldPath: metadata.name- name: POD_NAMESPACEvalueFrom:fieldRef:fieldPath: metadata.namespacevolumeMounts:- name: runmountPath: /run/flannel- name: flannel-cfgmountPath: /etc/kube-flannel/volumes:- name: runhostPath:path: /run/flannel- name: cnihostPath:path: /etc/cni/net.d- name: flannel-cfgconfigMap:name: kube-flannel-cfg

1.8 部署 CoreDNS

 在所有 node 节点上操作

#上传 coredns.tar 到 /opt 目录中
cd /opt
docker load -i coredns.tar

 在 master01 节点上操作

#上传 coredns.yaml 文件到 /opt/k8s 目录中,部署 CoreDNS 
cd /opt/k8s
kubectl apply -f coredns.yamlkubectl get pods -n kube-system 
kubectl run -it --rm dns-test --image=busybox:1.28.4 sh
/ # nslookup kubernetes

 coredns.yaml

# __MACHINE_GENERATED_WARNING__apiVersion: v1
kind: ServiceAccount
metadata:name: corednsnamespace: kube-systemlabels:kubernetes.io/cluster-service: "true"addonmanager.kubernetes.io/mode: Reconcile
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:labels:kubernetes.io/bootstrapping: rbac-defaultsaddonmanager.kubernetes.io/mode: Reconcilename: system:coredns
rules:
- apiGroups:- ""resources:- endpoints- services- pods- namespacesverbs:- list- watch
- apiGroups:- ""resources:- nodesverbs:- get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:annotations:rbac.authorization.kubernetes.io/autoupdate: "true"labels:kubernetes.io/bootstrapping: rbac-defaultsaddonmanager.kubernetes.io/mode: EnsureExistsname: system:coredns
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: system:coredns
subjects:
- kind: ServiceAccountname: corednsnamespace: kube-system
---
apiVersion: v1
kind: ConfigMap
metadata:name: corednsnamespace: kube-systemlabels:addonmanager.kubernetes.io/mode: EnsureExists
data:Corefile: |.:53 {errorshealth {lameduck 5s}readykubernetes cluster.local in-addr.arpa ip6.arpa {pods insecurefallthrough in-addr.arpa ip6.arpattl 30}prometheus :9153forward . /etc/resolv.conf {max_concurrent 1000}cache 30loopreloadloadbalance}
---
apiVersion: apps/v1
kind: Deployment
metadata:name: corednsnamespace: kube-systemlabels:k8s-app: kube-dnskubernetes.io/cluster-service: "true"addonmanager.kubernetes.io/mode: Reconcilekubernetes.io/name: "CoreDNS"
spec:# replicas: not specified here:# 1. In order to make Addon Manager do not reconcile this replicas parameter.# 2. Default is 1.# 3. Will be tuned in real time if DNS horizontal auto-scaling is turned on.strategy:type: RollingUpdaterollingUpdate:maxUnavailable: 1selector:matchLabels:k8s-app: kube-dnstemplate:metadata:labels:k8s-app: kube-dnsspec:securityContext:seccompProfile:type: RuntimeDefaultpriorityClassName: system-cluster-criticalserviceAccountName: corednsaffinity:podAntiAffinity:preferredDuringSchedulingIgnoredDuringExecution:- weight: 100podAffinityTerm:labelSelector:matchExpressions:- key: k8s-appoperator: Invalues: ["kube-dns"]topologyKey: kubernetes.io/hostnametolerations:- key: "CriticalAddonsOnly"operator: "Exists"nodeSelector:kubernetes.io/os: linuxcontainers:- name: corednsimage: k8s.gcr.io/coredns:1.7.0imagePullPolicy: IfNotPresentresources:limits:memory: 170Mirequests:cpu: 100mmemory: 70Miargs: [ "-conf", "/etc/coredns/Corefile" ]volumeMounts:- name: config-volumemountPath: /etc/corednsreadOnly: trueports:- containerPort: 53name: dnsprotocol: UDP- containerPort: 53name: dns-tcpprotocol: TCP- containerPort: 9153name: metricsprotocol: TCPlivenessProbe:httpGet:path: /healthport: 8080scheme: HTTPinitialDelaySeconds: 60timeoutSeconds: 5successThreshold: 1failureThreshold: 5readinessProbe:httpGet:path: /readyport: 8181scheme: HTTPsecurityContext:allowPrivilegeEscalation: falsecapabilities:add:- NET_BIND_SERVICEdrop:- allreadOnlyRootFilesystem: truednsPolicy: Defaultvolumes:- name: config-volumeconfigMap:name: corednsitems:- key: Corefilepath: Corefile
---
apiVersion: v1
kind: Service
metadata:name: kube-dnsnamespace: kube-systemannotations:prometheus.io/port: "9153"prometheus.io/scrape: "true"labels:k8s-app: kube-dnskubernetes.io/cluster-service: "true"addonmanager.kubernetes.io/mode: Reconcilekubernetes.io/name: "CoreDNS"
spec:selector:k8s-app: kube-dnsclusterIP: 10.0.0.2ports:- name: dnsport: 53protocol: UDP- name: dns-tcpport: 53protocol: TCP- name: metricsport: 9153protocol: TCP

注:如果出现以下报错
[root@master01 k8s]# kubectl run -it  --image=busybox:1.28.4 sh
If you don't see a command prompt, try pressing enter.
Error attaching, falling back to logs: unable to upgrade connection: Forbidden (user=system:anonymous, verb=create, resource=nodes, subresource=proxy)
Error from server (Forbidden): Forbidden (user=system:anonymous, verb=get, resource=nodes, subresource=proxy) ( pods/log sh)

需要添加 rbac的权限  直接使用kubectl绑定  clusteradmin 管理员集群角色  授权操作权限

[root@master01 k8s]# kubectl create clusterrolebinding cluster-system-anonymous --clusterrole=cluster-admin --user=system:anonymous

 1.9 master02 节点部署

scp -r /opt/etcd/ root@192.168.133.20:/opt/
scp -r /opt/kubernetes/ root@192.168.133.20:/opt
scp -r /root/.kube root@192.168.133.20:/root
scp /usr/lib/systemd/system/{kube-apiserver,kube-controller-manager,kube-scheduler}.service root@192.168.133.20:/usr/lib/systemd/system/

修改配置文件kube-apiserver中的IP
 

vim /opt/kubernetes/cfg/kube-apiserver

在 master02 节点上启动各服务并设置开机自启
 

systemctl start kube-apiserver.service
systemctl enable kube-apiserver.service
systemctl start kube-controller-manager.service
systemctl enable kube-controller-manager.service
systemctl start kube-scheduler.service
systemctl enable kube-scheduler.service

查看node节点状态
 

ln -s /opt/kubernetes/bin/* /usr/local/bin/
kubectl get nodes
kubectl get nodes -o wide

 1.10 负载均衡部署

在master、backup节点上操作

cat > /etc/yum.repos.d/nginx.repo << 'EOF'
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/7/$basearch/
gpgcheck=0
EOFyum install nginx -y
vim /etc/nginx/nginx.conf

修改nginx配置文件

stream {log_format  main  '$remote_addr $upstream_addr - [$time_local] $status $upstream_bytes_sent';access_log  /var/log/nginx/k8s-access.log  main;upstream k8s-apiserver {server 192.168.133.80:6443;server 192.168.133.20:6443;}server {listen 6443;proxy_pass k8s-apiserver;}
}
//检查配置文件语法
nginx -t   //启动nginx服务,查看已监听6443端口
systemctl start nginx
systemctl enable nginx
netstat -natp | grep nginx //部署keepalived服务
yum install keepalived -yvim /etc/keepalived/keepalived.conf

修改keepalived配置文件

! Configuration File for keepalivedglobal_defs {# 接收邮件地址notification_email {acassen@firewall.locfailover@firewall.locsysadmin@firewall.loc}# 邮件发送地址notification_email_from Alexandre.Cassen@firewall.locsmtp_server 127.0.0.1smtp_connect_timeout 30router_id NGINX_MASTER	#master节点的为 NGINX_MASTER,backup节点的为 NGINX_BACKUP
}#添加一个周期性执行的脚本
vrrp_script check_nginx {script "/etc/nginx/check_nginx.sh"	#指定检查nginx存活的脚本路径
}vrrp_instance VI_1 {state MASTER			#master节点的为 MASTER,backup节点的为 BACKUPinterface ens33			#指定网卡名称 ens33virtual_router_id 51	#指定vrid,两个节点要一致priority 100			#master节点的为 100,backup节点的为 90advert_int 1authentication {auth_type PASSauth_pass 1111}virtual_ipaddress {192.168.10.100/24	#指定 VIP}track_script {check_nginx			#指定vrrp_script配置的脚本}
}

创建nginx状态检查脚本

#!/bin/bash
#egrep -cv "grep|$$" 用于过滤掉包含grep 或者 $$ 表示的当前Shell进程ID,即脚本运行的当前进程ID号
count=$(ps -ef | grep nginx | egrep -cv "grep|$$")if [ "$count" -eq 0 ];thensystemctl stop keepalived
fi
chmod +x /etc/nginx/check_nginx.sh
systemctl start keepalived
systemctl enable keepalived
ip a				#查看VIP是否生成

修改node节点上的bootstrap.kubeconfig,kubelet.kubeconfig配置文件为VIP
 

cd /opt/kubernetes/cfg/
vim bootstrap.kubeconfig 
server: https://192.168.133.100:6443vim kubelet.kubeconfig
server: https://192.168.133.100:6443vim kube-proxy.kubeconfig
server: https://192.168.133.100:6443//重启kubelet和kube-proxy服务
systemctl restart kubelet.service 
systemctl restart kube-proxy.service

在master 上查看

netstat -natp | grep nginx

在 master01 节点上操作

//测试创建pod
kubectl run nginx --image=nginx//查看Pod的状态信息
kubectl get podkubectl get podkubectl get pod -o wide//在对应网段的node节点上操作,可以直接使用浏览器或者curl命令访问
curl 10.244.1.4//这时在master01节点上查看nginx日志
kubectl logs nginx

1.11 部署 Dashboard 

在node节点导入

在 master01 节点上操作

cd /opt/k8s
vim recommended.yaml
# Copyright 2017 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.apiVersion: v1
kind: Namespace
metadata:name: kubernetes-dashboard---apiVersion: v1
kind: ServiceAccount
metadata:labels:k8s-app: kubernetes-dashboardname: kubernetes-dashboardnamespace: kubernetes-dashboard---kind: Service
apiVersion: v1
metadata:labels:k8s-app: kubernetes-dashboardname: kubernetes-dashboardnamespace: kubernetes-dashboard
spec:ports:- port: 443targetPort: 8443nodePort: 30001type: NodePortselector:k8s-app: kubernetes-dashboard---apiVersion: v1
kind: Secret
metadata:labels:k8s-app: kubernetes-dashboardname: kubernetes-dashboard-certsnamespace: kubernetes-dashboard
type: Opaque---apiVersion: v1
kind: Secret
metadata:labels:k8s-app: kubernetes-dashboardname: kubernetes-dashboard-csrfnamespace: kubernetes-dashboard
type: Opaque
data:csrf: ""---apiVersion: v1
kind: Secret
metadata:labels:k8s-app: kubernetes-dashboardname: kubernetes-dashboard-key-holdernamespace: kubernetes-dashboard
type: Opaque---kind: ConfigMap
apiVersion: v1
metadata:labels:k8s-app: kubernetes-dashboardname: kubernetes-dashboard-settingsnamespace: kubernetes-dashboard---kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:labels:k8s-app: kubernetes-dashboardname: kubernetes-dashboardnamespace: kubernetes-dashboard
rules:# Allow Dashboard to get, update and delete Dashboard exclusive secrets.- apiGroups: [""]resources: ["secrets"]resourceNames: ["kubernetes-dashboard-key-holder", "kubernetes-dashboard-certs", "kubernetes-dashboard-csrf"]verbs: ["get", "update", "delete"]# Allow Dashboard to get and update 'kubernetes-dashboard-settings' config map.- apiGroups: [""]resources: ["configmaps"]resourceNames: ["kubernetes-dashboard-settings"]verbs: ["get", "update"]# Allow Dashboard to get metrics.- apiGroups: [""]resources: ["services"]resourceNames: ["heapster", "dashboard-metrics-scraper"]verbs: ["proxy"]- apiGroups: [""]resources: ["services/proxy"]resourceNames: ["heapster", "http:heapster:", "https:heapster:", "dashboard-metrics-scraper", "http:dashboard-metrics-scraper"]verbs: ["get"]---kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:labels:k8s-app: kubernetes-dashboardname: kubernetes-dashboard
rules:# Allow Metrics Scraper to get metrics from the Metrics server- apiGroups: ["metrics.k8s.io"]resources: ["pods", "nodes"]verbs: ["get", "list", "watch"]---apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:labels:k8s-app: kubernetes-dashboardname: kubernetes-dashboardnamespace: kubernetes-dashboard
roleRef:apiGroup: rbac.authorization.k8s.iokind: Rolename: kubernetes-dashboard
subjects:- kind: ServiceAccountname: kubernetes-dashboardnamespace: kubernetes-dashboard---apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:name: kubernetes-dashboard
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: kubernetes-dashboard
subjects:- kind: ServiceAccountname: kubernetes-dashboardnamespace: kubernetes-dashboard---kind: Deployment
apiVersion: apps/v1
metadata:labels:k8s-app: kubernetes-dashboardname: kubernetes-dashboardnamespace: kubernetes-dashboard
spec:replicas: 1revisionHistoryLimit: 10selector:matchLabels:k8s-app: kubernetes-dashboardtemplate:metadata:labels:k8s-app: kubernetes-dashboardspec:containers:- name: kubernetes-dashboardimage: kubernetesui/dashboard:v2.0.0imagePullPolicy: Alwaysports:- containerPort: 8443protocol: TCPargs:- --auto-generate-certificates- --namespace=kubernetes-dashboard# Uncomment the following line to manually specify Kubernetes API server Host# If not specified, Dashboard will attempt to auto discover the API server and connect# to it. Uncomment only if the default does not work.# - --apiserver-host=http://my-address:portvolumeMounts:- name: kubernetes-dashboard-certsmountPath: /certs# Create on-disk volume to store exec logs- mountPath: /tmpname: tmp-volumelivenessProbe:httpGet:scheme: HTTPSpath: /port: 8443initialDelaySeconds: 30timeoutSeconds: 30securityContext:allowPrivilegeEscalation: falsereadOnlyRootFilesystem: truerunAsUser: 1001runAsGroup: 2001volumes:- name: kubernetes-dashboard-certssecret:secretName: kubernetes-dashboard-certs- name: tmp-volumeemptyDir: {}serviceAccountName: kubernetes-dashboardnodeSelector:"kubernetes.io/os": linux# Comment the following tolerations if Dashboard must not be deployed on mastertolerations:- key: node-role.kubernetes.io/mastereffect: NoSchedule---kind: Service
apiVersion: v1
metadata:labels:k8s-app: dashboard-metrics-scrapername: dashboard-metrics-scrapernamespace: kubernetes-dashboard
spec:ports:- port: 8000targetPort: 8000selector:k8s-app: dashboard-metrics-scraper---kind: Deployment
apiVersion: apps/v1
metadata:labels:k8s-app: dashboard-metrics-scrapername: dashboard-metrics-scrapernamespace: kubernetes-dashboard
spec:replicas: 1revisionHistoryLimit: 10selector:matchLabels:k8s-app: dashboard-metrics-scrapertemplate:metadata:labels:k8s-app: dashboard-metrics-scraperannotations:seccomp.security.alpha.kubernetes.io/pod: 'runtime/default'spec:containers:- name: dashboard-metrics-scraperimage: kubernetesui/metrics-scraper:v1.0.4ports:- containerPort: 8000protocol: TCPlivenessProbe:httpGet:scheme: HTTPpath: /port: 8000initialDelaySeconds: 30timeoutSeconds: 30volumeMounts:- mountPath: /tmpname: tmp-volumesecurityContext:allowPrivilegeEscalation: falsereadOnlyRootFilesystem: truerunAsUser: 1001runAsGroup: 2001serviceAccountName: kubernetes-dashboardnodeSelector:"kubernetes.io/os": linux# Comment the following tolerations if Dashboard must not be deployed on mastertolerations:- key: node-role.kubernetes.io/mastereffect: NoSchedulevolumes:- name: tmp-volumeemptyDir: {}
kubectl apply -f recommended.yaml
#创建service account并绑定默认cluster-admin管理员集群角色
kubectl create serviceaccount dashboard-admin -n kube-system
kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin
kubectl describe secrets -n kube-system $(kubectl -n kube-system get secret | awk '/dashboard-admin/{print $1}')

 使用输出的token登录Dashboard

 

 

本文来自互联网用户投稿,该文观点仅代表作者本人,不代表本站立场。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如若转载,请注明出处:http://www.rhkb.cn/news/334625.html

如若内容造成侵权/违法违规/事实不符,请联系长河编程网进行投诉反馈email:809451989@qq.com,一经查实,立即删除!

相关文章

如何将云服务器上操作系统由centos切换为ubuntu

本文将介绍如何将我们购买的云服务器上之前装的centos切换为ubuntu&#xff0c;云服务器以华为云为例&#xff0c;要切换的ubuntu版本为ubuntu20.04。 参考官方文档&#xff1a;切换操作系统_弹性云服务器 ECS (huaweicloud.com) 首先打开华为云官网&#xff0c;登录后点击右…

每日AIGC最新进展(10):符号音乐生成SYMPLEX、新型图像编辑数据集ReasonPix2Pix、角色一致性插画生成、高级的风格个性化扩散模型

Diffusion Models专栏文章汇总:入门与实战 SYMPLEX: Controllable Symbolic Music Generation using Simplex Diffusion with Vocabulary Priors http://arxiv.org/abs/2405.12666v1 本文介绍了一种新的符号音乐生成方法,名为SYMPLEX,它基于单纯形扩散(Simplex Diffusion…

C++算术运算和自增自减运算

一 引言 表示运算的符号称为运算符。 算术运算&#xff1b; 比较运算&#xff1b; 逻辑运算&#xff1b; 位运算&#xff1b; 1 算术运算 算术运算包括加、减、乘、除、乘方、指数、对数、三角函数、求余函数&#xff0c;这些都是算术运算。 C中用、-、*、/、%分别表示加、减…

Vue3学习使用axios和qs进行POST请求和响应处理

提示&#xff1a;文章写完后&#xff0c;目录可以自动生成&#xff0c;如何生成可参考右边的帮助文档 文章目录 一、前言1.准备工作2.发送POST请求3.处理响应数据4.总结 一、前言 在前端开发中&#xff0c;经常需要与后端进行数据交互&#xff0c;其中包括发送POST请求并处理响…

C语言学习笔记之操作符篇

目录 算术运算符 移位操作符 整型在内存中的存储&#xff08;补充知识&#xff09; ​编辑左移操作符 右移操作符 位操作符 赋值操作符 复合赋值操作符 单目操作符 关系操作符 逻辑操作符 && 与 || 的计算特点 条件操作符 逗号表达式 下标引用操作符 函…

为什么会有websocket(由来)

一、HTTP 协议的缺点和解决方案 1、HTTP 协议的缺点和解决方案 用户在使用淘宝、京东这样的网站的时候&#xff0c;每当点击一个按钮其实就是发送一个http请求。那我们先来回顾一下http请求的请求方式。 一个完整的http请求是被分为request请求节点和response响应阶段的&…

springboot项目部署到linux服务器

springboot后端 修改前 修改后 vue前端 修改前 将地址中的 localhost改为 ip 重新生成war包 war上传到linux的tomcat的webapps下 其他环境配置和macOS大差不差 Tomcat安装使用与部署Web项目的三种方法_tomcat部署web项目-CSDN博客

go升级后 编译的exe在win7上无法正常运行

D:/Go/src/runtime/sys_windows_amd64.s:65 x75 fpx22fca sp-0x22fc8日 升级到go 1.21后报一堆错误&#xff0c;要死了啊 原来是go 1.21不支持win7了&#xff0c;必须把go退回到1.20版本 谷歌发布编程语言 Go 1.21 版本&#xff1a;取消支持微软 Win7/8 及苹果 macOS 10.13/10…

linux安装mysql后,配置mysql,并连接navicate软件

Xshell连接登陆服务器 输入全局命令 mysql -u root -p 回车后&#xff0c;输入密码&#xff0c;不显示输入的密码 注意mysql服务状态&#xff0c;是否运行等 修改配置文件my.cnf&#xff0c;这里没找到就找my.ini&#xff0c;指定有一个是对的 find / -name my.cnf 接下…

MySQL数据库案例实战教程:数据类型、语法与高级查询详解

✨✨ 欢迎大家来访Srlua的博文&#xff08;づ&#xffe3;3&#xffe3;&#xff09;づ╭❤&#xff5e;✨✨ &#x1f31f;&#x1f31f; 欢迎各位亲爱的读者&#xff0c;感谢你们抽出宝贵的时间来阅读我的文章。 我是Srlua小谢&#xff0c;在这里我会分享我的知识和经验。&am…

Flink 通过 paimon 关联维表,内存降为原来的1/4

你好&#xff0c;我是 shengjk1&#xff0c;多年大厂经验&#xff0c;努力构建 通俗易懂的、好玩的编程语言教程。 欢迎关注&#xff01;你会有如下收益&#xff1a; 了解大厂经验拥有和大厂相匹配的技术等 希望看什么&#xff0c;评论或者私信告诉我&#xff01; 文章目录 一…

Windows UWP ContentDialog去掉阴影(全透明)的实现

一、前言 在WIndows开发中&#xff0c;使用UWP&#xff08;Universal WIndows&#xff09;项目开发过程中&#xff0c;使用ContentDialog 的过程中&#xff0c;我们可能并不满足现有的样式&#xff0c;这时就需要自定义样式。笔者在自定义样式过程中&#xff0c;遇到了一个难题…

18 - grace数据处理 - 补充 - 地下水储量计算过程分解 - 地表水储量变化Glads水文数据处理

18 - grace数据处理 - 补充 - 地下水储量计算过程分解 - 地表水储量变化 0 引言1 Grace陆地水储量过程整合0 引言 由水量平衡方程可以将地下水储量的计算过程分解为3个部分,第一部分计算陆地水储量变化、第二部分计算地表水储量变化、第三部分计算地下水储量变化。本篇简单介绍…

Android 逆向学习【1】——版本/体系结构/代码学习

#Android 历史版本 参考链接&#xff1a;一篇文章让你了解Android各个版本的历程 - 知乎 (zhihu.com) 三个部分&#xff1a;api等级、版本号、代号&#xff08;这三个东西都是指的同一个系统&#xff09; API等级&#xff1a;在APP开发的时候写在清单列表里面的 版本号&…

【Python-OS】os.path.splitext()

作用&#xff1a;将文件路径分割成文件名和扩展名两部分。 slide_id, _ os.path.splitext(slide) print("slide:") print(slide) print("slide_id:") print(slide_id)注&#xff1a; slide是文件名&#xff0c;可以自行赋值

【Go语言入门学习笔记】Part3.指针和运算符、以及基本输入

一、前言 仍然好多和C语言类似&#xff0c;计算机的学生应该是很容易入门这一环节&#xff0c;我还在最后的输入中看到了一些些Java输入的影子&#xff0c;而自动的变量类型推断更是有Python那个味道&#xff0c;正可谓几百家之所长了。 二、学习代码 package mainimport (&q…

Angular(1):使用Angular CLI创建空项目

要创建一个空的 Angular 项目&#xff0c;可以使用 Angular CLI&#xff08;命令行界面&#xff09;。以下是使用 Angular CLI 创建一个新项目的步骤&#xff1a; 1、安装 Angular CLI&#xff1a; 打开你的命令行界面&#xff08;在 Windows 上是 CMD、PowerShell 或 Git Bas…

动态内存管理—C语言通讯录

目录 一&#xff0c;动态内存函数的介绍 1.1 malloc和free 1.2 calloc 1.3 realloc 1.4C/C程序的内存开辟 二&#xff0c;通讯录管理系统 动态内存函数的介绍 malloc free calloc realloc 一&#xff0c;动态内存函数的介绍 1.1 malloc和free void* malloc (…

mysql实战——Mysql8.0高可用之双主+keepalived

一、介绍 利用keepalived实现Mysql数据库的高可用&#xff0c;KeepalivedMysql双主来实现MYSQL-HA&#xff0c;两台Mysql数据库的数据保持完全一致&#xff0c;实现方法是两台Mysql互为主从关系&#xff0c;通过keepalived配置VIP&#xff0c;实现当其中的一台Mysql数据库宕机…

Vite + Vue3 + Electron 创建打包桌面程序

10 【Vite Vue3 Electron 创建打包桌面程序】 1.使用 Vite 构建 Electron 项目 1.1 创建 Vite 应用&#xff0c;安装 Electron 依赖 创建一个 Vite 项目 npm init vitelatest安装 Electron 相关依赖 npm install electron -D npm install vite-plugin-electron -D 1.2 在…