一、常见的k8s部署方式

1、使用kubeadm工具部署

kubeadm 是 Kubernetes 官方推荐的简化部署工具，能够快速初始化 Master 节点并加入 Worker 节点，自动化了很多设置过程。它还支持集群的升级和维护操作。这种方式相比手动部署更简便，但需要满足一定的硬件和软件环境要求。

2、基于二进制文件的部署方式

这种方式需要手动下载和安装 k8s 组件，包括 kube-apiserver、kube-controller-manager、kube-scheduler、kubelet、kube-proxy 等组件，并手动配置参数和启动命令。适合需要自定义配置和精细控制的场景，但也需要花费更多的时间和精力进行维护和升级。

3、云服务提供商的托管 Kubernetes 服务

众多云服务提供商如 Amazon EKS、Microsoft AKS、Google GKE 等提供了托管 Kubernetes 服务。通过这些服务，用户可以快速创建和管理 Kubernetes 集群，无需关心底层基础设施的维护，同时还能享受到云平台的自动扩展、监控和安全服务等高级功能。

4、使用容器镜像部署或自动化部署工具

包括但不限于使用容器镜像部署 Kubernetes 组件，或是利用自动化部署工具如 Ansible、Terraform、Kubespray、Rancher 等。这些方法通过预配置的模板或剧本自动化集群的安装和配置过程，适用于追求部署效率和标准化的场景。例如，Kubespray 是一个使用 Ansible 来部署 Kubernetes 集群的工具，支持多种云平台和本地环境。

本文将从使用kubeadm工具部署、二进制文件部署、容器镜像部署三种方式部署k8s

二、使用kubeadm工具部署

1、硬件准备（虚拟主机）

角色	主机名	ip地址
master	k8s-master	192.168.112.10
node	k8s-node1	192.168.112.20
node	k8s-node2	192.168.112.30

CentOS Linux release 7.9.2009 (Core)

至少2核CPU、3GB以上内存

使用命令hostnamectl set-hostname临时修改主机名

2、环境准备

2.1、所有机器关闭防火墙

systemctl stop firewalld	#关闭
systemctl disable firewalld		#开机不自启
systemctl status firewalld		#查看状态

2.2、所有机器关闭selinux

sed -i 's/enforcing/disabled/' /etc/selinux/config  setenforce 0

2.3、所有机器关闭swap

swapoff -a	# 临时关闭
sed -ri 's/.*swap.*/#&/' /etc/fstab  #永久关闭
cat /etc/fstab | grep swap	#验证

2.4、所有机器上添加主机名与ip的对应关系

vim /etc/hosts127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.112.10 k8s-master
192.168.112.20 k8s-node1
192.168.112.30 k8s-node2

2.5、将所有主机上桥接的ipv4流量传递到iptables链

cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOFsysctl --system  # 生效

2.6、时间同步

yum install -y ntpdate
ntpdate time.windows.com

3、为所有节点安装docker

yum install wget.x86_64 -y
rm -rf /etc/yum.repos.d/*
wget -O /etc/yum.repos.d/centos7.repo http://mirrors.aliyun.com/repo/Centos-7.repo
wget -O /etc/yum.repos.d/epel-7.repo http://mirrors.aliyun.com/repo/epel-7.repo
wget -O /etc/yum.repos.d/docker-ce.repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum install docker-ce -y
systemctl start docker && systemctl enable docker

3.1、配置镜像下载加速器

cat > /etc/docker/daemon.json <<EOF
{"exec-opts": ["native.cgroupdriver=systemd"],"registry-mirrors": ["https://jzjzrggd.mirror.aliyuncs.com"]}
EOF
[root@k8s-master ~]# systemctl daemon-reload
[root@k8s-master ~]# systemctl restart docker.service

4、集群部署

4.1、为所有节点修改从库，安装kubeadm、kubelet、kubectl

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
yum install kubelet-1.22.2 kubeadm-1.22.2 kubectl-1.22.2 -y
systemctl enable kubelet && systemctl start kubelet

4.2、部署master节点

kubeadm init \
--apiserver-advertise-address=192.168.112.10 \
--image-repository registry.aliyuncs.com/google_containers \
--kubernetes-version v1.22.2 \
--control-plane-endpoint k8s-master \
--service-cidr=172.16.0.0/16 \
--pod-network-cidr=10.244.0.0/16

kubeadm join k8s-master:6443 --token 6maoc1.6pu6yorxwkooilwz \--discovery-token-ca-cert-hash sha256:206839f900f5cc01c75a6191274853bc2c49dadfde1c66a227d4575deafbd65d

记得保存好这段命令是用于将一个工作节点（worker node）加入到已存在的 Kubernetes 集群中的过程。

4.3.1、遇到报错

Here is one example how you may list all Kubernetes containers running in docker:- 'docker ps -a | grep kube | grep -v pause'Once you have found the failing container, you can inspect its logs with:- 'docker logs CONTAINERID'error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher

4.3.2、解决办法

rm -rf /etc/containerd/config.toml
systemctl restart containerd

4.4、按照指令执行

To start using your cluster, you need to run the following as a regular user

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
export KUBECONFIG=/etc/kubernetes/admin.conf

4.5、查看master节点kubelet.service状态

systemctl status kubelet.service

4.6、查看节点状态

由于网络插件还没有部署，还没有准备就绪 NotReady

[root@k8s-master ~]# kubectl get nodes
NAME         STATUS     ROLES                  AGE   VERSION
k8s-master   NotReady   control-plane,master   10m   v1.22.2

5、安装网络插件

Flannel是K8s中的一个网络插件，它负责实现容器在集群中的网络通信。

官方文档

# 最好手动提前拉取所需镜像
[root@k8s-master ~]# docker pull quay.io/coreos/flannel:v0.14.0
[root@k8s-master ~]# wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
[root@k8s-master ~]# kubectl apply -f kube-flannel.yml 

6、添加node节点

# 为node拉取网络插件镜像
[root@k8s-node1 ~]# docker pull quay.io/coreos/flannel:v0.14.0
[root@k8s-node2 ~]# docker pull quay.io/coreos/flannel:v0.14.0
[root@k8s-node1 ~]# kubeadm join k8s-master:6443 --token 6maoc1.6pu6yorxwkooilwz \
>         --discovery-token-ca-cert-hash sha256:206839f900f5cc01c75a6191274853bc2c49dadfde1c66a227d4575deafbd65d
[root@k8s-node2 ~]# kubeadm join k8s-master:6443 --token 6maoc1.6pu6yorxwkooilwz \
>         --discovery-token-ca-cert-hash sha256:206839f900f5cc01c75a6191274853bc2c49dadfde1c66a227d4575deafbd65d

6.1、k8s集群中Node节点报错"The connection to the server localhost:8080 was refused - did you specify the right host or port?"

6.2、解决方案

6.2.1、将Master节点上的"/etc/kubernetes/admin.conf"拷贝到Node上"/etc/kubernetes/"

Master节点上操作

[root@k8s-master ~]# scp /etc/kubernetes/admin.conf 192.168.112.20:/etc/kubernetes/
[root@k8s-master ~]# scp /etc/kubernetes/admin.conf 192.168.112.30:/etc/kubernetes/

6.2.2、在从节点上设置环境变量

Node节点操作

#Node1
[root@k8s-node1 ~]# echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile
[root@k8s-node1 ~]# source ~/.bash_profile#Node2
[root@k8s-node2 ~]# echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile
[root@k8s-node2 ~]# source ~/.bash_profile

至此kubeadm工具部署k8s集群完成

三、基于二进制文件部署

1、硬件准备（虚拟机）

角色	主机名	ip地址	组件
master	k8s-master1	192.168.112.10	kube-apiserver，kube-controller-manager，kube-scheduler，kubelet，kube-proxy，docker，etcd
node	k8s-node1	192.168.112.20	kubelet，kube-proxy，docker，etcd
node	k8s-node2	192.168.112.30	kubelet，kube-proxy，docker，etcd

CentOS Linux release 7.9.2009 (Core)

至少2核CPU、3GB以上内存

使用命令hostnamectl set-hostname临时修改主机名

2、环境准备

2.1、所有机器关闭防火墙

systemctl stop firewalld	#关闭
systemctl disable firewalld		#开机不自启
systemctl status firewalld		#查看状态

2.2、所有机器关闭selinux

sed -i 's/enforcing/disabled/' /etc/selinux/config 
setenforce 0

2.3、所有机器关闭swap

swapoff -a # 临时关闭
sed -ri 's/.*swap.*/#&/' /etc/fstab  #永久关闭

2.4、所有机器上添加主机名与ip的对应关系

cat >> /etc/hosts << EOF
192.168.112.10 k8s-master
192.168.112.20 k8s-node1
192.168.112.30 k8s-node2
EOF

2.5、将所有主机上桥接的ipv4流量传递到iptables链

cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOFsysctl --system	#失效

2.6、时间同步

yum install -y ntpdate
ntpdate time.windows.com

3、部署etcd集群

3.1、etcd简介

etcd是CoreOS开发的一个分布式Key-Value存储系统，内部采用 raft 协议作为一致性算法，用于可靠、快速地保存关键数据，并提供访问、共享配置和服务发现。通过分布式锁、leader选举和写屏障(write barriers)，来实现可靠的分布式协作。Kubernetes使用ETCD存储所有集群数据，包括Pods、服务、服务发现等。

etcd 服务作为 Kubernetes 集群的主数据库，在安装 Kubernetes 各服务之前需要首先安装和启动。

3.2、对应节点规划

节点名称	ip地址
etcd-1	192.168.112.10
etcd-2	192.168.112.20
etcd-3	192.168.112.30

etcd可以与k8s节点复用，也可以部署在k8s机器外

只要apiserver可以连接到即可

3.3、cfssl证书生成工具准备

3.3.1、创建存放cfssl工具的目录

cfssl 是一个开源的证书管理工具，使用 json 文件生成证书。

master节点执行

mkdir /software-cfssl
#创建目录存放cfssl工具

3.3.2、下载cfssl工具

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -P /software-cfssl/ --no-check-certificate
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -P /software-cfssl/ --no-check-certificate
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -P /software-cfssl/ --no-check-certificate

3.3.3、给cfssl相关工具赋可执行权限，并复制到对应目录下

cd /software-cfssl/
chmod +x *
cp cfssl_linux-amd64 /usr/local/bin/cfssl
cp cfssljson_linux-amd64 /usr/local/bin/cfssljson
cp cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo

3.4、自签证书颁发机构（CA）

3.4.1、创建工作目录

master节点执行

mkdir -p ~/TLS/{etcd,k8s}
cd ~/TLS/etcd/

3.4.2、生成自签证书CA配置

cat > ca-config.json << EOF
{"signing": {"default": {"expiry": "87600h"},"profiles": {"www": {"expiry": "87600h","usages": ["signing","key encipherment","server auth","client auth"]}}}
}
EOF

1、ca-config.json # 可以定义多个profiles，分别指定不同的过期时间、使用场景等参数，后续在签名证书时使用某个profile参数配置服务证书的有效期。

2、signing # 表示该证书可用于签名其它证书，生成的ca.pem证书中CA=TRUE。

3、server auth # 表示client可以用该CA对server提供的证书进行验证。

4、client auth # 表示 server 可以用该CA对client提供的证书进行验证。

3.4.3、自签CA证书签名请求文件

cat > ca-csr.json << EOF
{"CN": "etcd CA","key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","L": "Beijing","ST": "Beijing"}]
}
EOF

3.4.4、生成自签CA证书和私钥

cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

3.4.5、验证CA证书ca.pem

ca证书的有效期是5年

cfssl certinfo -cert ca.pem | grep not

3.4.6、验证有效期

验证网站

3.5、使用自签CA签发etcd https证书

3.5.1、创建证书申请文件

cat > server-csr.json << EOF
{"CN": "etcd","hosts": ["192.168.112.10","192.168.112.20","192.168.112.30"],"key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","L": "Beijing","ST": "Beijing"}]
}
EOF

3.5.2、生成证书

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server

3.5.3、查看当前生成的一些相关证书

[root@k8s-master etcd]# ls
ca-config.json  ca-csr.json  ca.pem      server-csr.json  server.pem
ca.csr          ca-key.pem   server.csr  server-key.pem

3.6、下载etcd二进制文件

wget https://github.com/etcd-io/etcd/releases/download/v3.4.9/etcd-v3.4.9-linux-amd64.tar.gz

3.7、部署etcd集群

master节点操作

3.7.1、创建工作目录并解压二进制文件

mkdir -p /opt/etcd/{bin,cfg,ssl}
mv -f /root/TLS/etcd/etcd-v3.4.9-linux-amd64.tar.gz ~
tar zxvf etcd-v3.4.9-linux-amd64.tar.gz
mv etcd-v3.4.9-linux-amd64/{etcd,etcdctl} /opt/etcd/bin/

3.7.2、创建etcd配置文件

cat > /opt/etcd/cfg/etcd.conf << EOF
#[Member]
ETCD_NAME="etcd-1"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.112.10:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.112.10:2379"#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.112.10:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.112.10:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.112.10:2380,etcd-2=https://192.168.112.20:2380,etcd-3=https://192.168.112.30:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF

ETCD_NAME：节点名称，集群中唯一

ETCD_DATA_DIR：数据目录

ETCD_LISTEN_PEER_URLS：集群通讯监听地址

ETCD_LISTEN_CLIENT_URLS：客户端访问监听地址

ETCD_INITIAL_CLUSTER：集群节点地址

ETCD_INITIALCLUSTER_TOKEN：集群Token

ETCD_INITIALCLUSTER_STATE：加入集群的状态，new是新集群，existing表示加入已有集群

3.8、systemd管理etcd

cat > /usr/lib/systemd/system/etcd.service << EOF
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target[Service]
Type=notify
EnvironmentFile=/opt/etcd/cfg/etcd.conf
ExecStart=/opt/etcd/bin/etcd \
--cert-file=/opt/etcd/ssl/server.pem \
--key-file=/opt/etcd/ssl/server-key.pem \
--peer-cert-file=/opt/etcd/ssl/server.pem \
--peer-key-file=/opt/etcd/ssl/server-key.pem \
--trusted-ca-file=/opt/etcd/ssl/ca.pem \
--peer-trusted-ca-file=/opt/etcd/ssl/ca.pem \
--logger=zap
Restart=on-failure
LimitNOFILE=65536[Install]
WantedBy=multi-user.target
EOF

3.9、三个机器做ssh免密

ssh-keygen -t rsa
ssh-copy-id -i ~/.ssh/id_rsa.pub k8s-master
ssh-copy-id -i ~/.ssh/id_rsa.pub k8s-node1
ssh-copy-id -i ~/.ssh/id_rsa.pub k8s-node2

3.10、将master节点生成的文件拷贝到node1和node2上

#!/bin/bash
cp ~/TLS/etcd/ca*pem ~/TLS/etcd/server*pem /opt/etcd/ssl
for i in {2..3}
do
scp -r /opt/etcd/ root@192.168.112.${i}0:/opt/
scp /usr/lib/systemd/system/etcd.service root@192.168.112.${i}0:/usr/lib/systemd/system
done

3.10.1、查看master节点的树形结构

[root@k8s-master ~]# tree /opt/etcd/
/opt/etcd/
├── bin
│   ├── etcd
│   └── etcdctl
├── cfg
│   └── etcd.conf
└── ssl├── ca-key.pem├── ca.pem├── server-key.pem└── server.pem3 directories, 7 files

3.10.2、查看etcd服务是否存在

tree /usr/lib/systemd/system/ | grep etcd

以上两个命令在所有节点都操作验证一下

3.11、修改node1和node2节点的etcd.conf配置文件

node1

vim /opt/etcd/cfg/etcd.conf#[Member]
ETCD_NAME="etcd-2"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.112.20:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.112.20:2379"#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.112.20:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.112.20:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.112.10:2380,etcd-2=https://192.168.112.20:2380,etcd-3=https://192.168.112.30:2380"  
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

node2

vim /opt/etcd/cfg/etcd.conf#[Member]
ETCD_NAME="etcd-3"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.112.30:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.112.30:2379"#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.112.30:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.112.30:2379"
ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.112.10:2380,etcd-2=https://192.168.112.20:2380,etcd-3=https://192.168.112.30:2380"  
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

3.12、启动etcd并设置开机自启

etcd需要多个节点同时启动，单个启动会卡住。下面每个命令在三台服务器上同时执行。

systemctl daemon-reload
systemctl start etcd
systemctl enable etcd
systemctl status etcd

3.13、检查etcd集群状态

执行etcdctl cluster-health，验证etcd是否正确启动

master节点执行

ETCDCTL_API=3 /opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints="https://192.168.112.10:2379,https://192.168.112.20:2379,https://192.168.112.30:2379" endpoint health --write-out=table

3.13.1、报错解决

主要还是查看etcd日志，针对性排查

less /var/log/message
journalctl -u etcd

4、所有节点安装Docker

4.1、下载并解压二进制包

cd ~
wget https://download.docker.com/linux/static/stable/x86_64/docker-19.03.9.tgz
tar -zxvf docker-19.03.9.tgz | mv docker/* /usr/bin/

4.2、配置镜像加速

mkdir -p /etc/docker
cat >> /etc/docker/daemon.json << EOF
{"registry-mirrors": ["https://jzjzrggd.mirror.aliyuncs.com"]
}
EOF

4.3、配置docker.service

cat > /usr/lib/systemd/system/docker.service << EOF
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target firewalld.service
Wants=network-online.target[Service]
Type=notify
ExecStart=/usr/bin/dockerd --selinux-enabled=false --insecure-registry=127.0.0.1
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
#TasksMax=infinity
TimeoutStartSec=0
Delegate=yes
KillMode=process
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s[Install]
WantedBy=multi-user.target
EOF

4.4、启动docker并设置开机自启

systemctl daemon-reload
systemctl start docker
systemctl enable docker
systemctl status docker

5、部署master节点

5.1、生成kube-apiserver

5.1.1、自签证书（CA）

cd ~/TLS/k8scat > ca-config.json << EOF
{"signing": {"default": {"expiry": "87600h"},"profiles": {"kubernetes": {"expiry": "87600h","usages": ["signing","key encipherment","server auth","client auth"]}}}
}
EOF

5.1.2、自签CA证书签名请求文件

cat > ca-csr.json << EOF
{"CN": "kubernetes","key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","L": "Beijing","ST": "Beijing","O": "k8s","OU": "System"}]
}
EOF

5.1.3、生成证书

cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

5.1.4、查看生成的证书文件

[root@k8s-master k8s]# ls
ca-config.json  ca.csr  ca-csr.json  ca-key.pem  ca.pem

5.1.5、创建证书申请文件

cat > server-csr.json << EOF
{"CN": "kubernetes","hosts": ["10.0.0.1","127.0.0.1","192.168.112.10","192.168.112.20","192.168.112.30","kubernetes","kubernetes.default","kubernetes.default.svc","kubernetes.default.svc.cluster","kubernetes.default.svc.cluster.local"],"key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","L": "BeiJing","ST": "BeiJing","O": "k8s","OU": "System"}]
}
EOF

5.1.6、生成证书

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes server-csr.json | cfssljson -bare server

5.1.7、查看证书生成的文件

[root@k8s-master k8s]# ls
ca-config.json  ca-csr.json  ca.pem      server-csr.json  server.pem
ca.csr          ca-key.pem   server.csr  server-key.pem

5.2、下载并解压二进制包

master节点操作

mkdir -p /opt/kubernetes/{bin,cfg,ssl,logs} 
wget https://storage.googleapis.com/kubernetes-release/release/v1.20.15/kubernetes-server-linux-amd64.tar.gz
tar zxvf kubernetes-server-linux-amd64.tar.gz

5.3、复制kube-apiserver、kube-controller-manager 和 kube-scheduler 到 /opt/kubernetes/bin 目录

cd kubernetes/server/bin
cp kube-apiserver kube-scheduler kube-controller-manager kubelet kube-proxy /opt/kubernetes/bin
cp kubectl /usr/bin/

5.4、检查复制情况

[root@k8s-master ~]# tree /opt/kubernetes/bin
/opt/kubernetes/bin
├── kube-apiserver
├── kube-controller-manager
├── kubelet
├── kube-proxy
└── kube-scheduler0 directories, 5 files
[root@k8s-master ~]# tree /usr/bin/ | grep kubectl
├── kubectl

5.5、部署kube-apiserver

5.5.1、创建配置文件

master节点操作

cat > /opt/kubernetes/cfg/kube-apiserver.conf << EOF
KUBE_APISERVER_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--etcd-servers=https://192.168.112.10:2379,https://192.168.112.20:2379,https://192.168.112.30:2379 \\
--bind-address=192.168.112.10 \\
--secure-port=6443 \\
--advertise-address=192.168.112.10 \\
--allow-privileged=true \\
--service-cluster-ip-range=10.0.0.0/24 \\
--enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,NodeRestriction \\
--authorization-mode=RBAC,Node \\
--enable-bootstrap-token-auth=true \\
--token-auth-file=/opt/kubernetes/cfg/token.csv \\
--service-node-port-range=30000-32767 \\
--kubelet-client-certificate=/opt/kubernetes/ssl/server.pem \\
--kubelet-client-key=/opt/kubernetes/ssl/server-key.pem \\
--tls-cert-file=/opt/kubernetes/ssl/server.pem  \\
--tls-private-key-file=/opt/kubernetes/ssl/server-key.pem \\
--client-ca-file=/opt/kubernetes/ssl/ca.pem \\
--service-account-key-file=/opt/kubernetes/ssl/ca-key.pem \\
--service-account-issuer=api \\
--service-account-signing-key-file=/opt/kubernetes/ssl/server-key.pem \\
--etcd-cafile=/opt/etcd/ssl/ca.pem \\
--etcd-certfile=/opt/etcd/ssl/server.pem \\
--etcd-keyfile=/opt/etcd/ssl/server-key.pem \\
--requestheader-client-ca-file=/opt/kubernetes/ssl/ca.pem \\
--proxy-client-cert-file=/opt/kubernetes/ssl/server.pem \\
--proxy-client-key-file=/opt/kubernetes/ssl/server-key.pem \\
--requestheader-allowed-names=kubernetes \\
--requestheader-extra-headers-prefix=X-Remote-Extra- \\
--requestheader-group-headers=X-Remote-Group \\
--requestheader-username-headers=X-Remote-User \\
--enable-aggregator-routing=true \\
--audit-log-maxage=30 \\
--audit-log-maxbackup=3 \\
--audit-log-maxsize=100 \\
--audit-log-path=/opt/kubernetes/logs/k8s-audit.log"
EOF

5.5.2、拷贝刚才生成的证书

cp ~/TLS/k8s/ca*pem ~/TLS/k8s/server*pem /opt/kubernetes/ssl/

5.2.3、创建配置文件的token文件

cat > /opt/kubernetes/cfg/token.csv << EOF
62ee59c473229084c6b57a19eec91e26,kubelet-bootstrap,10001,"system:node-bootstrapper"
EOF

token也可自行生成替换：

head -c 16 /dev/urandom | od -An -t x | tr -d ' '

5.2.4、systemd管理apiserver

cat > /usr/lib/systemd/system/kube-apiserver.service << EOF
[Unit]
Description=Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-apiserver.conf
ExecStart=/opt/kubernetes/bin/kube-apiserver \$KUBE_APISERVER_OPTS
Restart=on-failure[Install]
WantedBy=multi-user.target
EOF

5.2.5、启动kube-apiserver并设置开机启动

systemctl daemon-reload
systemctl start kube-apiserver 
systemctl enable kube-apiserver
systemctl status kube-apiserver

6、部署kube-controller-manager

6.1、创建配置文件

master节点操作

cat > /opt/kubernetes/cfg/kube-controller-manager.conf << EOF
KUBE_CONTROLLER_MANAGER_OPTS="--logtostderr=false \\
--v=2 \\
--log-dir=/opt/kubernetes/logs \\
--leader-elect=true \\
--kubeconfig=/opt/kubernetes/cfg/kube-controller-manager.kubeconfig \\
--bind-address=127.0.0.1 \\
--allocate-node-cidrs=true \\
--cluster-cidr=10.244.0.0/16 \\
--service-cluster-ip-range=10.0.0.0/24 \\
--cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \\
--cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem  \\
--root-ca-file=/opt/kubernetes/ssl/ca.pem \\
--service-account-private-key-file=/opt/kubernetes/ssl/ca-key.pem \\
--cluster-signing-duration=87600h0m0s"
EOF

6.2、生成kubeconfig文件

[root@k8s-master1 ~]# cat /opt/kubernetes/cfg/kube-controller-manager.kubeconfig
apiVersion: v1
clusters:
- cluster:certificate-authority-data: LS0tserver: https://192.168.112.10:6443name: kubernetes
contexts:
- context:cluster: kubernetesuser: kube-controller-managername: default
current-context: default
kind: Config
preferences: {}
users:
- name: kube-controller-manageruser:client-certificate-data: LS0tclient-key-data: LS0t

6.3、创建证书请求文件

cd ~/TLS/k8scat > kube-controller-manager-csr.json << EOF
{"CN": "system:kube-controller-manager","hosts": [],"key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","L": "BeiJing", "ST": "BeiJing","O": "system:masters","OU": "System"}]
}
EOF

6.4、生成kube-controller-manager证书

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes kube-controller-manager-csr.json | cfssljson -bare kube-controller-manager

6.5、查看生成的证书文件

[root@k8s-master k8s]# ls kube-controller-manager*
kube-controller-manager.csr       kube-controller-manager-key.pem
kube-controller-manager-csr.json  kube-controller-manager.pem

6.6、生成kubeconfig文件

KUBE_CONFIG="/opt/kubernetes/cfg/kube-controller-manager.kubeconfig"
KUBE_APISERVER="https://192.168.112.10:6443"kubectl config set-cluster kubernetes \--certificate-authority=/opt/kubernetes/ssl/ca.pem \--embed-certs=true \--server=${KUBE_APISERVER} \--kubeconfig=${KUBE_CONFIG}kubectl config set-credentials kube-controller-manager \--client-certificate=./kube-controller-manager.pem \--client-key=./kube-controller-manager-key.pem \--embed-certs=true \--kubeconfig=${KUBE_CONFIG}kubectl config set-context default \--cluster=kubernetes \--user=kube-controller-manager \--kubeconfig=${KUBE_CONFIG}kubectl config use-context default --kubeconfig=${KUBE_CONFIG}

6.7、systemd管理controller-manager

cat > /usr/lib/systemd/system/kube-controller-manager.service << EOF
[Unit]
Description=Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
After=kube-apiserver.service
Requires=kube-apiserver.service[Service]
EnvironmentFile=/opt/kubernetes/cfg/kube-controller-manager.conf
ExecStart=/opt/kubernetes/bin/kube-controller-manager \$KUBE_CONTROLLER_MANAGER_OPTS
Restart=on-failure[Install]
WantedBy=multi-user.target
EOF

6.7.1、查看kube-controller-manager.service服务是否添加成功

ls /usr/lib/systemd/system/ | grep kube-controller-manager

6.8、启动kube-controller-manager并设置开机自启

systemctl daemon-reload
systemctl start kube-controller-manager 
systemctl enable kube-controller-manager
systemctl status kube-controller-manager

6.8.1、报错invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable

journalctl -u kube-controller-manager

6.8.2、解决方案

暂时写到这，有空再解决