k8s的CICD实施项目

环境需求：
目前领导需要做一个需求，临时把我从运维岗位，把我调度到到专家组让我主导cicd的项目实施
目前环境资源
k8s环境，28台服务器，上面是k8s集群，要实施一个测试环境的cicd以及一个生产环境的cicd
gitlab代码服务器
jenkins服务器
镜像仓库采用的是ACR仓库

一、部署jenkins服务器

mkdir /data/jenkins_data -p
chmod -R 777 /data/jenkins_data

docker run -d --name=jenkins --restart=always -e  \
JENKINS_PASSWORD=admin123 -e JENKINS_USERNAME=admin -e   \
JENKINS_HTTP_PORT_NUMBER=8080 -p 8080:8080 -p 50000:50000 -v   \
/data/jenkins_data:/bitnami/jenkins   bitnami/jenkins:2.479.2-debian-12-r0

访问的时候用
http://192.168.8.9:9090
输入账号密码 admin/admin123 进行访问
在这里插入图片描述
按照对应的插件

Git
Git Parameter
Git Pipeline for Blue Ocean
GitLab
Credentials
Credentials Binding
Blue Ocean
Blue Ocean Pipeline Editor
Blue Ocean Core JS
Pipeline SCM API for Blue Ocean
Dashboard for Blue Ocean
Build With Parameters
Dynamic Extended Choice Parameter Plug-In
Dynamic Parameter Plug-in
Extended Choice Parameter
List Git Branches Parameter
Pipeline
Pipeline: Declarative
Kubernetes
Kubernetes CLI
Kubernetes Credentials
Image Tag Parameter
Active Choices

二、部署gitlab服务器
gitlab相关的软件包在清华源可以找到

yum -y install gitlab-ce

三、部署harbor的镜像仓库

jenkinsfile不带审批功能

#没有添加审批都是可用过的-已测试过
pipeline {agent {kubernetes {cloud 'study-kubernetes'slaveConnectTimeout 1200workspaceVolume hostPathWorkspaceVolume(hostPath: "/opt/workspace", readOnly: false)yaml '''
apiVersion: v1
kind: Pod
spec:containers:- args: [\'$(JENKINS_SECRET)\', \'$(JENKINS_NAME)\']image: 'registry.cn-beijing.aliyuncs.com/dotbalo/jnlp-agent-docker:latest'name: jnlpimagePullPolicy: IfNotPresentvolumeMounts:- mountPath: "/etc/localtime"name: "localtime"readOnly: false- command:- "cat"env:- name: "LANGUAGE"value: "en_US:en"- name: "LC_ALL"value: "en_US.UTF-8"- name: "LANG"value: "en_US.UTF-8"image: "registry.cn-beijing.aliyuncs.com/citools/maven:3.5.3"imagePullPolicy: "IfNotPresent"name: "build"tty: truevolumeMounts:- mountPath: "/etc/localtime"name: "localtime"- mountPath: "/root/.m2/"name: "cachedir"readOnly: false- command:- "cat"env:- name: "LANGUAGE"value: "en_US:en"- name: "LC_ALL"value: "en_US.UTF-8"- name: "LANG"value: "en_US.UTF-8"image: "registry.cn-beijing.aliyuncs.com/citools/kubectl:self-1.17"imagePullPolicy: "IfNotPresent"name: "kubectl"tty: truevolumeMounts:- mountPath: "/etc/localtime"name: "localtime"readOnly: false- command:- "cat"env:- name: "LANGUAGE"value: "en_US:en"- name: "LC_ALL"value: "en_US.UTF-8"- name: "LANG"value: "en_US.UTF-8"image: "registry.cn-beijing.aliyuncs.com/citools/docker:19.03.9-git"imagePullPolicy: "IfNotPresent"name: "docker"tty: truevolumeMounts:- mountPath: "/etc/localtime"name: "localtime"readOnly: false- mountPath: "/var/run/docker.sock"name: "dockersock"readOnly: falserestartPolicy: "Never"nodeSelector:build: "true"securityContext: {}volumes:- hostPath:path: "/var/run/docker.sock"name: "dockersock"- hostPath:path: "/usr/share/zoneinfo/Asia/Shanghai"name: "localtime"- name: "cachedir"hostPath:path: "/opt/m2"
'''}
}stages {stage('Pulling Code') {parallel {stage('Pulling Code by Jenkins') {when {expression {env.gitlabBranch == null}}steps {git(changelog: true, poll: true, url: 'git@192.168.8.8:dev/spring-boot-project.git', branch: "${BRANCH}", credentialsId: 'gitlab-key')script {COMMIT_ID = sh(returnStdout: true, script: "git log -n 1 --pretty=format:'%h'").trim()TAG = BUILD_TAG + '-' + COMMIT_IDprintln "Current branch is ${BRANCH}, Commit ID is ${COMMIT_ID}, Image TAG is ${TAG}"}}}stage('Pulling Code by trigger') {when {expression {env.gitlabBranch != null}}steps {git(url: 'git@192.168.8.8:dev/spring-boot-project.git', branch: env.gitlabBranch, changelog: true, poll: true, credentialsId: 'gitlab-key')script {COMMIT_ID = sh(returnStdout: true, script: "git log -n 1 --pretty=format:'%h'").trim()TAG = BUILD_TAG + '-' + COMMIT_IDprintln "Current branch is ${BRANCH}, Commit ID is ${COMMIT_ID}, Image TAG is ${TAG}"}}}}}stage('Building') {steps {container(name: 'build') {sh """mvn clean install -DskipTests"""}}}stage('Docker build for creating image') {environment {HARBOR_USER     = credentials('HARBOR_ACCOUNT')}steps {container(name: 'docker') {sh """echo ${HARBOR_USER_USR} ${HARBOR_USER_PSW} ${TAG}docker build -t ${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG} .docker login -u ${HARBOR_USER_USR} -p ${HARBOR_USER_PSW} ${HARBOR_ADDRESS}docker push ${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG}"""}}}stage('Deploying to K8s') {environment {MY_KUBECONFIG = credentials('study-k8s-kubeconfig')}steps {container(name: 'kubectl'){sh """/usr/local/bin/kubectl --kubeconfig $MY_KUBECONFIG set image deploy -l app=${IMAGE_NAME} ${IMAGE_NAME}=${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG} -n $NAMESPACE"""}}}}environment {COMMIT_ID = ""HARBOR_ADDRESS = "192.168.8.101"REGISTRY_DIR = "dev"IMAGE_NAME = "spring-boot-project"NAMESPACE = "dev"TAG = ""}parameters {gitParameter(branch: '', branchFilter: 'origin/(.*)', defaultValue: '', description: 'Branch for build and deploy', name: 'BRANCH', quickFilterEnabled: false, selectedValue: 'NONE', sortMode: 'NONE', tagFilter: '*', type: 'PT_BRANCH')}
}

Jenkins带审批功能

已测试过的
pipeline {agent {kubernetes {cloud 'study-kubernetes'slaveConnectTimeout 1200workspaceVolume hostPathWorkspaceVolume(hostPath: "/opt/workspace", readOnly: false)yaml '''
apiVersion: v1
kind: Pod
spec:containers:- args: [\'$(JENKINS_SECRET)\', \'$(JENKINS_NAME)\']image: 'registry.cn-beijing.aliyuncs.com/dotbalo/jnlp-agent-docker:latest'name: jnlpimagePullPolicy: IfNotPresentvolumeMounts:- mountPath: "/etc/localtime"name: "localtime"readOnly: false- command:- "cat"env:- name: "LANGUAGE"value: "en_US:en"- name: "LC_ALL"value: "en_US.UTF-8"- name: "LANG"value: "en_US.UTF-8"image: "registry.cn-beijing.aliyuncs.com/citools/maven:3.5.3"imagePullPolicy: "IfNotPresent"name: "build"tty: truevolumeMounts:- mountPath: "/etc/localtime"name: "localtime"- mountPath: "/root/.m2/"name: "cachedir"readOnly: false- command:- "cat"env:- name: "LANGUAGE"value: "en_US:en"- name: "LC_ALL"value: "en_US.UTF-8"- name: "LANG"value: "en_US.UTF-8"image: "registry.cn-beijing.aliyuncs.com/citools/kubectl:self-1.17"imagePullPolicy: "IfNotPresent"name: "kubectl"tty: truevolumeMounts:- mountPath: "/etc/localtime"name: "localtime"readOnly: false- command:- "cat"env:- name: "LANGUAGE"value: "en_US:en"- name: "LC_ALL"value: "en_US.UTF-8"- name: "LANG"value: "en_US.UTF-8"image: "registry.cn-beijing.aliyuncs.com/citools/docker:19.03.9-git"imagePullPolicy: "IfNotPresent"name: "docker"tty: truevolumeMounts:- mountPath: "/etc/localtime"name: "localtime"readOnly: false- mountPath: "/var/run/docker.sock"name: "dockersock"readOnly: falserestartPolicy: "Never"nodeSelector:build: "true"securityContext: {}volumes:- hostPath:path: "/var/run/docker.sock"name: "dockersock"- hostPath:path: "/usr/share/zoneinfo/Asia/Shanghai"name: "localtime"- name: "cachedir"hostPath:path: "/opt/m2"
'''}
}stages {stage('Pulling Code') {parallel {stage('Pulling Code by Jenkins') {when {expression {env.gitlabBranch == null}}steps {git(changelog: true, poll: true, url: 'git@192.168.8.8:dev/spring-boot-project.git', branch: "${BRANCH}", credentialsId: 'gitlab-key')script {COMMIT_ID = sh(returnStdout: true, script: "git log -n 1 --pretty=format:'%h'").trim()TAG = BUILD_TAG + '-' + COMMIT_IDprintln "Current branch is ${BRANCH}, Commit ID is ${COMMIT_ID}, Image TAG is ${TAG}"}}}stage('Pulling Code by trigger') {when {expression {env.gitlabBranch != null}}steps {git(url: 'git@192.168.8.8:dev/spring-boot-project.git', branch: env.gitlabBranch, changelog: true, poll: true, credentialsId: 'gitlab-key')script {COMMIT_ID = sh(returnStdout: true, script: "git log -n 1 --pretty=format:'%h'").trim()TAG = BUILD_TAG + '-' + COMMIT_IDprintln "Current branch is ${BRANCH}, Commit ID is ${COMMIT_ID}, Image TAG is ${TAG}"}}}}}stage('Building') {steps {container(name: 'build') {sh """mvn clean install -DskipTests"""}}}stage('Docker build for creating image') {environment {HARBOR_USER     = credentials('HARBOR_ACCOUNT')}steps {container(name: 'docker') {sh """echo ${HARBOR_USER_USR} ${HARBOR_USER_PSW} ${TAG}docker build -t ${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG} .docker login -u ${HARBOR_USER_USR} -p ${HARBOR_USER_PSW} ${HARBOR_ADDRESS}docker push ${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG}"""}}}stage('Deploying to K8s') {environment {MY_KUBECONFIG = credentials('study-k8s-kubeconfig')}steps {container(name: 'kubectl'){sh """/usr/local/bin/kubectl --kubeconfig $MY_KUBECONFIG set image deploy -l app=${IMAGE_NAME} ${IMAGE_NAME}=${HARBOR_ADDRESS}/${REGISTRY_DIR}/${IMAGE_NAME}:${TAG} -n $NAMESPACE"""}}}}environment {COMMIT_ID = ""HARBOR_ADDRESS = "192.168.8.101"REGISTRY_DIR = "dev"IMAGE_NAME = "spring-boot-project"NAMESPACE = "dev"TAG = ""}parameters {gitParameter(branch: '', branchFilter: 'origin/(.*)', defaultValue: '', description: 'Branch for build and deploy', name: 'BRANCH', quickFilterEnabled: false, selectedValue: 'NONE', sortMode: 'NONE', tagFilter: '*', type: 'PT_BRANCH')}
}