Kubernetes实战——DevOps集成SpringBoot项目

一、安装Gitlab

1、安装并配置Gitlab

1.1 、下载安装包

1.2、安装

1.3、修改配置文件

1.4、更新配置并重启

2、配置

2.1、修改密码

2.2、禁用注册功能

2.3、取消头像

2.4、修改中文配置

2.5、配置 webhook

3、卸载

二、安装镜像私服Harbor

1、下载安装包

2、安装

三、安装代码质量扫描工具sonarqube

1、pgsql.yaml

2、sonarqube.yaml

3、安装

四、安装Jenkins

1、构建带有maven和sonarqube的镜像

2、安装jenkins

2.1、创建Harbor secret

2.2、 jenkins-configmap.yaml

2.3、jenkins-pvc.yaml

2.4、jenkins-serviceAccount.yaml

2.5、jenkins-deployment.yaml

2.6、jenkins-service.yaml

2.7、创建资源

3、配置

3.1 、获取初始密码

3.2、安装插件

3.2.1、Build Authorization Token Root

3.2.2、Gitlab

3.2.3、SonarQube Scanner

3.2.4、Node and Label parameter

3.2.5、Kubernetes

3.2.6、Config File Provider

3.2.7、Git Parameter

3.3、插件配置

3.3.1、SonarQube Scanner配置

3.3.2、kubernetes配置

3.3.3、创建gitlab凭证

五、构建项目

1、创建gitlab的secret

2、配置jenkins任务

2.1、创建流水线Job

2.2、配置流水线编辑

2.3、配置gitlab的Webhooks编辑

2.4、测试构建

2.5、测试

3、构建SpringBoot项目

3.1、配置项目流水线

3.2、创建镜像私服全局凭证

3.3、创建kubeconfig文件

3.4、创建sonarqube的webhook

3.5、 cicd-demo.yaml配置文件

3.6、cicd-demo-dev.yaml

3.7、Dockerfile

3.8、Jenkinsfile

3.9、完整代码见附录

3.10、部署成功效果

六、附录

1、Gitlab安装包

2、Harbor安装包

3、带有Maven的Jenkins镜像

4、源码地址

一、安装Gitlab

1、安装并配置Gitlab

这里使用安装包的方式安装，如果需要Docker方式安装请点击这里

1.1 、下载安装包

wget https://mirrors.tuna.tsinghua.edu.cn/gitlab-ce/yum/el7/gitlab-ce-15.9.1-ce.0.el7.x86_64.rpm

1.2、安装

rpm -i gitlab-ce-15.9.1-ce.0.el7.x86_64.rpm

1.3、修改配置文件

vim /etc/gitlab/gitlab.rb

#修改 external_url 访问路径
http://<ip>:<port>
例如我这里改为 
external_url 'http://192.168.139.184:9000'

# 修改时区
gitlab_rails['time_zone'] = 'Asia/Shanghai'

1.4、更新配置并重启

gitlab-ctl reconfigure
gitlab-ctl restart

2、配置

2.1、修改密码

访问上面配置的external_url http://192.168.139.184:9000/

查看默认密码

cat /etc/gitlab/initial_root_password

默认账号是root 密码为查看到密码

右上角头像 > Perferences > Password

2.2、禁用注册功能

点击左上角三横 > Admin>Settings > General > Sign-up restrictions > 取消 Sign-up enabled > Save changes

2.3、取消头像

Settings > General > Account and limit > 取消 Gravatar enabled > Save changes

2.4、修改中文配置

Settings > Preferences > Localization > Default language > 选择简体中文 > Save changes

为当前用户设置中文，保存后刷新页面即可

右上角用户头像 > Preferences > Localization > Language > 选择简体中文 > Save changes

2.5、配置 webhook

设置>网络>出站请求>勾选>保存

3、卸载

# 停止服务
gitlab-ctl stop

# 卸载 rpm 软件（注意安装的软件版本是 ce 还是 ee）
rpm -e gitlab-ce

# 查看进程
ps -ef|grep gitlab 
# 干掉第一个 runsvdir -P /opt/gitlab/service log 进程

# 删除 gitlab 残余文件
find / -name *gitlab* | xargs rm -rf
find / -name gitlab | xargs rm -rf

二、安装镜像私服Harbor

1、下载安装包

#下载安装包
wget https://github.com/goharbor/harbor/releases/download/v1.10.19/harbor-offline-installer-v1.10.19.tgz

#解压
tar -zxf harbor-offline-installer-v1.10.19.tgz

2、安装

修改配置文件

vim harbor.yml

安装

 ./install.sh

访问http://192.168.139.184:8899/ 默认账号为admin 密码为Harbor12345

配置docker文件，并重启docker服务。

{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "registry-mirrors": [
        "https://do.nark.eu.org",
        "https://dc.j8.work",
        "https://docker.m.daocloud.io",
        "https://dockerproxy.com",
        "https://docker.mirrors.ustc.edu.cn",
        "https://docker.nju.edu.cn"
    ],
"insecure-registries": ["http://192.168.139.184:8899"]  #新增的配置
}

三、安装代码质量扫描工具sonarqube

1、pgsql.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-data
  namespace: kube-devops
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: "manager-nfs-storage"  #前面安装的storageclass
  resources:
    requests:
      storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres-sonar
  namespace: kube-devops
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres-sonar
  template:
    metadata:
      labels:
        app: postgres-sonar
    spec:
      containers:
      - name: postgres-sonar
        image: postgres:14.2
        ports:
        - containerPort: 5432
        env:
        - name: POSTGRES_DB
          value: "sonarDB"
        - name: POSTGRES_USER
          value: "sonarUser"
        - name: POSTGRES_PASSWORD 
          value: "123456"
        volumeMounts:
          - name: data
            mountPath: /var/lib/postgresql/data
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: postgres-data
---
apiVersion: v1
kind: Service
metadata:
  name: postgres-sonar
  namespace: kube-devops
  labels:
    app: postgres-sonar
spec:
  type: NodePort
  ports:
  - name: postgres-sonar
    port: 5432
    targetPort: 5432
    protocol: TCP
  selector:
    app: postgres-sonar

2、sonarqube.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: sonarqube-data
  namespace: kube-devops
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: "manager-nfs-storage"
  resources:
    requests:
      storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sonarqube
  namespace: kube-devops
  labels:
    app: sonarqube
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sonarqube
  template:
    metadata:
      labels:
        app: sonarqube
    spec:
      initContainers:
      - name: init-sysctl
        image: busybox:1.28.4
        command: ["sysctl", "-w", "vm.max_map_count=262144"]
        securityContext:
          privileged: true
      containers:
      - name: sonarqube
        image: sonarqube:9.9-community
        ports:
        - containerPort: 9000
        env:
        - name: SONARQUBE_JDBC_USERNAME
          value: "sonarUser"
        - name: SONARQUBE_JDBC_PASSWORD
          value: "123456"
        - name: SONARQUBE_JDBC_URL
          value: "jdbc:postgresql://postgres-sonar:5432/sonarDB"
        livenessProbe:
          httpGet:
            path: /sessions/new
            port: 9000
          initialDelaySeconds: 60
          periodSeconds: 30
        readinessProbe:
          httpGet:
            path: /sessions/new
            port: 9000
          initialDelaySeconds: 60
          periodSeconds: 30
          failureThreshold: 6
        volumeMounts:
        - mountPath: /opt/sonarqube/conf
          name: data
        - mountPath: /opt/sonarqube/data
          name: data
        - mountPath: /opt/sonarqube/extensions
          name: data
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: sonarqube-data 
---
apiVersion: v1
kind: Service
metadata:
  name: sonarqube
  namespace: kube-devops
  labels:
    app: sonarqube
spec:
  type: NodePort
  ports:
  - name: sonarqube
    port: 9000
    targetPort: 9000
    protocol: TCP
  selector:
    app: sonarqube

3、安装

#创建资源
kubectl create -f pgsql.yaml -f sonarqube.yaml

#查看资源
kubectl get po,svc -n kube-devops -o wide

访问该端口下的服务，默认账号密码admin/admin

四、安装Jenkins

1、构建带有maven和sonarqube的镜像

Dockerfile

FROM jenkins/jenkins:jdk17
ADD ./apache-maven-3.9.9-bin.tar.gz /usr/local/
ADD ./sonar-scanner-cli-4.8.0.2856-linux.zip /usr/local/

USER root

WORKDIR /usr/local/
RUN unzip sonar-scanner-cli-4.8.0.2856-linux.zip
RUN mv sonar-scanner-4.8.0.2856-linux sonar-scanner-cli
RUN ln -s /usr/local/sonar-scanner-cli/bin/sonar-scanner /usr/bin/sonar-scanner

ENV MAVEN_HOME=/usr/local/apache-maven-3.9.9
ENV PATH=$JAVA_HOME/bin:$MAVEN_HOME/bin:$PATH

RUN echo "jenkins ALL=NOPASSWD: ALL" >> /etc/sudoers
USER jenkins

#构建镜像
docker build -t 192.168.139.184:8899/wsnail-harbor/jenkins-maven:jdk-17 .

#登录docker私服
docker login -uadmin 192.168.139.184:8899

#推送到仓库
docker push 192.168.139.184:8899/wsnail-harbor/jenkins-maven:jdk-17

注意：1、该操作和docker私服在同一台服务器上

2、请确认、etc/docker/damon.json文件配置了"insecure-registries": ["http://192.168.139.184:8899"]

2、安装jenkins

2.1、创建Harbor secret


kubectl create secret docker-registry harbor-secret --docker-server=192.168.139.184:8899 --docker-username=admin --docker-password=Xiaojie12345 -n kube-devops

2.2、 jenkins-configmap.yaml

apiVersion: v1 
kind: ConfigMap 
metadata: 
  name: mvn-settings
  namespace: kube-devops
  labels: 
    app: jenkins-server
data: 
  settings.xml: |- 
    <?xml version="1.0" encoding="UTF-8"?>
    <settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
        <localRepository>/var/jenkins_home/repository</localRepository>
         <servers>
        <server>
            <!--此处id需要对应项目中pom.xml文件中的<distributionManagement> 标签下的id-->
            <id>release</id>
            <username>admin</username>
            <password>123456</password>
        </server>
        <server>
            <id>snapshots</id>
            <username>admin</username>
            <password>123456</password>
        </server>
    </servers>

        <mirrors>
                <mirror>
                     <id>releases</id>
                     <name>nexus maven</name>
                     <mirrorOf>*</mirrorOf>
                     <url>http://192.168.139.184:8081/repository/maven-public/</url>
                </mirror>
                <mirror>
                     <id>aliMaven</id>
                     <name>aliyun maven</name>	  	  
                     <url>	https://maven.aliyun.com/repository/public</url>
                     <mirrorOf>central</mirrorOf>
                </mirror>
        </mirrors>

        <pluginGroups>
                <pluginGroup>org.sonarsource.scanner.maven</pluginGroup>
        </pluginGroups>
        <profiles>
                <profile>
                        <id>releases</id>
                        <activation>
                                <activeByDefault>true</activeByDefault>
                                <jdk>1.8</jdk>
                        </activation>
                        <properties>
                                <sonar.host.url>http://sonarqube:9000</sonar.host.url>
                        </properties>

                        <repositories>
                                <repository>
                                        <id>repository</id>
                                        <name>Nexus Repository</name>
                                        <url>http://192.168.139.184:8081/repository/snail-group/</url>   
                                        <releases>
                                                <enable>true</enable>
                                        </releases>
                                        <snapshots>
                                                <enable>true</enable>
                                        </snapshots>
                                </repository>
                        </repositories>
                </profile>
        </profiles>
    </settings>

2.3、jenkins-pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jenkins-pvc
  namespace: kube-devops
spec:
  storageClassName: manager-nfs-storage
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi

2.4、jenkins-serviceAccount.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins-admin
  namespace: kube-devops
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: jenkins-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: jenkins-admin
  namespace: kube-devops

2.5、jenkins-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins
  namespace: kube-devops
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jenkins-server
  template:
    metadata:
      labels:
        app: jenkins-server
    spec:
      serviceAccountName: jenkins-admin
      #Default: 继承节点的DNS配置；ClusterFirst: 使用coredns作为DNS配置；
      #ClusterFirstWithHostNet:当Pod.spec.hostNetwork=true时，Pod的DNS策略被强制转换为Default，即继承节点的DNS配置；
      #  若Pod要使用coredns作为DNS配置，则需配置pod.spec.dnsPolicy=ClusterFirstWithHostNet;
      #   None: 没有DNS配置；
      dnsPolicy: Default  #解决pod内没有DNS解析不到插件地址
      imagePullSecrets:
        - name: harbor-secret # harbor 访问 secret
      containers:
        - name: jenkins
          image: 192.168.139.184:8899/wsnail-harbor/jenkins-maven:jdk-17   #这里是自己镜像的地址
          imagePullPolicy: IfNotPresent
          securityContext:
            privileged: true
            runAsUser: 0 # 使用 root 用户运行容器
          resources:
            limits:
              memory: "2Gi"
              cpu: "1000m"
            requests:
              memory: "500Mi"
              cpu: "500m"
          ports:
            - name: httpport
              containerPort: 8080
            - name: jnlpport
              containerPort: 50000
          livenessProbe:
            httpGet:
              path: "/login"
              port: 8080
            initialDelaySeconds: 90
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 5
          readinessProbe:
            httpGet:
              path: "/login"
              port: 8080
            initialDelaySeconds: 60
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 3
          volumeMounts:
            - name: jenkins-data
              mountPath: /var/jenkins_home
            - name: docker
              mountPath: /run/docker.sock
            - name: docker-home
              mountPath: /usr/bin/docker
            - name: mvn-setting
              mountPath: /usr/local/apache-maven-3.9.9/conf/settings.xml
              subPath: settings.xml
            - name: daemon
              mountPath: /etc/docker/daemon.json
              subPath: daemon.json
            - name: kubectl
              mountPath: /usr/bin/kubectl
      volumes:
        - name: kubectl
          hostPath:
            path: /usr/bin/kubectl
        - name: jenkins-data
          persistentVolumeClaim:
              claimName: jenkins-pvc
        - name: docker
          hostPath:
            path: /run/docker.sock # 将主机的 docker 映射到容器中
        - name: docker-home
          hostPath:
            path: /usr/bin/docker
        - name: mvn-setting
          configMap:
            name: mvn-settings
            items:
            - key: settings.xml
              path: settings.xml
        - name: daemon
          hostPath: 
            path: /etc/docker/

2.6、jenkins-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: jenkins-service
  namespace: kube-devops
  annotations:
      prometheus.io/scrape: 'true'
      prometheus.io/path:   /
      prometheus.io/port:   '8080'
spec:
  selector:
    app: jenkins-server
  type: NodePort
  ports:
    - port: 8080
      targetPort: 8080

2.7、创建资源

#上面的配置文件均在manifests文件下
kubectl apply -f manifests/

#查看资源
kubectl get po,svc -n kube-devops

3、配置

3.1 、获取初始密码

 kubectl logs jenkins-598b49d974-glv5x -n kube-devops

3.2、安装插件

3.2.1、Build Authorization Token Root

3.2.2、Gitlab

3.2.3、SonarQube Scanner

3.2.4、Node and Label parameter

3.2.5、Kubernetes

3.2.6、Config File Provider

3.2.7、Git Parameter

3.3、插件配置

3.3.1、SonarQube Scanner配置

注意：如果不是k8s部署可以使用外网访问，ip:port

在SonarQube创建token

在jenkins中添加sonarqube的token

3.3.2、kubernetes配置

3.3.3、创建gitlab凭证

五、构建项目

1、创建gitlab的secret

后面构建项目时候需要用到

#用户名
echo root > ./username

#密码
echo xiaojie123456 > password

#创建secret
kubectl create secret generic git-user-pwd --from-file=./username --from-file=./password -n kube-devops

#查看
kubectl get  secret -n kube-devops

2、配置jenkins任务

2.1、创建流水线Job

2.2、配置流水线

2.3、配置gitlab的Webhooks

2.4、测试构建

2.5、测试

3、构建SpringBoot项目

3.1、配置项目流水线

3.2、创建镜像私服全局凭证

3.3、创建kubeconfig文件

cat ~/.kube/config

将文件内容复制

3.4、创建sonarqube的webhook

3.5、 cicd-demo.yaml配置文件

---
#开发环境的deploy
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: k8s-demo
    component: wssnail-devops
    tier: backend
  name: k8s-demo
  namespace: ks-k8s-demo
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  selector:
    matchLabels:
      app: k8s-demo
      component: wssnail-devops
      tier: backend
  strategy:
    rollingUpdate:
      maxSurge: 100%
      maxUnavailable: 100%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: k8s-demo
        component: wssnail-devops
        tier: backend
    spec:
      imagePullSecrets:
        - name: harbor-secret #该secret必须和该项目在同一个namespace下
      containers:
        - name: k8s-demo
          image: REGISTRY/DOCKERHUB_NAMESPACE/APP_NAME:SNAPSHOT-BUILD_NUMBER   #该内容会在构建镜像时自动替换为实际内容
          readinessProbe:
            httpGet:
              path: /index
              port: 8080
            timeoutSeconds: 10
            failureThreshold: 30
            periodSeconds: 5
          imagePullPolicy: Always
          ports:
            - containerPort: 8080
              protocol: TCP
          resources:
            limits:
              cpu: 300m
              memory: 600Mi
            requests:
              cpu: 100m
              memory: 100Mi
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: k8s-demo
    component: wssnail-devops
  name: k8s-demo
  namespace: ks-k8s-demo
spec:
  ports:
    - name: http
      port: 8080
      protocol: TCP
      targetPort: 8080
  selector:
    app: k8s-demo
    component: wssnail-devops
    tier: backend
  sessionAffinity: None
  type: NodePort

3.6、cicd-demo-dev.yaml

---
#开发环境的deploy
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: k8s-demo
    component: wssnail-devops
    tier: backend
  name: k8s-demo
  namespace: ks-k8s-demo
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  selector:
    matchLabels:
      app: k8s-demo
      component: wssnail-devops
      tier: backend
  strategy:
    rollingUpdate:
      maxSurge: 100%
      maxUnavailable: 100%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: k8s-demo
        component: wssnail-devops
        tier: backend
    spec:
      imagePullSecrets:
        - name: harbor-secret #该secret必须和该项目在同一个namespace下
      containers:
        - name: k8s-demo
          image: REGISTRY/DOCKERHUB_NAMESPACE/APP_NAME:SNAPSHOT-BUILD_NUMBER   #该内容会在构建镜像时自动替换为实际内容
          readinessProbe:
            httpGet:
              path: /index
              port: 8080
            timeoutSeconds: 10
            failureThreshold: 30
            periodSeconds: 5
          imagePullPolicy: Always
          ports:
            - containerPort: 8080
              protocol: TCP
          resources:
            limits:
              cpu: 300m
              memory: 600Mi
            requests:
              cpu: 100m
              memory: 100Mi
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: k8s-demo
    component: wssnail-devops
  name: k8s-demo
  namespace: ks-k8s-demo
spec:
  ports:
    - name: http
      port: 8080
      protocol: TCP
      targetPort: 8080
  selector:
    app: k8s-demo
    component: wssnail-devops
    tier: backend
  sessionAffinity: None
  type: NodePort

3.7、Dockerfile

## 基础镜像
FROM openjdk:17-slim

## 作者
LABEL org.opencontainers.image.authors="wssnail"

## 定义参数

## 创建并进入工作目录
RUN mkdir -p /wssnail
WORKDIR /wssnail

## maven 插件构建时得到 buildArgs 种的值
COPY target/*.jar app.jar

## 设置 TZ 时区
## 设置 JAVA_OPTS 环境变量，可通过 docker run -e "JAVA_OPTS=" 进行覆盖
ENV TZ=Asia/Shanghai JAVA_OPTS="-Xms256m -Xmx256m"

## 暴露端口
EXPOSE 8080

## 容器启动命令
## CMD 第一个参数之后的命令可以在运行时被替换
CMD java ${JAVA_OPTS} -Djava.security.egd=file:/dev/./urandom -jar app.jar

3.8、Jenkinsfile

pipeline {

  agent {
    node {
      label 'maven'  // 这里配置的标签和jenkins中配置节点添加的标签一致
    }
  }

  environment {
    REGISTRY = '192.168.139.184:8899'   //harbor地址
    DOCKER_CREDENTIAL_ID = 'harbor-user-pwd' //harbor凭证，对应jenkins创建凭证的id
    GIT_REPO_URL = '192.168.139.184:9000'    //gitlab地址
    GIT_CREDENTIAL_ID = 'git-user-pwd'   //gitlab凭证，对应jenkins创建凭证的id
    KUBECONFIG_CREDENTIAL_ID = '30629742-ddae-4c80-a3ad-5807ffc6ff5a'  //对应jenkins创建kubeconfig文件时的id
    DOCKERHUB_NAMESPACE = 'wsnail-harbor' //镜像私服的命名空间
    GITHUB_ACCOUNT = 'root'  //git账号
    APP_NAME = 'demo-k8s'  //应用名称
    SONAR_SERVER_URL='http://192.168.139.208:32061'  //sonarqube地址
    SONAR_CREDENTIAL_ID='sonarqube-token'  //jenkins创建的凭证
  }

  //拉取代码
  stages {

    stage('clone code') {
      steps {
        //配置代码仓库地址 ，credentialsId和jenkins创建的凭证的id保持一致
          sh 'echo  start pull code  start'
          git(url: 'http://192.168.139.184:9000/gitlab-instance-e9e80190/demo-k8s.git', credentialsId: 'git-user-pwd', branch: 'master', changelog: true, poll: false)
          sh 'echo  start pull code end'
        }
    }

    //单元测试
    stage('unit test') {
      steps {
          sh 'mvn clean test'
      }
    }
    //代码质量分析，如果jenkins配置的service连接不上，使用外网地址访问
    stage('sonarqube analysis') {
      agent none
      //withSonarQubeEnv()的值和jenkins中配置的sonarqube的名字保持一致
      steps {
        withCredentials([string(credentialsId : 'sonarqube-token' ,variable : 'SONAR_TOKEN' ,)]) {
          withSonarQubeEnv('sonarqube') {
              sh '''mvn sonar:sonar -Dsonar.projectKey=$APP_NAME
              echo "mvn sonar:sonar -Dsonar.projectKey=$APP_NAME"'''
          }

            //代码质量扫描等待时间
          timeout(unit: 'MINUTES', activity: true, time: 10) {
            waitForQualityGate 'true'
          }
        }
      }
    }

    //打包构建镜像
    stage('build & push') {
      steps {
          sh 'mvn clean package -DskipTests'
          sh 'docker build -f Dockerfile -t $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BUILD_NUMBER .'
          withCredentials([usernamePassword(credentialsId : 'harbor-user-pwd' ,passwordVariable : 'DOCKER_PASSWORD' ,usernameVariable : 'DOCKER_USERNAME' ,)]) {
            sh '''echo "$DOCKER_PASSWORD" | docker login $REGISTRY -u "$DOCKER_USERNAME" --password-stdin
            docker push $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BUILD_NUMBER'''
          }
      }
    }

    stage('push latest') {
      //是master 分支时，构建镜像的tag为latest保证上线的代码是最新的
      when {
        branch 'master'
      }
      steps {
          sh 'docker tag $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BUILD_NUMBER $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:latest'
          sh 'docker push $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:latest'
      }
    }

    stage('deploy to dev') {
    //开发分支时部署到开发分支，这里分支灵活配置，注意镜像地址要与主分支分开
      steps {
          input(id: 'deploy-to-dev', message: 'deploy to dev?')
            sh '''sed -i\'\' "s#REGISTRY#$REGISTRY#" deploy/cicd-demo-dev.yaml
            sed -i\'\' "s#DOCKERHUB_NAMESPACE#$DOCKERHUB_NAMESPACE#" deploy/cicd-demo-dev.yaml
            sed -i\'\' "s#APP_NAME#$APP_NAME#" deploy/cicd-demo-dev.yaml
            sed -i\'\' "s#BUILD_NUMBER#$BUILD_NUMBER#" deploy/cicd-demo-dev.yaml
            kubectl apply -f deploy/cicd-demo-dev.yaml'''

      }
    }

    stage('push with tag') {
    //为gitlab打tag
      agent none
      when {
        expression {
        //当匹配tag 为v开头的分支时
          params.TAG_NAME =~ /v.*/
        }

      }
      steps {
        input(message: 'release image with tag?', submitter: '')
        withCredentials([usernamePassword(credentialsId : 'git-user-pwd' ,passwordVariable : 'GIT_PASSWORD' ,usernameVariable : 'GIT_USERNAME' ,)]) {
          sh 'git config --global user.email "whisper_snail@163.com" '
          sh 'git config --global user.name "wssnail" '
          sh 'git tag -a $TAG_NAME -m "$TAG_NAME" '
          sh 'git push http://$GIT_USERNAME:$GIT_PASSWORD@$GIT_REPO_URL/gitlab-instance-e9e80190/demo-k8s.git --tags --ipv4'
        }
          sh 'docker tag $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BUILD_NUMBER $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:$TAG_NAME'
          sh 'docker push $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:$TAG_NAME'
      }
    }

//是否部署到生产环境
    stage('deploy to production') {
      agent none
      when {
        expression {
          params.TAG_NAME =~ /v.*/
        }

      }
      steps {
        input(message: 'deploy to production?', submitter: '')
          sh '''sed -i\'\' "s#REGISTRY#$REGISTRY#" deploy/cicd-demo.yaml
          sed -i\'\' "s#DOCKERHUB_NAMESPACE#$DOCKERHUB_NAMESPACE#" deploy/cicd-demo.yaml
          sed -i\'\' "s#APP_NAME#$APP_NAME#" deploy/cicd-demo.yaml
          sed -i\'\' "s#TAG_NAME#$TAG_NAME#" deploy/cicd-demo.yaml
          kubectl apply -f deploy/cicd-demo.yaml'''
      }
    }
  }

//参数选择
  parameters {
    string(name: 'BRANCH_NAME', defaultValue: 'master', description: '请选择要发布的分支')
    choice(name: 'NAMESPACE',choices:['dev','test','master'],description: '命名空间')
    string(name: 'TAG_NAME', defaultValue: 'snapshot', description: '标签名称，必须以 v 开头，例如：v1、v1.0.0')
  }

}