微服务项目部署流水线编写

1. 部署环境说明

序号	管理地址	作用
1	192.168.31.199	GitLab
2	192.168.31.104	Harbor
3	192.168.31.131	kubesphere

1.1 GitLab

请添加图片描述

1.2 流水线

1.2.1 创建流水线

请添加图片描述

1.2.2 创建凭证

请添加图片描述

1.2.3 创建kubeconfig凭证

请添加图片描述

这里需要注意的是,config中如果使用的是域名,那么需要想办法把他解析或者改为ip地址

1.2.4 编辑流水线

[创建自定义流水线]
请添加图片描述

[确定],[确定],[确定]

这样一个简单的拉取代码已经完成.

请添加图片描述

运行

请添加图片描述

1.3 集成Sonarqube

部署步骤参见 https://blog.csdn.net/qq_29974229/article/details/129829094 第二章

token

4b0a43841b5feb73fcef631c374e9dadc0babad3

2. 参数化构建

在Jenkinsfile中追加

parameters {
    string(name: 'PROJECT_VERSION', defaultValue: 'v1.0', description: '')
    string(name: 'PROJECT_NAME', defaultValue: '', description: '')
  }

请添加图片描述

此时运行后会弹出窗口

请添加图片描述

3. 添加凭证

3.1 密码凭证和kubeconfig

添加harbor和gitlab的密码凭证及kubeconfig

请添加图片描述

3.2 Sonarqube凭证

93617fa07455d52234720d122f9179208ee56a95

请添加图片描述

创建Sonarqube凭证

请添加图片描述

创建

3.3 Webhook

获得的端口是30180,其实就是devops-jenkins的nodeport端口

kubectl get -n kubesphere-devops-system svc devops-jenkins -o jsonpath="{.spec.ports[0].nodePort}"

请添加图片描述

webhook地址为:

http://192.168.31.131:30180/sonarqube-webhook/

3.4 将webhook集成到ks-install

kubectl edit cc -n kubesphere-system ks-installer

确认token和url地址正确

    sonarqube:
      externalSonarToken: 93617fa07455d52234720d122f9179208ee56a95
      externalSonarUrl: http://192.168.31.131:30322

3.5 将Sonarqube添加到Jenkins

进入Jenkins,[系统管理] [Manage Credentials] [Jenkins] [全局凭证] [添加凭证]

请添加图片描述

确认

请添加图片描述

[系统管理] [系统配置] [SonarQube Server]

URL中填写sonarqube的管理页面访问地址

请添加图片描述

3.6 SonarQube URL添加到Kubesphere

kubectl edit cm -n kubesphere-system ks-console-config

添加3行内容

apiVersion: v1
data:
  local_config.yaml: |
    server:
      http:
        hostname: localhost
        port: 8000
        static:
          production:
            /public: server/public
            /assets: dist/assets
            /dist: dist
      redis:
        port: 6379
        host: redis.kubesphere-system.svc
      redisTimeout: 5000
      sessionTimeout: 7200000
    client:
      version:
        kubesphere: v3.1.0
        kubernetes: v1.21.5
        openpitrix: v3.1.0
      enableKubeConfig: true
      # 添加以下2行内容
      devops:
        sonarqubeURL: http://192.168.31.131:30322
    defaultClusterName: default
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: ks-core
    meta.helm.sh/release-namespace: kubesphere-system
  creationTimestamp: "2023-03-22T06:40:55Z"
  labels:
    app.kubernetes.io/managed-by: Helm
  name: ks-console-config
  namespace: kubesphere-system
  resourceVersion: "153798"
  uid: 4f928d03-28e4-42c6-89de-d61f0347a101
~

3.7 重启Devops服务

kubectl -n kubesphere-devops-system rollout restart deployment devops-apiserver
kubectl -n kubesphere-system rollout restart deployment ks-console
kubectl rollout restart deployment -n kubesphere-devops-system

4. 添加环境变量到Jenkinsfile

将以下配置加到pipeline中stages的同级

  environment {
    DOCKER_CREDENTIAL_ID = 'harbor'
    GITEE_CREDENTIAL_ID = 'gitlab'
    KUBECONFIG_CREDENTIAL_ID= 'sangomall-kubeconfig'
    REGISTRY = '192.168.31.104'
    DOCKERHUB_NAMESPACE = 'sangomall'
    GITEE_ACCOUNT = 'sangomall'
    SONAR_CREDENTIAL_ID = 'sonar-qube'
  }

添加后的pipeline

pipeline {
  agent {
    node {
      label 'maven'
    }
  }
  parameters {
    string(name: 'PROJECT_VERSION', defaultValue: 'v1.0', description: '')
    string(name: 'PROJECT_NAME', defaultValue: '', description: '')
  }
  stages {
    stage('代码拉取') {
      agent none
      steps {
        git(url: 'http://192.168.31.199/deploy/test.git', credentialsId: 'gitlab', branch: 'master', changelog: true, poll: false)
      }
    }
  }
  environment {
    DOCKER_CREDENTIAL_ID = 'harbor'
    GITEE_CREDENTIAL_ID = 'gitlab'
    KUBECONFIG_CREDENTIAL_ID= 'sangomall-kubeconfig'
    REGISTRY = '192.168.31.104'
    DOCKERHUB_NAMESPACE = 'sangomall'
    GITEE_ACCOUNT = 'sangomall'
    SONAR_CREDENTIAL_ID = 'sonar-qube'
  }
}

5. 代码扫描

添加[指定容器] maven

[添加凭证] SONAR_TOKEN

请添加图片描述

[添加嵌套步骤] [Sonarqube配置] sonar

请添加图片描述

[添加嵌套步骤] [shell]

mvn sonar:sonar -gs `pwd`/mvn_settings.xml -Dsonar.login=$SONAR_TOKEN

请添加图片描述

mvn_settings.xml的内容

<settings>
    <mirrors>
        <mirror>
            <id>nexus-aliyun</id>
            <mirrorOf>central</mirrorOf>
            <name>Nexus aliyun</name>
            <url>http://maven.aliyun.com/nexus/content/groups/public</url>
        </mirror>
    </mirrors>
</settings>

添加超时

请添加图片描述

在超时后面添加代码质量检查

请添加图片描述

Jenkinsfile

pipeline {
  agent {
    node {
      label 'maven'
    }
  }
  stages {
    stage('代码拉取') {
      agent none
      steps {
        git(url: 'http://192.168.31.199/deploy/test.git', credentialsId: 'gitlab', branch: 'master', changelog: true, poll: false)
      }
    }
    stage('代码分析') {
      agent none
      steps {
        container('maven') {
          withCredentials([string(credentialsId : 'sonar-qube' ,variable : 'SONAR_TOKEN' ,)]) {
            withSonarQubeEnv('sonar') {
              sh 'mvn verify sonar:sonar -gs `pwd`/mvn_settings.xml -Dsonar.login=$SONAR_TOKEN'
            }
          }
          timeout(unit: 'MINUTES', activity: true, time: 30) {
            waitForQualityGate 'true'
          }
        }
      }
    }
  }
  environment {
    DOCKER_CREDENTIAL_ID = 'harbor'
    GITEE_CREDENTIAL_ID = 'gitlab'
    KUBECONFIG_CREDENTIAL_ID = 'sangomall-kubeconfig'
    REGISTRY = '192.168.31.104'
    DOCKERHUB_NAMESPACE = 'sangomall'
    GITEE_ACCOUNT = 'sangomall'
    SONAR_CREDENTIAL_ID = 'sonar-qube'
  }
  parameters {
    string(name: 'PROJECT_VERSION', defaultValue: 'v1.0', description: '')
    string(name: 'PROJECT_NAME', defaultValue: '', description: '')
  }
}

执行效果

请添加图片描述

此时sonarqube上可以看到sangomall的项目扫描结果

请添加图片描述

这里可能会因为代码质量扫描异常,可以调整Sonarqube质量阈来实现扫描通过

请添加图片描述

6. 容器构建

执行构建的前提是:

jar包(mvn生成)
Dockerfile
harbor仓库
harbor凭证

6.1 构建镜像

制作jar包

mvn clean package -Dmaven.test.skip=true -gs `pwd`/mvn_settings.xml

镜像制作

cd $PROJECT_NAME && docker build -f Dockerfile -t $REGISTRY/$DOCKERHUB_NAMESPACE/$PROJECT_NAME:SANPSHOT-$BUILD_NUMBER .

6.2 镜像上传到仓库

添加凭证,这里要注意下,上面是密码,下面是用户名

HARBOR_PASSWORD
HARBOR_USERNAME

请添加图片描述

登录harbor

echo "$HARBOR_PASSWORD" | docker login $REGISTRY -u "$HARBOR_USERNAME" --password-stdin

推送镜像

docker tag $REGISTRY/$DOCKERHUB_NAMESPACE/$PROJECT_NAME:SANPSHOT-$BUILD_NUMBER $REGISTRY/$DOCKERHUB_NAMESPACE/$PROJECT_NAME:latest
docker push $REGISTRY/$DOCKERHUB_NAMESPACE/$PROJECT_NAME:SANPSHOT-$BUILD_NUMBER
docker push $REGISTRY/$DOCKERHUB_NAMESPACE/$PROJECT_NAME:latest

请添加图片描述

6.3 Harbor登录失败处理

这里可能会出现maven镜像登录harbor失败.在ks-node节点的/etc/systemd/system/docker.service中添加以下配置

ExecStart=/usr/bin/dockerd  --insecure-registry 192.168.31.104 --insecure-registry harbor.intra.com

配置添加后需要重启docker服务

systemctl daemon-reload && systemctl restart docker

6.4 运行构建

请添加图片描述

Jenkinsfile

pipeline {
  agent {
    node {
      label 'maven'
    }

  }
  stages {
    stage('代码拉取') {
      agent none
      steps {
        git(url: 'http://192.168.31.199/deploy/sangomall.git', credentialsId: 'gitlab', branch: 'master', changelog: true, poll: false)
      }
    }

    stage('代码分析') {
      agent none
      steps {
        container('maven') {
          withCredentials([string(credentialsId : 'sonar-qube' ,variable : 'SONAR_TOKEN' ,)]) {
            withSonarQubeEnv('sonar') {
              sh 'mvn verify sonar:sonar -gs `pwd`/mvn_settings.xml -Dsonar.login=$SONAR_TOKEN'
            }

          }

          timeout(unit: 'HOURS', activity: true, time: 1) {
            waitForQualityGate 'true'
          }

        }

      }
    }

    stage('容器构建') {
      agent none
      steps {
        container('maven') {
          sh 'mvn clean install -Dmaven.test.skip=true -gs `pwd`/mvn_settings.xml'
          sh 'cd $PROJECT_NAME && docker build -f Dockerfile -t $REGISTRY/$DOCKERHUB_NAMESPACE/$PROJECT_NAME:SANPSHOT-$BUILD_NUMBER .'
          withCredentials([usernamePassword(credentialsId : 'harbor' ,passwordVariable : 'HARBOR_PASSWORD' ,usernameVariable : 'HARBOR_USERNAME' ,)]) {
            sh 'echo "$HARBOR_PASSWORD" | docker login $REGISTRY -u "$HARBOR_USERNAME" --password-stdin'
            sh 'docker tag $REGISTRY/$DOCKERHUB_NAMESPACE/$PROJECT_NAME:SANPSHOT-$BUILD_NUMBER $REGISTRY/$DOCKERHUB_NAMESPACE/$PROJECT_NAME:latest'
            sh 'docker push $REGISTRY/$DOCKERHUB_NAMESPACE/$PROJECT_NAME:SANPSHOT-$BUILD_NUMBER'
            sh 'docker push $REGISTRY/$DOCKERHUB_NAMESPACE/$PROJECT_NAME:latest'
          }

        }

      }
    }

  }
  environment {
    DOCKER_CREDENTIAL_ID = 'harbor'
    GITEE_CREDENTIAL_ID = 'gitlab'
    KUBECONFIG_CREDENTIAL_ID = 'sangomall-kubeconfig'
    REGISTRY = 'harbor.intra.com'
    DOCKERHUB_NAMESPACE = 'sangomall'
    GITEE_ACCOUNT = 'sangomall'
    SONAR_CREDENTIAL_ID = 'sonar-qube'
  }
  parameters {
    string(name: 'PROJECT_VERSION', defaultValue: 'v1.0', description: '')
    string(name: 'PROJECT_NAME', defaultValue: 'mall-gateway', description: '')
  }
}

同时在ks-node节点中可以看到相应镜像的生成,这取决于maven运行在哪个节点上.

root@ks-node4:~# docker images|grep mall
harbor.intra.com/sangomall/mall-gateway                                    SANPSHOT-83   09d37b7fdbe9   About a minute ago   571MB
harbor.intra.com/sangomall/mall-gateway                                    latest        09d37b7fdbe9   About a minute ago   571MB
harbor.intra.com/sangomall/mall-gateway                                    SANPSHOT-81   0c988d315159   11 minutes ago       571MB
harbor.intra.com/sangomall/mall-gateway                                    SANPSHOT-80   2ea22f1109a3   17 minutes ago       571MB
harbor.intra.com/sangomall/mall-gateway                                    SANPSHOT-78   26bc7c406339   41 minutes ago       571MB
harbor.intra.com/sangomall/mall-gateway                                    SANPSHOT-76   974154795e4a   About an hour ago    571MB
harbor.intra.com/sangomall/mall-gateway                                    SANPSHOT-72   a3eba99f1956   2 hours ago          571MB
harbor.intra.com/sangomall/mall-gateway                                    SANPSHOT-70   3f792a1b690d   2 hours ago          571MB
harbor.intra.com/sangomall/mall-order                                      SANPSHOT-68   15c9899f6e4d   2 hours ago          610MB
harbor.intra.com/sangomall/mall-gateway                                    SANPSHOT-67   78eca4b72558   2 hours ago          571MB
registry.cn-shanghai.aliyuncs.com/kbsphere/mall-gateway                    SANPSHOT-64   6697c227d030   42 hours ago         571MB
registry.cn-shanghai.aliyuncs.com/kbsphere/mall-gateway                    SANPSHOT-62   4d6db79b8400   42 hours ago         571MB
registry.cn-shanghai.aliyuncs.com/kbsphere/mall-gateway                    SANPSHOT-59   eb5d1a775497   42 hours ago         571MB
registry.cn-shanghai.aliyuncs.com/kbsphere/mall-gateway                    SANPSHOT-55   58c4ef9f7571   42 hours ago         571MB
harbor.intra.com/sangomall/mall-gateway                                    SANPSHOT-52   ecff5a92358f   43 hours ago         571MB
192.168.31.104/sangomall/mall-gateway                                      SANPSHOT-50   056fc1fb7b3d   43 hours ago         571MB

7. 项目Release

7.1 创建TAG判断

return params.PROJECT_VERSION =~ /V.*/

请添加图片描述

@project-admin 
是否允许推送本次项目代码及容器镜像

请添加图片描述

添加凭证

GITLAB_PASSWORD
GITLAB_USERNAME

请添加图片描述

添加shell命令

git config --global user.email "project-admin@intra.com"
git config --global user.name "project-admin"

添加TAG(shell命令)

git tag -a $PROJECT_VERSION -m "$PROJECT_VERSION"

上传

git push http://$GITLAB_USERNAME:$GITLAB_PASSWORD@gitlab.intra.com/deploy/sangomall.git --tags --ipv4

7.2 添加harbor

添加凭证

HARBOR_PASSWORD
HARBOR_USERNAME

镜像tag

docker tag $REGISTRY/$DOCKERHUB_NAMESPACE/$PROJECT_NAME:SANPSHOT-$BUILD_NUMBER $REGISTRY/$DOCKERHUB_NAMESPACE/$PROJECT_NAME:$PROJECT_VERSION
docker push $REGISTRY/$DOCKERHUB_NAMESPACE/$PROJECT_NAME:$PROJECT_VERSION

请添加图片描述

Jenkinsfile

pipeline {
  agent {
    node {
      label 'maven'
    }

  }
  stages {
    stage('代码拉取') {
      agent none
      steps {
        git(url: 'http://192.168.31.199/deploy/sangomall.git', credentialsId: 'gitlab', branch: 'master', changelog: true, poll: false)
      }
    }

    stage('代码分析') {
      agent none
      steps {
        container('maven') {
          withCredentials([string(credentialsId : 'sonar-qube' ,variable : 'SONAR_TOKEN' ,)]) {
            withSonarQubeEnv('sonar') {
              sh 'mvn verify sonar:sonar -gs `pwd`/mvn_settings.xml -Dsonar.login=$SONAR_TOKEN'
            }

          }

          timeout(unit: 'HOURS', activity: true, time: 1) {
            waitForQualityGate 'true'
          }

        }

      }
    }

    stage('容器构建') {
      agent none
      steps {
        container('maven') {
          sh 'mvn clean install -Dmaven.test.skip=true -gs `pwd`/mvn_settings.xml'
          sh 'cd $PROJECT_NAME && docker build -f Dockerfile -t $REGISTRY/$DOCKERHUB_NAMESPACE/$PROJECT_NAME:SANPSHOT-$BUILD_NUMBER .'
          withCredentials([usernamePassword(credentialsId : 'harbor' ,passwordVariable : 'HARBOR_PASSWORD' ,usernameVariable : 'HARBOR_USERNAME' ,)]) {
            sh 'echo "$HARBOR_PASSWORD" | docker login $REGISTRY -u "$HARBOR_USERNAME" --password-stdin'
            sh 'docker tag $REGISTRY/$DOCKERHUB_NAMESPACE/$PROJECT_NAME:SANPSHOT-$BUILD_NUMBER $REGISTRY/$DOCKERHUB_NAMESPACE/$PROJECT_NAME:latest'
            sh 'docker push $REGISTRY/$DOCKERHUB_NAMESPACE/$PROJECT_NAME:latest'
          }

        }

      }
    }

    stage('TAG定义') {
      agent none
      when {
        expression {
          return params.PROJECT_VERSION =~ /V.*/
        }

      }
      steps {
        container('maven') {
          input(message: '''@project-admin  
是否允许推送本次项目代码及容器镜像''', submitter: 'project-admin')
          withCredentials([usernamePassword(credentialsId : 'gitlab' ,passwordVariable : 'GITLAB_PASSWORD' ,usernameVariable : 'GITLAB_USERNAME' ,)]) {
            sh 'git config --global user.email "project-admin@intra.com"'
            sh 'git config --global user.name "project-admin"'
            sh 'git tag -a $PROJECT_VERSION -m "$PROJECT_VERSION"'
            sh 'git push http://$GITLAB_USERNAME:$GITLAB_PASSWORD@192.168.31.199/deploy/sangomall.git --tags --ipv4'
          }

          withCredentials([usernamePassword(credentialsId : 'harbor' ,passwordVariable : 'HARBOR_PASSWORD' ,usernameVariable : 'HARBOR_USERNAME' ,)]) {
            sh 'docker tag $REGISTRY/$DOCKERHUB_NAMESPACE/$PROJECT_NAME:SANPSHOT-$BUILD_NUMBER $REGISTRY/$DOCKERHUB_NAMESPACE/$PROJECT_NAME:$PROJECT_VERSION'
            sh 'docker push $REGISTRY/$DOCKERHUB_NAMESPACE/$PROJECT_NAME:$PROJECT_VERSION'
          }

        }

      }
    }

  }
  environment {
    DOCKER_CREDENTIAL_ID = 'harbor'
    GITEE_CREDENTIAL_ID = 'gitlab'
    KUBECONFIG_CREDENTIAL_ID = 'sangomall-kubeconfig'
    REGISTRY = 'harbor.intra.com'
    DOCKERHUB_NAMESPACE = 'sangomall'
    GITEE_ACCOUNT = 'sangomall'
    SONAR_CREDENTIAL_ID = 'sonar-qube'
  }
  parameters {
    string(name: 'PROJECT_VERSION', defaultValue: 'V1.0', description: '')
    string(name: 'PROJECT_NAME', defaultValue: 'mall-gateway', description: '')
  }
}

7.3 运行构建

请添加图片描述

project-admin审批

请添加图片描述

审批后构筑继续

请添加图片描述

至此CI部分已经完成

8. 部署项目到kubesphere

8.1 添加审核

这步如果需要就增加,实际生产环境中不一定需要

@project-admin 
是否允许部署到K8s环境

8.2 部署容器

添加kubeconfig的凭证

KUBECONFIG_CONTENT

伸缩

mkdir ~/.kube && echo "$KUBECONFIG_CONTENT" > ~/.kube/config && envsubst < $PROJECT_NAME/deploy/deploy.yaml | kubectl apply -f -

请添加图片描述

至此CD部分也完成了

9. JenkinsFile

调整parameters和env位置

将以下内容添加到gitlab sangomall项目下的Jenkinsfile文件中

pipeline {
  agent {
    node {
      label 'maven'
    }

  }
  parameters {
    string(name: 'PROJECT_VERSION', defaultValue: 'V1.0', description: '')
    string(name: 'PROJECT_NAME', defaultValue: 'mall-gateway', description: '')
  }
  environment {
    DOCKER_CREDENTIAL_ID = 'harbor'
    GITEE_CREDENTIAL_ID = 'gitlab'
    KUBECONFIG_CREDENTIAL_ID = 'sangomall-kubeconfig'
    REGISTRY = 'harbor.intra.com'
    DOCKERHUB_NAMESPACE = 'sangomall'
    GITEE_ACCOUNT = 'sangomall'
    SONAR_CREDENTIAL_ID = 'sonar-qube'
  }
  stages {
    stage('代码拉取') {
      agent none
      steps {
        git(url: 'http://192.168.31.199/deploy/sangomall.git', credentialsId: 'gitlab', branch: 'master', changelog: true, poll: false)
      }
    }

    stage('代码分析') {
      agent none
      steps {
        container('maven') {
          withCredentials([string(credentialsId : 'sonar-qube' ,variable : 'SONAR_TOKEN' ,)]) {
            withSonarQubeEnv('sonar') {
              sh 'mvn verify sonar:sonar -gs `pwd`/mvn_settings.xml -Dsonar.login=$SONAR_TOKEN'
            }

          }

          timeout(unit: 'HOURS', activity: true, time: 1) {
            waitForQualityGate 'true'
          }

        }

      }
    }

    stage('容器构建') {
      agent none
      steps {
        container('maven') {
          sh 'mvn clean install -Dmaven.test.skip=true -gs `pwd`/mvn_settings.xml'
          sh 'cd $PROJECT_NAME && docker build -f Dockerfile -t $REGISTRY/$DOCKERHUB_NAMESPACE/$PROJECT_NAME:SANPSHOT-$BUILD_NUMBER .'
          withCredentials([usernamePassword(credentialsId : 'harbor' ,passwordVariable : 'HARBOR_PASSWORD' ,usernameVariable : 'HARBOR_USERNAME' ,)]) {
            sh 'echo "$HARBOR_PASSWORD" | docker login $REGISTRY -u "$HARBOR_USERNAME" --password-stdin'
            sh 'docker tag $REGISTRY/$DOCKERHUB_NAMESPACE/$PROJECT_NAME:SANPSHOT-$BUILD_NUMBER $REGISTRY/$DOCKERHUB_NAMESPACE/$PROJECT_NAME:latest'
            sh 'docker push $REGISTRY/$DOCKERHUB_NAMESPACE/$PROJECT_NAME:latest'
          }

        }

      }
    }

    stage('TAG定义') {
      agent none
      when {
        expression {
          return params.PROJECT_VERSION =~ /V.*/
        }

      }
      steps {
        container('maven') {
          input(message: '''@project-admin  
是否允许推送本次项目代码及容器镜像''', submitter: 'project-admin')
          withCredentials([usernamePassword(credentialsId : 'gitlab' ,passwordVariable : 'GITLAB_PASSWORD' ,usernameVariable : 'GITLAB_USERNAME' ,)]) {
            sh 'git config --global user.email "project-admin@intra.com"'
            sh 'git config --global user.name "project-admin"'
            sh 'git tag -a $PROJECT_VERSION -m "$PROJECT_VERSION"'
            sh 'git push http://$GITLAB_USERNAME:$GITLAB_PASSWORD@192.168.31.199/deploy/sangomall.git --tags --ipv4'
          }

          withCredentials([usernamePassword(credentialsId : 'harbor' ,passwordVariable : 'HARBOR_PASSWORD' ,usernameVariable : 'HARBOR_USERNAME' ,)]) {
            sh 'docker tag $REGISTRY/$DOCKERHUB_NAMESPACE/$PROJECT_NAME:SANPSHOT-$BUILD_NUMBER $REGISTRY/$DOCKERHUB_NAMESPACE/$PROJECT_NAME:$PROJECT_VERSION'
            sh 'docker push $REGISTRY/$DOCKERHUB_NAMESPACE/$PROJECT_NAME:$PROJECT_VERSION'
          }

        }

      }
    }

    stage('部署到Kubesphere') {
      agent none
      steps {
        input(message: '''@project-admin  
是否允许部署到K8s环境''', submitter: 'project-admin')
        container('maven') {
          withCredentials([kubeconfigContent(credentialsId : 'sangomall-kubeconfig' ,variable : 'KUBECONFIG_CONTENT' ,)]) {
            sh 'mkdir ~/.kube && echo "$KUBECONFIG_CONTENT" > ~/.kube/config && envsubst < $PROJECT_NAME/deploy/deploy.yaml | kubectl apply -f -'
          }

        }

      }
    }

  }
}