摆脱Jenkins - 使用google cloudbuild 部署 java service 到 compute engine VM

在之前介绍 cloud build 的文章中

初探 Google 云原生的CICD - CloudBuild

已经介绍过，用cloud build 去部署1个 spring boot service 到 cloud run 是很简单的，因为部署cloud run 无非就是用gcloud 去部署1个 GAR 上的docker image 到cloud run 容器
yaml file 例子：

steps:
  - id: check maven and jdk version
    name: maven:3.9.6-sapmachine-17 # https://hub.docker.com/_/maven
    entrypoint: mvn
    args: ['--version']

  - id: run maven test
    name: maven:3.9.6-sapmachine-17 # https://hub.docker.com/_/maven
    entrypoint: mvn
    args: ['test']

  - id: run maven package
    name: maven:3.9.6-sapmachine-17 # https://hub.docker.com/_/maven
    entrypoint: mvn
    args: ['package', '-Dmaven.test.skip=true']

  # https://cloud.google.com/build/docs/configuring-builds/substitute-variable-values
  - id: build docker image
    name: 'gcr.io/cloud-builders/docker'
    args: ['build', '-t', 'europe-west2-docker.pkg.dev/$PROJECT_ID/my-docker-repo/demo-cloud-user', '.']

  - id: upload docker image to GAR
    name: 'gcr.io/cloud-builders/docker'
    args: [ 'push', 'europe-west2-docker.pkg.dev/$PROJECT_ID/my-docker-repo/demo-cloud-user']

  # deploy to Cloud run
  - id: deploy image to cloud run
    name: 'gcr.io/cloud-builders/gcloud'
    args: ['run', 'deploy', 'demo-cloud-user',
           '--image=europe-west2-docker.pkg.dev/$PROJECT_ID/my-docker-repo/demo-cloud-user',
           '--port=8080',
           '--platform=managed',
           '--region=europe-west2',
           '--no-allow-unauthenticated',
           '--service-account=vm-common@jason-hsbc.iam.gserviceaccount.com',
           '--key=projects/$PROJECT_ID/locations/europe-west2/keyRings/mykeyring/cryptoKeys/mycmek']
# https://stackoverflow.com/questions/68779751/error-publishing-source-code-from-cloud-build-to-a-bucket-using-triggers
logsBucket: gs://jason-hsbc_cloudbuild/logs/
options: # https://cloud.google.com/cloud-build/docs/build-config#options
  logging: GCS_ONLY # or CLOUD_LOGGING_ONLY https://cloud.google.com/cloud-build/docs/build-config#logging

部署到GCE的问题

但是cloud build 本身是1个non-vpc product, 是无法直接通过 GCE vm的subnet ip address 去连接vm的.

但是
gcloud compute ssh 本身是可以直接用 ssh key file 验证的
而且
gcloud compute ssh 后面可以带 – 参数执行1段指定的命令

所以实际上不同过内网ip 连接. 那就是讲用cloud build 部署service 到 vm的方案是可行的

部署到GCE的思路

首先准备1对ssh key pair, 并把public key安装在对应的vm中，确保可以用private key登陆
把这对key pari 放在 google security manager 中
在 cloudbuild yaml 中把这对key pari 引入，虽然理论上只需要私钥就可以，但是cloudbuild 也需要校验public key 奇怪了

编写cloudbuild yaml

a. mvn build
b. build docker image
c. push docker image to GAR
d. 利用 gcloud compute ssh 连接vm
	执行：
	1. docker stop current container
	2. docker pull latest image
	3. docker run container

创建1个cloud build trigger, 当有新的代码push 到指定branch的时候自动出发cloud build, 有1个前提，这个 code repository 必须是github, github Enterprise, GitLab, BitBucket 之一，国内的一些代码仓库就算了

实现

1. 准备一对key pair

具体步骤忽略
确保可以用ssh key 登陆对应的vm

[gateman@manjaro-x13 keys]$ gcloud compute ssh tf-vpc0-subnet0-vm0 --ssh-key-file=/home/gateman/.ssh/id_rsa
No zone specified. Using zone [europe-west2-c] for instance: [tf-vpc0-subnet0-vm0].
External IP address was not found; defaulting to using IAP tunneling.
WARNING: 

To increase the performance of the tunnel, consider installing NumPy. For instructions,
please see https://cloud.google.com/iap/docs/using-tcp-forwarding#increasing_the_tcp_upload_bandwidth

Linux tf-vpc0-subnet0-vm0 5.10.0-30-cloud-amd64 #1 SMP Debian 5.10.218-1 (2024-06-01) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Sat Jun  8 16:15:23 2024 from 35.235.242.17
gateman@tf-vpc0-subnet0-vm0:~$

2. 把这对key pari 放在 google security manager 中

key pair 是敏感数据，建议不要用terraform, 避免敏感数据check in 到代码仓库
在这里插入图片描述

3. 编写cloudbuild.yaml

在这里插入图片描述

注意这里的第一个步是多余的并不起作用，因为在
deploy image to GCE 这个step中

会利用gcloud 命令重新下载key file pairs
只是简单介绍下引入 secret manager item的方法

至于为何不用第一步的方法，因为遇到了1个奇怪的错误, prviate key format is not valid.
我已经raised 1个ticket 给google support, 等下文

在这里插入图片描述

cloudbuild-gce.yaml:

steps:

  # to prepare ssh private key file
  - id: prepare ssh private key file
    name: 'ubuntu'
    entrypoint: bash
    args:
      - '-c'
      - |
        echo $$SSH_PRIVATE_KEY | cut -c 1-30
        echo $$SSH_PRIVATE_KEY > /workspace/ssh_key_file
        echo $$SSH_PUBLIC_KEY > /workspace/ssh_key_file.pub
        chmod 600 /workspace/ssh_key_file
        chmod 600 /workspace/ssh_key_file.pub
    secretEnv:
      - 'SSH_PRIVATE_KEY'
      - 'SSH_PUBLIC_KEY'



  - id: run maven package
    name: maven:3.9.6-sapmachine-17 # https://hub.docker.com/_/maven
    entrypoint: mvn
    args: [ 'package' ]

  # https://cloud.google.com/build/docs/configuring-builds/substitute-variable-values
  # i guess I should combine the two steps into one
  - id: build docker image
    name: 'gcr.io/cloud-builders/docker'
    args: [ 'build', '-t', 'europe-west2-docker.pkg.dev/$PROJECT_ID/my-docker-repo/${_APP_NAME}', '.' ]

  - id: upload docker image to GAR
    name: 'gcr.io/cloud-builders/docker'
    args: [ 'push', 'europe-west2-docker.pkg.dev/$PROJECT_ID/my-docker-repo/${_APP_NAME}' ]


  - id: deploy image to GCE
    name: 'gcr.io/cloud-builders/gcloud'
    entrypoint: bash
    args:
      - '-c'
      - |
        whoami
        set -x
        mkdir -p /root/.ssh
        gcloud secrets versions access latest --secret=gateman-private-ssh-key > /root/.ssh/id_rsa
        gcloud secrets versions access latest --secret=gateman-public-ssh-key > /root/.ssh/id_rsa.pub
        chmod 600 /root/.ssh/id_rsa
        chmod 600 /root/.ssh/id_rsa.pub
        gcloud compute ssh gateman@${_VM_HOST} --zone=europe-west2-c --quiet --ssh-key-file=/root/.ssh/id_rsa -- "whoami" 
        gcloud compute ssh gateman@${_VM_HOST} --zone=europe-west2-c --quiet --ssh-key-file=/root/.ssh/id_rsa -- "sudo docker container prune -f; sudo docker ps -a"  
        gcloud compute ssh gateman@${_VM_HOST} --zone=europe-west2-c --quiet --ssh-key-file=/root/.ssh/id_rsa -- "sudo docker stop ${_APP_NAME} && sudo docker rm ${_APP_NAME}" 
        gcloud compute ssh gateman@${_VM_HOST} --zone=europe-west2-c --quiet --ssh-key-file=/root/.ssh/id_rsa -- "sudo docker pull europe-west2-docker.pkg.dev/$PROJECT_ID/my-docker-repo/${_APP_NAME}:${_APP_TAG}"
        gcloud compute ssh gateman@${_VM_HOST} --zone=europe-west2-c --quiet --ssh-key-file=/root/.ssh/id_rsa -- "sudo docker run -d -p ${_PORT}:8080 --name ${_APP_NAME} europe-west2-docker.pkg.dev/$PROJECT_ID/my-docker-repo/${_APP_NAME}:${_APP_TAG}"
        echo ok
        
 
logsBucket: gs://jason-hsbc_cloudbuild/logs/
options: # https://cloud.google.com/cloud-build/docs/build-config#options
  logging: GCS_ONLY # or CLOUD_LOGGING_ONLY https://cloud.google.com/cloud-build/docs/build-config#logging

# to define
availableSecrets:
  secretManager:
    - versionName: projects/$PROJECT_ID/secrets/gateman-private-ssh-key/versions/latest
      env: 'SSH_PRIVATE_KEY'
    - versionName: projects/$PROJECT_ID/secrets/gateman-public-ssh-key/versions/latest
      env: 'SSH_PUBLIC_KEY'

substitutions:
  _APP_NAME: demo-cloud-user
  _APP_TAG: latest
  _PORT: "8081"

4. 创建1个cloudbuild trigger


# difference between data and resource: data is read only, resource is read and write
data google_service_account "cloudbuild_sa" {
  project = var.project_id
  account_id = "terraform"
}


# referring https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/cloudbuild_trigger
resource "google_cloudbuild_trigger" "demo_cloud_user-gce-trigger" {
  name = "demo-cloud-user-gce-trigger" # could not contains underscore

  location = var.region_id

  # when use github then should use trigger_template
  github {
    name = "demo_cloud_user"
    owner = "nvd11"
    push {
      branch = "main"
      invert_regex = false # means trigger on branch
    }
  }

  # the
  substitutions = {
    _VM_HOST = "tf-vpc0-subnet0-vm0"
  }

  filename = "cloudbuild-gce.yaml"
  # projects/jason-hsbc/serviceAccounts/terraform@jason-hsbc.iam.gserviceaccount.com
  service_account = data.google_service_account.cloudbuild_sa.id 
}